Hey @nix2intel can you say a bit more about the package and its uses here in your post?

I feel using a NIF for this is overkill, as ultimately the bottleneck will be always the requests you send to those permutated domains.

What’s the significance of a NIF (Rust) here? Why is it needed?

I don’t disagree and at some point I may come back and write the permutation engine in elixir. I mention the NIF because I want folks to know what they are signing up for, as it is my understanding NIFS can bring down the beam correct? I am very new to elixir and this was more about getting things done for work (we were using dnstwist at work written in python hosted on an api) twistrs was already written for the permutations and it felt like a quick win and a way to solve a problem and add a nice library (work is great about letting us open source libraries but not main functionality. I have been doing a lot of dev work since moving from security architecture to threat intelligence and I used elixir to rewrite a tool at work that parses through sec filings looking for data breaches, the first iteration was terrible and would fail whenever a websocket connection failed, I was researching to try and fix it and discovered elixir. Has been a game changer at work and this will go a long way to updating our stack. Having said that could you look at my task implementation on the elixir side and make suggestions for speed ups? As you said requests are going to be the most time consuming piece, I’m considering flame on kubernetes cluster but may be overkill? I’ve been using elixir for less than a year and still very much learning.

• The Rustler version you are using there is quite old and lacks (e.g.) forward compatibility in the build process
• You have committed the binaries
• You can simplify the NIF function quite a bit by using more of Rustler’s (and Rust’s) features:

use rustler::NifResult;
use std::collections::HashSet;
use twistrs::permutate::Domain;

// 1. Rustler has builtin support for maps with atom keys
// -------------------------------------------------------
#[derive(rustler::NifMap)]
struct Result {
    fqdn: String,
    tld: String,
    kind: String,
}

// 2. You can return anything that is convertible to a Term, you don't need to do inline encoding (and thus don't need `env`, together with `NifMap`)
// ------------------------------------------
#[rustler::nif]
fn generate_permutations(domain_str: String) -> NifResult<Vec<Result>> {
    let domain = match Domain::new(&domain_str) {
        Ok(d) => d,
        Err(_) => return Ok(Default::default()),
    };

    // 3. No need to convert the HashSet into a Vec if you just want to
    //    iterate over it again
    // -----------------------------------------
    let perms = match domain.all() {
        Ok(p) => p.collect::<HashSet<_>>(),
        Err(_) => return Ok(Default::default()),
    };

    let results = perms
        .iter()
        .map(|p| Result {
            fqdn: p.domain.fqdn.clone(),
            tld: p.domain.tld.clone(),
            kind: format!("{:?}", p.kind),
        })
        .collect();

    Ok(results)
}

rustler::init!("Elixir.DomainTwistex");

It would be even shorter if you didn’t use NifResult (which is unnecessary as you don’t report any errors back).