After 3 days of pain, I successfully added rustler_precompiled to kuzu_nif and published to hex.pm. Users of kuzu_nif on supported targets (see below) no longer need to compile the Rust crate.

Why? This saves us a few minutes on every build. For me, I’m using kuzu_nif in a Phoenix Framework project. Having a precompiled version of kuzu_nif means 1) I don’t need to install a Rust compiler in my Phoenix Framework dockerfile and 2) I save precious time compiling my Phoenix app.

• hex.pm/packages/kuzu_nif
• GitHub: Release v0.7.0

The supported targets are indicated in the packaged release files. TLDR; Apple arm/x86 and Debian Linux arm/x64.

• libkuzu_ex-v0.7.0-nif-2.17-aarch64-apple-darwin.so.tar.gz
• libkuzu_ex-v0.7.0-nif-2.17-aarch64-unknown-linux-gnu.so.tar.gz
• libkuzu_ex-v0.7.0-nif-2.17-x86_64-apple-darwin.so.tar.gz
• libkuzu_ex-v0.7.0-nif-2.17-x86_64-unknown-linux-gnu.so.tar.gz

Figuring out how to successfully compile on aarch64 linux was the hardest part. In the end, I had to compile a custom Docker image just for building aarch64-unknown-linux-gnu.

Enjoy!

The thread creation stuff in enif only exists to provide a platform-independent API for threading in C. They don’t have (to my knowledge) any deeper “integration” with the BEAM.

One thing to keep in mind is that not all work has to happen in the actual NIF call. You can easily spawn a thread (or a pool of threads) and pass the PID of the caller that you then use to send the result of the query back to. This is what I use in my XML processing faster_xml/rust_src/faster_xml_nif/src/lib.rs at master · filmor/faster_xml · GitHub and it works fine in production, even without an actual thread pool.

So, simple suggestion to get started: Have a Database stored in a resource object, and on every query, spawn a thread that creates a Connection and sends the result back.

Later you can refine this to use a thread pool (with some retry logic on the Elixir side).

That way, the NIF will be quick enough that you don’t need to worry about scheduling and you get proper concurrent processing.

NIFs are executed by regular scheduled threads of the VM. See [the Erlang docs] (erl_nif — erts v15.1.2) for the big fat warning on long-running work, special care should be taken when a call takes more time than about 1ms. I usually just scheduled NIFs on an async scheduler (dirty CPU mostly) when the NIF was supposed to work for some longer time.

Note that Rustler doesn’t implement all of erl_nif, for example creating threads in the VM is not implemented as rust threads can be spawned if necessary.

Thanks for reminding me about this. I just updated the kuzu crate from 0.6.0 to 0.7.0 and improved the Elixir unit test.

We can do some pretty useful things with just run_query, like create a kuzudb using COPY FROM commands.

I was curious so I took a peek, and I think this function might benefit from being scheduled on a dirty scheduler: kuzu_nif/native/kuzu_ex/src/lib.rs at f94c2e243983650db26c4b19bb0f73835fd52ec5 · bgoosmanviz/kuzu_nif · GitHub and re: “mapping processes to threads”, I think usually libraries for wrappers around embedded databases create a NIF resource per database connection and then Elixir / Erlang manage them however they want, no 3rd-party threads involved. Looking at kuzu_nif/native/kuzu_ex/src/lib.rs at f94c2e243983650db26c4b19bb0f73835fd52ec5 · bgoosmanviz/kuzu_nif · GitHub, it seems like the API might be similar to DuckDB wrappers, since they also have a notion of “databases” and “connections” (SQLite only has “databases”). Two example DuckDB wrappers in Elixir:

• GitHub - AlexR2D2/duckdbex: The Library embeds C++ DuckDB database into you Elixir application.
• GitHub - ruslandoga/duxdb: DuckDB bindings for Elixir

So I think the API could be:

# NIF resource, DB is closed when the underlying reference refcount goes to 0
# can be a Dirty IO NIF
db = Kuzu.open() 

conns =
  for _ <- 1..:erlang.system_info(:dirty_io_schedulers) do
    # also NIF resources, since Kuzu is C++
    # they probably reference and keep DB alive too,
    # so it's easy to just wrap them info resources individually.
    # When none of them are referenced anymore, Erlang would "garbage collect"
    # them and call NIF resource destructors, which would destroy Kuzu connections.
    # once Kuzu connections are destroyed, Kuzu would probably close DB as well
    # since it would no longer be referenced either.
    # can be a Dirty CPU NIF or a plain NIF, depending on what Kuzu does here
    Kuzu.connect(db)
  end

{:ok, pool} = GenServer.start_link(SomePool, conns)
{ref, conn} = GenServer.call(pool, :checkout)

result =
  try do
    # since it might trigger disk access, it would be safer if it was a Dirty IO NIF
    Kuzu.query(conn, query)
    # or it can be a Kuzu.prepare followed by a Kuzu.execute
    # this would allow statement caching and query params
    # Kuzu.prepare can be a Dirty CPU or a plain NIF
    # Kuzu.execute would be a Dirty IO NIF for the same reasons as Kuzu.query
  after
    GenServer.cast(pool, {:checkin, ref, conn})
  end

# can probably be a plain NIF
Kuzu.to_arrow(result)

I think this would be useful in the upcoming AoC