sdr

sdr

High memory usage when performance testing simple Rustler NIFs

I’ve been playing around with Rustler NIFs to get familiar with them and understand how they interact with Elixir.

While running some simple toy NIFS and benchmarking them to be impressed with the performance differences, I was surprised with :process-memory-high-watermark warnings and opening the observer showed that memory had ballooned up to a couple of GBs.

The code isn’t creating any long term data and simply sums the numbers from A…B so I was surprised to see the jump in memory. The create is built release mode and benched with Benchee and it’s unclear what’s causing the memory jump.

Code is as follows

#[rustler::nif]
fn add(a: i64, b: i64) -> i64 {
    let mut r:i64 = 0;
    for x in a..b {
        r += x;
    }
    return r;
}

rustler::init!("Elixir.FastMath", [add]);

Any hints on where to look or insights into what’s causing the high memory usage?

Marked As Solved

sdr

sdr

After building some elixir/erlang muscle this has nothing to do with NIFs but is caused by Benchee collecting a list of samples for each iteration.

The same memory usage can be seen with a no-op function:
Benchee.run(%{fast: fn -> :ok end})

For short function calls this results in a large list (order of millions) of samples which explains the the memory spike.

Also Liked

sdr

sdr

A minimal repo to replicate this can be found at https://github.com/sross/niftest

sdr

sdr

Can confirm that I’m still seeing the same behaviour with rustler 0.26.0 and benchee 1.1.0 on Erlang 1.14 with OTP 25 under WSL.

Memory spikes to just under 3GB and the drops back to 1.2GB. the drop back to normal levels at the 6 second mark is when I ran a manual garbage collect.

bdarla

bdarla

I experimented with the provided repo and here are the results in my machine (Intel MBP with latest versions of Elixir and Rust as well as the employed libraries, i.e. rustler v0.26 and benchee v1.1).
I am not sure whether the issue remains or not. If anyone could judge better than me, please, do so.

Where Next?

Popular in Questions Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
LegitStack
I’m hoping you guys can give me some general advice and perhaps code examples if you’re feeling up to it. I’m very interested in Elixir,...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
logicmason
Hi there, I'm working through my first release with elixir/phoenix. I've built a release with distillery and found that it crashes when I...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New

Other popular topics Top

peerreynders
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
326 29600 154
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New

We're in Beta

About us Mission Statement