sdr

sdr

High memory usage when performance testing simple Rustler NIFs

I’ve been playing around with Rustler NIFs to get familiar with them and understand how they interact with Elixir.

While running some simple toy NIFS and benchmarking them to be impressed with the performance differences, I was surprised with :process-memory-high-watermark warnings and opening the observer showed that memory had ballooned up to a couple of GBs.

The code isn’t creating any long term data and simply sums the numbers from A…B so I was surprised to see the jump in memory. The create is built release mode and benched with Benchee and it’s unclear what’s causing the memory jump.

Code is as follows

#[rustler::nif]
fn add(a: i64, b: i64) -> i64 {
    let mut r:i64 = 0;
    for x in a..b {
        r += x;
    }
    return r;
}

rustler::init!("Elixir.FastMath", [add]);

Any hints on where to look or insights into what’s causing the high memory usage?

Marked As Solved

sdr

sdr

After building some elixir/erlang muscle this has nothing to do with NIFs but is caused by Benchee collecting a list of samples for each iteration.

The same memory usage can be seen with a no-op function:
Benchee.run(%{fast: fn -> :ok end})

For short function calls this results in a large list (order of millions) of samples which explains the the memory spike.

Also Liked

sdr

sdr

A minimal repo to replicate this can be found at https://github.com/sross/niftest

sdr

sdr

Can confirm that I’m still seeing the same behaviour with rustler 0.26.0 and benchee 1.1.0 on Erlang 1.14 with OTP 25 under WSL.

Memory spikes to just under 3GB and the drops back to 1.2GB. the drop back to normal levels at the 6 second mark is when I ran a manual garbage collect.

bdarla

bdarla

I experimented with the provided repo and here are the results in my machine (Intel MBP with latest versions of Elixir and Rust as well as the employed libraries, i.e. rustler v0.26 and benchee v1.1).
I am not sure whether the issue remains or not. If anyone could judge better than me, please, do so.

Where Next?

Popular in Questions Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
LegitStack
I’m hoping you guys can give me some general advice and perhaps code examples if you’re feeling up to it. I’m very interested in Elixir,...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New

Other popular topics Top

pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 45766 226
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 27727 240
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 42633 214
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New

We're in Beta

About us Mission Statement