solanav
Dirty CPU NIF blocking scheduler
Hello everyone!
I’m currently working on a web spider/metadata extractor and because writing parsers for many file types would be too much work, I’m using libextractor.
I wrote a NIF which I learned only allows for functions that return in less than ~1ms. So I just changed this:
static ErlNifFunc funcs[] = {
{ "extract", 2, extract }
};
into this:
static ErlNifFunc funcs[] = {
{ "extract", 2, extract, ERL_NIF_DIRTY_JOB_CPU_BOUND }
};
The problem: it seems like the scheduler gets to 100% utilization which is bad because the spider downloads a lot of files in parallel so it becomes impossible to do things like launch the observer if I’m calling the NIF. Any idea if this is normal? Is there a way I can prevent the NIF from blocking the rest of the system?
Here is a picture of the observer while I test the calls to my main function:
Thanks!
Marked As Solved
sasajuric
While a nif is running on a scheduler its utilization will be plotted as 100%. This is normal.
If the NIF is running on a dirty scheduler it should not block regular schedulers. Here’s how I tested it locally:
- In the C code of the dirty NIF function I added
sleep(10); - I started the shell with
ELIXIR_ERL_OPTIONS="+S 1" iex -S mix(to make sure there’s just one scheduler and one dirty CPU scheduler) - From the shell I started the nif as
spawn(&my_nif/0) - As soon as process was spawned the iex shell was responsive and I was able to start the observer.
If you invoke a lot of longer-running dirty nifs concurrently, you may end up occupying all the dirty scheduler threads, which might block some other operations. For example, length will switch to dirty schedulers for larger list, and if all schedulers are busy for a couple of seconds, length will be blocked. I was able to reproduce this by spawning a long-running NIF, and then invoking length(Enum.to_list(1..1_000_000)) which blocked until NIF finished.
Based on the description of the scenario this looks like a case where port might be a better fit.
Also Liked
dch
For NIF related code I usually refer to Paul Davis, Jesper Andersen, & Steve Vinoski, in no particular order. These are all in erlang, but that’s not really important for NIFs anyway.
If people have other examples in Elixir – I expect Frank Hunleth has a bunch of amazing Nerves ones. I’d hope they can chime in with some examples.
Paul’s authored jiffy, probably the most widely used NIF of all, Steve was instrumental in documenting and authoring the first working dirty scheduler code to my knowledge, and Jesper’s enacl NIF comes with excellent explanatory notes in addition.
- https://github.com/davisp/jiffy & https://github.com/apache/couchdb-erlfdb
- https://github.com/jlouis/enacl & https://medium.com/@jlouis666/erlang-dirty-scheduler-overhead-6e1219dcc7
- https://github.com/basho/enm
The first two are both used extensively and show IMO canonical examples of building cross-platform compilation solutions. The middle is really well documented, and the latter shows how you can ship C source as part of the code and have it compiled locally.
NobbZ
Are you sure that your NIF is CPU bound and not IO bound?
solanav
Thank you so much for the explanation, I will rewrite the NIF as a port to see if it’s a better choice.
I’m a big fan by the way, keep up the good work!
ityonemo
Can you set the number of dirty nif schedulers to number of cores - 1? Probably easiest way to test this is to do the following at startup:
:erlang.system_flag(:dirty_cpu_schedulers, System.schedulers_online() - 1)
nonblockio
It seems that the performance surge only least for one or two seconds from the chart. :msacc.start(1000) is kinda too short to catch the vibration since it only analyze scheduler in 1000ms. Could you increase the time of :msacc.start/1 and try to fit the surge in that period?
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Develop your skills with books, videos, and courses.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"