leonardorame

leonardorame

Long running C++ NIF segfaults

Hi, I built a NIF out of a well known C++ Dicom library called DCMTK. The purpouse of this NIF is to stay loaded for the whole life of my Elixir/Phoenix application, where it’s functions are called many times.

The module that loads this NIF is called from an Agent and I thought it stay there forever, but as it is segfaulting after using it for a while I’m starting to think it is removed from memory after some time…

Let me show my code:

defmodule Prueba3.ImageCache do                                                                                                                                                                                                              
  use Agent                                                                                                                                                                                                                                  
  def start_link(_opts) do                                                                                                                                                                                                                   
    Dcmtknif.initialize()                                                                                                                                                                                                                    
    Agent.start_link(fn -> %{} end, name: :imagecache)                                                                                                                                                                                       
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def get(key) do                                                                                                                                                                                                                            
    Agent.get(:imagecache, &Map.get(&1, key))                                                                                                                                                                                                
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def put(key, value) do                                                                                                                                                                                                                     
    Agent.update(:imagecache, &Map.put(&1, key, value))                                                                                                                                                                                      
  end                                                                                                                                                                                                                                        
end
defmodule Dcmtknif do                                                                                                                                                                                                                        
  @on_load :load_nifs                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def load_nifs do                                                                                                                                                                                                                           
    :erlang.load_nif('./ssr/libssr', 0)                                                                                                                                                                                                      
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def _loadDicomFile(_dicomfile) do                                                                                                                                                                                                          
    raise "NIF _loadDicomFile/1 not implemented"                                                                                                                                                                                             
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def _getTagValue(_dcmff, _group, _element) do                                                                                                                                                                                              
    raise "NIF getTagValue/1 not implemented"                                                                                                                                                                                                
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def _getHeader(_dcmff) do                                                                                                                                                                                                                  
    raise "NIF getHeader/1 not implemented"                                                                                                                                                                                                  
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def _getPNG(_dcmff) do                                                                                                                                                                                                                     
    raise "NIF getPNG/1 not implemented"                                                                                                                                                                                                     
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def _setMinMaxWindow(_dcmff) do                                                                                                                                                                                                            
    raise "NIF setMinMaxWindow/1 not implemented"                                                                                                                                                                                            
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def _createScaledImage(_dcmff, _width, _height, _interpolate, _aspect) do                                                                                                                                                                  
    raise "NIF createScaledImage/5 not implemented"                                                                                                                                                                                          
  end;                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                             
  def _initialize() do                                                                                                                                                                                                                       
    IO.puts("Dcmtknif.initialize()")                                                                                                                                                                                                         
    raise "NIF initialize/0 not implemented"                                                                                                                                                                                                 
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def _finalize() do                                                                                                                                                                                                                         
    raise "NIF finalize/0 not implemented"                                                                                                                                                                                                   
  end   
  ... more functions
end

Do you see something wrong?

Most Liked

whatyouhide

whatyouhide

Elixir Core Team

I’m not an expert in NIFs, but in general the VM yields execution to the NIF when you call it, so you’re really not supposed to have “long-running NIFs”. When you enter a NIF, the VM cannot do GC, context, switching, and so on anymore.

I think a common solution in a case like yours, where you have essentially an external program that needs to run continuously, is to use ports. You spawn the program you want to run and you let your Elixir application manage its lifetime. Then, you communicate with the spawned program through stdout.

Does that make sense?

leonardorame

leonardorame

Hi @whatyouhide, I think I wrongly explained what this NIF does. With “long-running” I mean it has state (internal variables pointing to data structures that can be accessed from the parent Elixir process), please don’t confuse with long loops or heavy processes inside it.

Regarding your ports suggestion, I can do that, but it adds complexity on the C++ side and I’m trying to maintain that part as small as possible.

jhogberg

jhogberg

Erlang Core Team

I can assure you that the NIF hasn’t been unloaded. You would need to do that manually, and even then the VM takes great care not to pull the rug while anything references it.

I think I can see why it crashes though. What’s preventing image/2 from being called concurrently?

jhogberg

jhogberg

Erlang Core Team

Imagine that you get several concurrent calls to image/2 with the same sopinstanceuid.

Isn’t there then a pretty good chance that they all end up calling createScaledImage_nif, setMinMaxWindow_nif, and getPNG_nif with the same resource?

Also, would you care to share the code for getPNG_nif?

jhogberg

jhogberg

Erlang Core Team

Thanks, having read the NIF and DCMTK source it’s very likely that this is a thread-safety issue. None of the DCMTK functions you use are documented as thread-safe and I saw too many thread-unsafe things to count while reading their source.

That the problem went away after commenting-out lines 27 through 38 also suggests that this is the case.

Try caching results instead of resources if width/height are mostly static, or if they’re very dynamic, create a new process for each instance and store those in the cache. When a new request comes in, ask the cached process to scale the image.

Where Next?

Popular in Questions Top

openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
pgiesin
This should be a simple problem but I just can’t seem to figure it out. I have a standalone Elixir app that won’t find the database. Dep...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
dokuzbir
Hello, I am trying to convert my lists to string without losing brackets.For start i have 3 map. They look like these buyer = %{ id: ...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
Exadra37
Sometimes I want to check if the input into a function is not a blank string. My first approach: defmodule Example do def do_stuff(s...
New
jc00ke
Expanding on this topic: https://forum.elixirforum.net/t/map-typespec-question/19217 Let’s say I have a map with required and optional k...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Other popular topics Top

vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement