leonardorame

leonardorame

Long running C++ NIF segfaults

Hi, I built a NIF out of a well known C++ Dicom library called DCMTK. The purpouse of this NIF is to stay loaded for the whole life of my Elixir/Phoenix application, where it’s functions are called many times.

The module that loads this NIF is called from an Agent and I thought it stay there forever, but as it is segfaulting after using it for a while I’m starting to think it is removed from memory after some time…

Let me show my code:

defmodule Prueba3.ImageCache do                                                                                                                                                                                                              
  use Agent                                                                                                                                                                                                                                  
  def start_link(_opts) do                                                                                                                                                                                                                   
    Dcmtknif.initialize()                                                                                                                                                                                                                    
    Agent.start_link(fn -> %{} end, name: :imagecache)                                                                                                                                                                                       
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def get(key) do                                                                                                                                                                                                                            
    Agent.get(:imagecache, &Map.get(&1, key))                                                                                                                                                                                                
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def put(key, value) do                                                                                                                                                                                                                     
    Agent.update(:imagecache, &Map.put(&1, key, value))                                                                                                                                                                                      
  end                                                                                                                                                                                                                                        
end
defmodule Dcmtknif do                                                                                                                                                                                                                        
  @on_load :load_nifs                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def load_nifs do                                                                                                                                                                                                                           
    :erlang.load_nif('./ssr/libssr', 0)                                                                                                                                                                                                      
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def _loadDicomFile(_dicomfile) do                                                                                                                                                                                                          
    raise "NIF _loadDicomFile/1 not implemented"                                                                                                                                                                                             
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def _getTagValue(_dcmff, _group, _element) do                                                                                                                                                                                              
    raise "NIF getTagValue/1 not implemented"                                                                                                                                                                                                
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def _getHeader(_dcmff) do                                                                                                                                                                                                                  
    raise "NIF getHeader/1 not implemented"                                                                                                                                                                                                  
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def _getPNG(_dcmff) do                                                                                                                                                                                                                     
    raise "NIF getPNG/1 not implemented"                                                                                                                                                                                                     
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def _setMinMaxWindow(_dcmff) do                                                                                                                                                                                                            
    raise "NIF setMinMaxWindow/1 not implemented"                                                                                                                                                                                            
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def _createScaledImage(_dcmff, _width, _height, _interpolate, _aspect) do                                                                                                                                                                  
    raise "NIF createScaledImage/5 not implemented"                                                                                                                                                                                          
  end;                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                             
  def _initialize() do                                                                                                                                                                                                                       
    IO.puts("Dcmtknif.initialize()")                                                                                                                                                                                                         
    raise "NIF initialize/0 not implemented"                                                                                                                                                                                                 
  end                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                             
  def _finalize() do                                                                                                                                                                                                                         
    raise "NIF finalize/0 not implemented"                                                                                                                                                                                                   
  end   
  ... more functions
end

Do you see something wrong?

Most Liked

whatyouhide

whatyouhide

Elixir Core Team

I’m not an expert in NIFs, but in general the VM yields execution to the NIF when you call it, so you’re really not supposed to have “long-running NIFs”. When you enter a NIF, the VM cannot do GC, context, switching, and so on anymore.

I think a common solution in a case like yours, where you have essentially an external program that needs to run continuously, is to use ports. You spawn the program you want to run and you let your Elixir application manage its lifetime. Then, you communicate with the spawned program through stdout.

Does that make sense?

leonardorame

leonardorame

Hi @whatyouhide, I think I wrongly explained what this NIF does. With “long-running” I mean it has state (internal variables pointing to data structures that can be accessed from the parent Elixir process), please don’t confuse with long loops or heavy processes inside it.

Regarding your ports suggestion, I can do that, but it adds complexity on the C++ side and I’m trying to maintain that part as small as possible.

jhogberg

jhogberg

Erlang Core Team

I can assure you that the NIF hasn’t been unloaded. You would need to do that manually, and even then the VM takes great care not to pull the rug while anything references it.

I think I can see why it crashes though. What’s preventing image/2 from being called concurrently?

jhogberg

jhogberg

Erlang Core Team

Imagine that you get several concurrent calls to image/2 with the same sopinstanceuid.

Isn’t there then a pretty good chance that they all end up calling createScaledImage_nif, setMinMaxWindow_nif, and getPNG_nif with the same resource?

Also, would you care to share the code for getPNG_nif?

jhogberg

jhogberg

Erlang Core Team

Thanks, having read the NIF and DCMTK source it’s very likely that this is a thread-safety issue. None of the DCMTK functions you use are documented as thread-safe and I saw too many thread-unsafe things to count while reading their source.

That the problem went away after commenting-out lines 27 through 38 also suggests that this is the case.

Try caching results instead of resources if width/height are mostly static, or if they’re very dynamic, create a new process for each instance and store those in the cache. When a new request comes in, ask the cached process to scale the image.

Where Next?

Popular in Questions Top

yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
LegitStack
I’m hoping you guys can give me some general advice and perhaps code examples if you’re feeling up to it. I’m very interested in Elixir,...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New

Other popular topics Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
peerreynders
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
326 29600 154
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

We're in Beta

About us Mission Statement