ollran

Running Elixir with limited heap size

Hi,

Is it possible to somehow limit the memory usage of the Erlang virtual machine? I’m on a shared server and I don’t have root access.

For example, when I want to run Java I cannot use the default settings:

$ java
Error occurred during initialization of VM
Could not allocate metaspace: 1073741824 bytes

I have to start the JVM with limited heap size:

$ java -XX:MaxHeapSize=512m -XX:CompressedClassSpaceSize=64m -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

$ java -XX:MaxHeapSize=512m -XX:CompressedClassSpaceSize=64m HelloWorld
Hello, World

I am using the new release feature that came in Elixir 1.9. When I try to run my Elixir application I get this error:

$ _build/prod/rel/webapp/bin/webapp start
2019-07-02 15:53:49.850859
    args: [load_failed,"Failed to load NIF library: '/var/www/_build/prod/rel/webapp/lib/asn1- 5.0.9/priv/lib/asn1rt_nif.so: failed to map segment from shared object: Cannot allocate memory'"]
format: "Unable to load asn1 nif library. Failed with error:~n\"~p, ~s\"~n"
label: {error_logger,error_msg}
2019-07-02 15:53:49.852625 crash_report        #{label=>{proc_lib,crash},report=>[[{initial_call,{supervisor,kernel,['Argument__1']}},{pid,<0.976.0>},{registered_name,[]},{error_info,{exit,{on_load_function_failed,asn1rt_nif},[{init,run_on_load_handlers,0,[]},{kernel,init,1,[{file,"kernel.erl"},{line,187}]},{supervisor,init,1,[{file,"supervisor.erl"},{line,295}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}},{ancestors,[kernel_sup,<0.950.0>]},{message_queue_len,0},{messages,[]},{links,[<0.952.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,265}],[]]}
2019-07-02 15:53:49.852655 supervisor_report   #{label=>{supervisor,start_error},report=>[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{on_load_function_failed,asn1rt_nif}},{offender,[{pid,undefined},{id,kernel_safe_sup},{mfargs,{supervisor,start_link,[{local,kernel_safe_sup},kernel,safe]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]}
2019-07-02 15:53:50.858601 crash_report        #{label=>{proc_lib,crash},report=>[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.949.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,138}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}},{ancestors,[<0.948.0>]},{message_queue_len,1},{messages,[{'EXIT',<0.950.0>,normal}]},{links,[<0.948.0>,<0.947.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,193}],[]]}
2019-07-02 15:53:50.861379 std_info            #{label=>{application_controller,exit},report=>[{application,kernel},{exited,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}}},{type,permanent}]}
{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,asn1rt_nif}}},{kernel,start,[normal,[]]}}})

Crash dump is being written to: erl_crash.dump...done

Is it possible to limit the heap size in Elixir/Erlang?

14 comments

#beam #memory #mix-release

30 4950 14

2019-07-04 08:31:56 UTC

Marked As Solved

alvises

To limit VM heap size I’ve found this: [erlang-questions] Max heap size

It says to pass the options +MMsco true +Musac false +MMscs 60 (You find these options here Erlang -- erts_alloc) where 60 is the limit in MB.

I’ve tried it and when the VM reached the limit it crashes…

 eheap_alloc: Cannot allocate 600904 bytes of memory (of type "heap")

@ollran what does it happen to the JVM when it reaches the memory limit?

Post #3

Also Liked

garazdawi

Erlang Core Team

You may want to try +Meamin. This will disable all the erts memory allocators and fall back to malloc which can sometimes use less memory.

Post #15

NobbZ

http://erlang.org/documentation/doc-6.1/erts-6.1/doc/html/erl.html

There should be plenty of options available to deal with that, but it seems as if your problem is in a NIF. It won’t be restricted by the VMs Heap.

Post #2

NobbZ

Its not at all… Consider it an external program that runs side by side to your erlang program and both are talking via a defined interface. The BEAM can’t control anything about that program except for its lifecycle… Start and stop…

But how restricted is your memory? I had erlang applications starting with less than 100 MB available, I had myself a DO 5$ droplet where the application never used more than 150 MB peak…

Also, as the problem seems to be asn1 related, do you really need it or can you strip that dependency?

Post #5

jola

I don’t get it. You’re trying to run an app that requires X memory and you have <X available. It doesn’t matter if you’re using JVM or BEAM. You even demonstrated that yourself by testing the same thing with JVM and it crashing.

Post #6

NobbZ

Your resulting image is unnecessarily bloated.

You probably do not need a full debian:8 under your feets.

Using the builder images by @beardedeagle combined with a multistage build you can get image sizes far less than 100 MB.

Also remember, if done correctly, neither erlang, nor elixir are a runtime requirement.

So in theory you could even use a base image that is compatible to your server, build a release therein, extract that release and only deploy the extracted release to your server. This will probably result in the smallest deployment size.

Post #14