jola

jola

Why do releases use so much more memory than `mix run`?

A question was asked here whether releases use less memory, so checked in a project I’m working on and I was surprised to see almost twice the memory use for the same code. To make sure it wasn’t just my project, I tried with a plain Phoenix app and only the minimal steps to get releases started. This is what I did if you want to reproduce:

mix phx.new hello --no-webpack --no-html --no-ecto
# added {:distillery, "~> 2.0"} to mix.exs
cd hello
mix deps.get
MIX_ENV=prod mix release

Then to compare the memory use I relied on :erlang.memory/0. First using plain mix run in iex.

MIX_ENV=prod iex -S mix run

iex(1)> :erlang.memory 
[
  total: 31407808,
  processes: 7082192,
  processes_used: 7063440,
  system: 24325616,
  atom: 512625,
  atom_used: 504423,
  binary: 97440,
  code: 10298466,
  ets: 929712
]

and then starting the release in the equivalent console

_build/prod/rel/hello/bin/hello console

iex(hello@127.0.0.1)1> :erlang.memory 
[
  total: 55362432,
  processes: 14649704,
  processes_used: 14363384,
  system: 40712728,
  atom: 984241,
  atom_used: 961617,
  binary: 174464,
  code: 21349495,
  ets: 2029176
]

Total reported memory use went from 31MB to 55MB. And you see increases across the board. In my personal project the increase was even more significant, going from 45MB to 90MB.

One thing I guess could be related is that I believe releases preload all code while mix will load modules on demand. But does that really account for the whole difference?

I also want to add a disclaimer: I’m still using releases and I don’t find the memory used to be unreasonable. This is just curiosity!

Most Liked

michalmuskala

michalmuskala

I think the code loading part will be the major difference - more code means more atoms, more binaries (from literals in those modules) and more ets usage (since information which modules are loaded is held in an ets table by the code_server process). Additionally there might be some more system-level services running like SASL or the release handlers, which could account for the extra process size. Have you tried comparing the outputs of Process.list() on both systems? That could shed some more light on it. Similarly for ets tables you could investigate with :ets.info.

rvirding

rvirding

Creator of Erlang

Check to see which applications have been loaded, there may be more in the release. Also check which applications have been started. IIRC correctly the release will start all applications at start up time.

tristan

tristan

Rebar3 Core Team

Your release is likely starting in embedded mode while mix run is interactive mode. Embedded mode will load all beam modules on start up instead of only when they are used.

See http://erlang.org/doc/system_principles/system_principles.html#code-loading-strategy

rodrigues

rodrigues

passing --preload-modules to mix run seems to do the job, actually it loads 133 modules more than release, while the modules below are loaded by release but not loaded in mix run, even with preload:

[:sasl_report, :systools_rc, :alarm_handler, :release_handler_1,
 :format_lib_supp, :misc_supp, :systools_make, :sasl, :sasl_report_tty_h,
 :sasl_report_file_h, :erlsrv, :systools_lib, :release_handler, :systools, :rb,
 :rb_format_supp, :systools_relup]

It does increase memory significantly, but release still has more (gist with output).

rodrigues

rodrigues

I’ve had similar numbers here, @jola, using your steps.

Following @rvirding suggestion, I’ve ran this code in each:

Application.started_applications() |> Enum.map(&elem(&1, 0)) |> Enum.sort()

# release
# $ _build/prod/rel/hello/bin/hello console
[:artificery, :asn1, :compiler, :cowboy, :cowlib, :crypto, :distillery, :eex,
 :elixir, :gettext, :hello, :iex, :jason, :kernel, :logger, :mime, :mix,
 :phoenix, :phoenix_pubsub, :plug, :plug_cowboy, :plug_crypto, :public_key,
 :ranch, :runtime_tools, :sasl, :ssl, :stdlib]

# mix
# $ MIX_ENV=prod iex -S mix run
[:artificery, :asn1, :compiler, :cowboy, :cowlib, :crypto, :distillery, :eex,
 :elixir, :gettext, :hello, :hex, :iex, :inets, :jason, :kernel, :logger, :mime,
 :mix, :phoenix, :phoenix_pubsub, :plug, :plug_cowboy, :plug_crypto,
 :public_key, :ranch, :runtime_tools, :ssl, :stdlib]
  • release had extra: sasl
  • mix run had extra: hex, inets

I guess sasl is the likely culprit?

UPDATE: sasl start not relevant to memory. I’ve checked also the loaded applications, and it’s the same as started applications. That’s intriguing :+1:t3:

# :erlang.memory in mix option after start
[
  total: 34951112,
  processes: 13154512,
  processes_used: 13153312,
  system: 21796600,
  atom: 512625,
  atom_used: 505434,
  binary: 206016,
  code: 10368934,
  ets: 942080
]

# :erlang.memory in mix option, after starting sasl
[
  total: 35274176,
  processes: 13233528,
  processes_used: 13230984,
  system: 22040648,
  atom: 520817,
  atom_used: 517966,
  binary: 205984,
  code: 10547076,
  ets: 957688
]

# :erlang.memory in release option
[
  total: 60294456,
  processes: 21970552,
  processes_used: 21847048,
  system: 38323904,
  atom: 992433,
  atom_used: 971447,
  binary: 441208,
  code: 21443760,
  ets: 1992328
]

Where Next?

Popular in Questions Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
quazar
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New

Other popular topics Top

sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
977 41022 311
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New

We're in Beta

About us Mission Statement