Questions & Help>Questions

Memory leak with quantum and processing binary files

#performance#quantum#troubleshooting#memory
dfens

dfens

Memory leak with quantum and processing binary files

Currently I have phoenix app which also schedules quantum jobs (GitHub - quantum-elixir/quantum-core: ⌚ Cron-like job scheduler for Elixir) with overlap: false mode
Jobs are importing & exporting relatively large json files.

Now my problem is

Screenshot from 2020-03-06 16-29-16
Screenshot from 2020-03-06 16-29-16507×130 7.94 KB

Leaking memory

This is current status and it will get worse since different/bigger data sets are going to be processed.

For example on start on the left app starts with ~60MB

Screenshot from 2020-03-06 16-34-47
Screenshot from 2020-03-06 16-34-47613×255 11.5 KB

Flat line in the end is after quantum job was processed

I did check also this memory management - Solving large binaries leak - Stack Overflow but forced gc / tuning ERL_FULLSWEEP_AFTER collect did not seem to help

If I run ImportTask/ExportTask on my local machine but not as quantum and observe memory - then I cannot reproduce this behaviour

Any tips welcome :slight_smile:

7 comments
#performance#quantum#troubleshooting#memory
6 1414 7

Marked As Solved

NobbZ

NobbZ

Have you tried strings: :copy (assuming you use Jason)?

4
Post #2

Also Liked

dfens

dfens

@NobbZ jackpot!
as the matter of fact I was using Jaxon for parsing since files were pretty big before. but just tested with Jason strings: :copy and issue does not occur anymore. I suspect this was misuse of Jaxon.stream on my side but I am completely satisfied with Jason results

@benwilson512 yes I did try, without any effect

Thank you so much for your input guys

1
Post #4
NobbZ

NobbZ

From a quick glance it seems as if jaxon wasn’t copying and also did not provide an option to copy binaries.

You might still be able to use jaxon when you :binary.copy/1 strings it gives you manually.

1
Post #5

Where Next?

View thread on forum (has 7 responses!)
performance
quantum
troubleshooting
memory
Home Questions & Help>Questions
#performance#quantum#troubleshooting#memory
6 1415 7
Last post

Popular in Questions Top

logicmason
Questions & Help>Questions

What's the best way to serve restricted ports (e.g. 80, 443) with Phoenix?
Hi there, I'm working through my first release with elixir/phoenix. I've built a release with distillery and found that it crashes when I...
/phoenix#deployment#linux
32 16824 13
New
jononomo
Questions & Help>Questions

Where / How does the Mix environment variable get set?
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
#mix
31 22791 9
New
pmjoe
Questions & Help>Questions

What do you think of Gleam compared to Elixir?
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
#programminguages#gleam
24 16555 10
New
romenigld
Questions & Help>Questions

Docker run Error loading shared library libstdc++.so.6 and libgcc_s.so.1
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
/phoenix#deployment#docker-compose
30 19932 15
New
gazoon
Questions & Help>Questions

Current file (module) path
I want to know absolute current module path. In python i could do that: os.path.abspath(__file__) Does elixir have anything similar?
20 13594 7
New
mgjohns61585
Questions & Help>Questions

Convert a string to an integer?
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
#how-to-question
3 14200 6
New
vonH
Questions & Help>Questions

How do I kill a process ` #PID<0.186.0` in iex?
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
#iex#processes#how-to-question
36 21149 8
New
baxterw3b
Questions & Help>Questions

Anonymous functions with multiple body
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
19 21396 4
New
joeerl
Questions & Help>Questions

Upgrading Elixir - how to check versions, delete, and upgrade?
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
#upgrade
63 40940 13
New
jay1
Questions & Help>Questions

Why isn’t mnesia the most preferred database for use in Elixir/Phoenix?
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
/phoenix#mnesia
123 21186 22
New

Other popular topics Top

9mm
Questions & Help>Questions

Elixir way to conditionally update a map
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
#maps
55 19905 29
New
minhajuddin
Questions & Help>Questions

Using List.first instead of Enum.at(0)
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
#code-style
76 33273 14
New
JakeBecker
Chat & Discussions>Discussions

ElixirLS - the Elixir Language Server
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
#elixir-ls
1140 51847 244
New
pmjoe
Questions & Help>Questions

What do you think of Gleam compared to Elixir?
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
#programminguages#gleam
24 16555 10
New
albydarned
Questions & Help>Questions

Erlang and Elixir on Apple Silicon/M1 Chip
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
#erlang#troubleshooting
121 24599 65
New
ashish173
Questions & Help>Questions

Difference in between :utc_datetime and :naive_datetime in Ecto
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
#ecto#timestamps
79 27907 15
New
malloryerik
Learning Resources>Guides/Tuts/Tips/Info

How to use 'phoenix-heex' templates in VSCode
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
/phoenix#vscode#heex
118 16932 34
New
mgjohns61585
Questions & Help>Questions

Convert a string to an integer?
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
#how-to-question
3 14200 6
New
electic
Questions & Help>Questions

DateTime Format to string?
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
#datetime
19 24687 16
New
magnetic
Learning Resources>Guides/Tuts/Tips/Info

Configuring VSCode to use with Elixir and Phoenix Templates
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
#phoenix_html#vs-code#vscode-elixir#elixir-ls
43 16832 15
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

ElixirCasts

Producing high quality Elixir screencasts since 2017.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement