Questions & Help>Questions

Memory leak with quantum and processing binary files

#performance#quantum#troubleshooting#memory
dfens

dfens

Memory leak with quantum and processing binary files

Currently I have phoenix app which also schedules quantum jobs (GitHub - quantum-elixir/quantum-core: ⌚ Cron-like job scheduler for Elixir) with overlap: false mode
Jobs are importing & exporting relatively large json files.

Now my problem is

Screenshot from 2020-03-06 16-29-16
Screenshot from 2020-03-06 16-29-16507×130 7.94 KB

Leaking memory

This is current status and it will get worse since different/bigger data sets are going to be processed.

For example on start on the left app starts with ~60MB

Screenshot from 2020-03-06 16-34-47
Screenshot from 2020-03-06 16-34-47613×255 11.5 KB

Flat line in the end is after quantum job was processed

I did check also this memory management - Solving large binaries leak - Stack Overflow but forced gc / tuning ERL_FULLSWEEP_AFTER collect did not seem to help

If I run ImportTask/ExportTask on my local machine but not as quantum and observe memory - then I cannot reproduce this behaviour

Any tips welcome :slight_smile:

7 comments
#performance#quantum#troubleshooting#memory
6 1414 7

Marked As Solved

NobbZ

NobbZ

Have you tried strings: :copy (assuming you use Jason)?

4
Post #2

Also Liked

dfens

dfens

@NobbZ jackpot!
as the matter of fact I was using Jaxon for parsing since files were pretty big before. but just tested with Jason strings: :copy and issue does not occur anymore. I suspect this was misuse of Jaxon.stream on my side but I am completely satisfied with Jason results

@benwilson512 yes I did try, without any effect

Thank you so much for your input guys

1
Post #4
NobbZ

NobbZ

From a quick glance it seems as if jaxon wasn’t copying and also did not provide an option to copy binaries.

You might still be able to use jaxon when you :binary.copy/1 strings it gives you manually.

1
Post #5

Where Next?

View thread on forum (has 7 responses!)
performance
quantum
troubleshooting
memory
Home Questions & Help>Questions
#performance#quantum#troubleshooting#memory
6 1415 7
Last post

Popular in Questions Top

senggen
Questions & Help>Questions

How to fix *Bad argument in call to erlang:'++'(<<"xxx/crash.log">>, ".3") in lager_rotator_default:rotate_logfile/2 line 84*
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
#production#error#log
2 46314 2
New
JorisKok
Questions & Help>Questions

How is it possible to get 2 million websocket connections when you have 65536 available ports?
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
/phoenix
20 17308 4
New
openscript
Questions & Help>Questions

How to set up the Elixir SDK in Intellij IDEA with the intellij-elixir plugin?
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
17 22671 9
New
ycv005
Questions & Help>Questions

BEAM file compliation error after degrading Erlang version from 22 to 21
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
/phoenix#erlang
33 15699 4
New
Harrisonl
Questions & Help>Questions

Deploying Elixir into ECS causing many "'global' at node :"xxxxx@10.0.X.X" requested disconnect from node :"xxxx@10.0.X.X" in order to prevent overlapping partitions"
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
#deployment
2 85219 3
New
quazar
Questions & Help>Questions

How to encode ecto schema with Jason with all fields
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
/phoenix#json#jason
17 13698 9
New
hariharasudhan94
Questions & Help>Questions

What is the best IDE for elixir?
I would like to know what is the best IDE for elixir development?
#code-editors
33 49503 23
New
minhajuddin
Questions & Help>Questions

Using List.first instead of Enum.at(0)
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
#code-style
76 33273 14
New
electic
Questions & Help>Questions

DateTime Format to string?
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
#datetime
19 24687 16
New
jay1
Questions & Help>Questions

Why isn’t mnesia the most preferred database for use in Elixir/Phoenix?
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
/phoenix#mnesia
123 21186 22
New

Other popular topics Top

dotdotdotPaul
Questions & Help>Questions

Ecto: Validating belongs_to association is not nil?
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
#ecto
29 17806 25
New
vonH
Questions & Help>Questions

Why would I choose Elixir as a general purpose programming language?
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
#functional-programming#use-cases
65 34758 13
New
AstonJ
Learning Resources>Courses

Elixir for Programmers (PragDave)
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
#elixir-for-programmers-course
201 29468 78
New
_russellb
Questions & Help>Questions

Web scraping tools
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
#web-scraping
123 19127 45
New
malloryerik
Learning Resources>Guides/Tuts/Tips/Info

How to use 'phoenix-heex' templates in VSCode
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
/phoenix#vscode#heex
118 16932 34
New
chrismccord
News>Phoenix News

Phoenix v1.5.0 released!
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
/phoenix#phoenix-release
379 28447 94
New
freewebwithme
Questions & Help>Questions

Failed to run 'elixir' command error in vs code
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
#vscode#elixir-ls
48 15871 39
New
aadeshere1
Questions & Help>Questions

Write while loop equivalent in elixir
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
#while#loop
60 28200 36
New
belgoros
Questions & Help>Questions

Regex question for hyphen match
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
#learning-elixir
9 12939 10
New
skosch
Questions & Help>Questions

Put/update deep inside nested maps (and auto-create intermediate keys)
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
#data-structures#maps#immutability
52 20047 11
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

Manning

Develop your skills with books, videos, and courses.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement