CharlesO
Parse / Pattern match large binary data
I have read through this thread: binary-pattern-matching-when-input-is-a-stream on a related subject, but i’m not able to make any headway.
I get this error trying to load and parse a large binary directly:
iex> JQL.save "--1566364097905", "STMT.ENTRY"
eheap_alloc: Cannot allocate 18446744071692551144 bytes of memory (of type "heap").
Crash dump is being written to: erl_crash.dump...
I’m doing an ETL (extract - transform - load) for a reporting project, moving data from a proprietary platform into SQL Server for easier reporting.
The parser has worked fine until I hit this size issue.
Please how / can we use streams to handle these kind of situations?
(particularly when initial or existing parser had not been built with streaming in mind)
JQL Parser: https://gist.github.com/CharlesOkwuagwu/4c6c89d96db7876bc0d27fecd518340e (updated)
Sample large file (~850mb unzipped): https://paperlesssolutionsltd.com.ng/java/--1566364097905.7z
Thanks.
Marked As Solved
CharlesO
Found a really simple way to solve this. Just read the data-header, then parse row by row:
def read(src, cnt \\ nil) do
{:ok, fid} = :file.open("_cache/#{src}", [:raw, :read, :binary])
{:h, start, n} = _p(fid, :h)
cnt = cnt || n
Logger.debug("Reading: #{cnt} of #{n} rows")
Process.put(:start_pos, start)
for i <- 1..cnt do
{:r, start, row} = _p(fid, Process.get(:start_pos), :r)
Process.put(:start_pos, start)
Logger.debug(inspect({i, row}, @format))
end
:file.close(fid)
end
# READER
defp _p(fid, :h) do
# 000007775B4C4900000136490000000049FFFFFFFF4C490000006749000000004900000005
{:ok, <<n::32>>} = :file.pread(fid, 33, 4)
{:h, 37, n}
end
defp _p(fid, start, :r) do
# 490000000049000000006A62 00000165 3136333737303....
{:ok, <<l::32>>} = :file.pread(fid, start + 12, 4)
{:ok, b} = :file.pread(fid, start + 16, l)
{:r, start + l + 18, _split(b)}
end
defp _split(v) do
b = for(<<c <- v>>, c in 32..126 || c in [252, 253, 254], into: "", do: <<c>>)
b = :binary.replace(b, <<253>>, ";", [:global])
b = :binary.replace(b, <<252>>, "^", [:global])
:binary.split(b, <<254>>, [:global])
end
3
Also Liked
NobbZ
But the error message you posted, it claims that it wants to allocate another 18446744071692551144 byte, which is 18014398507512256 kiB, 17592186042492 MiB, 17179869182 GiB, 16777215 TiB, 16383 PiB, so ~16 EiB…
As you can see @hauleth even missed by a factor of ~100…
There might be plenty of reasons why such a huge amount of memory might be allocated…
4
hauleth
No surprise there as I assume that you do not have 163 PB of RAM available for your machine. It seems like you want to load a lot of data at once. Maybe try to split it into reasonable chunks?
3
NobbZ
Does the problem persist if you remove this line?
You are not even using the result of term_to_binary… And unless compressed, it will always require more memory than the input.
3
benwilson512
Author of Craft GraphQL APIs in Elixir with Absinthe
CharlesO
That line was just for testing something else. removing it still brings allocation errors, but smaller:
iex> JQL.save "--1566364097905", "STMT.ENTRY"
eheap_alloc: Cannot allocate 1898305688 bytes of memory (of type "heap").
Crash dump is being written to: erl_crash.dump...
def save(src, name) do
{:ok, <<len::32, "[", _::bits>> = bin} = :file.read_file("_cache/#{src}")
{:ok, f} = :file.open("_cache/#{name}.bin", [:raw, :binary, :write, :delayed_write])
m = _r(binary_part(bin, 5, len - 2), [])
# b = :erlang.term_to_binary(m)
:done
end
i’m able to read the file, but not parse it.
This is a smaller sample of using the parser to handle just a few kilobytes:
def decode(hex), do: _r(Base.decode16!(hex))
iex> JQL.decode "000007775B4C4900000136490000000049FFFFFFFF4C490000006749000000004900000005490000000049000000006A62000001653136333737303030313335323832312E303130303031FE3136333737303030313335323832312E303130303031FE3030303030303030303139373339FE4E4730303130303031FE333439312E3030FE323133FE414C4C4F434154494F4EFE414C4C4F434154494F4EFEFE31FE31FE35303032FE3230313231313031FEFE465431323330363030303636FEFE3230313231313031FE31FE33FE465431323330363030303636FE4654FE3230313231313031FE3136333737303030313335323832312E3031FD312D32FEFEFE31FE31323131303131343430FE4E474EFE333439312E3030FEFE4445424954FEFE3230313231313031FE3230313231313031FEFEFEFEFEFEFEFEFEFEFEFEFEFE41432E312E54522E4E474E2E353030322E312E353032302E343930302E4E472E4E472E313030302D312E312E2E2E2E2E4E4730303130303031FEFE44454641554C54FE4E4730303130303031FEFEFE3230313231313031FE31FEFE3B3B490000000049000000006A620000014D3136393233303030303936333935342E303130303032FE3136393233303030303936333935342E303130303032FE4E474E31313236303030303130303031FE4E4730303130303031FE2D3439343534392E3031FE323133FEFEFEFEFE31FE3131323630FE3230313430343239FEFE465431343131393030303036FEFE3230313430343239FE31FE37FE465431343131393030303036FE4654FE3230313430343239FE3136393233303030303936333935342E3031FD312D32FEFEFE31FE31343035303131373435FE4E474EFE2D3439343534392E3031FEFE4445424954FEFE3230313430343239FE3230313430343239FEFEFEFEFEFEFEFEFEFEFEFEFEFE41432E312E54522E4E474E2E31313236302E372E2E2E2E2E313030302D312E2E2E2E2E2E4E4730303130303031FEFE44454641554C54FE4E4730303130303031FEFEFE3230313430343239FE31FEFE3B3B490000000049000000006A62000001783136393834303030303534313938352E303230303032FE3136393834303030303534313938352E303230303032FE3030303030303030303231303234FE4E4730303130303031FE2D36393032322E3636FE323133FE434F4C4C454354494F4EFE434F4C4C454354494F4EFEFE31303030FE31FE35303033FE3230313430363330FEFE465431343138323030313432FEFE3230313430373031FE31FE33FE465431343138323030313432FE4654FE3230313430373031FE3136393834303030303534313938352E3032FD312D32FEFEFE31FE31343037303131313339FE4E474EFE2D36393032322E3636FEFE435245444954FEFE3230313430363330FE3230313430363330FEFEFEFEFEFEFEFEFE3230313430373031FEFEFEFEFE41432E312E54522E4E474E2E353030332E372E353032302E343930302E4E472E4E472E313030302D312E313030302E2E2E2E2E4E4730303130303031FEFE44454641554C54FE4E4730303130303031FEFEFE3230313430373031FE31FEFE3B3B490000000049000000006A620000015C3136393834303030303634363137372E303030303031FE3136393834303030303634363137372E303030303031FE3030303030303030303139383434FE4E4730303130303032FE3230353038302E3838FE323133FEFEFEFE32303030FE31FE31323036FE3230313430373031FEFE465431343138323030333438FEFE3230313430373031FE31FE37FE465431343138323030333438FE4654FE3230313430373031FE3136393834303030303634363137372E3030FD31FD31FEFEFE31FE31343037303131323439FE4E474EFE3230353038302E3838FEFE435245444954FEFE3230313430373031FE3230313430373031FEFEFEFEFEFEFEFEFEFEFEFEFEFE41432E312E54522E4E474E2E313230362E372E353032302E343930302E4E472E4E472E323030302D312E323030302E2E2E2E2E4E4730303130303032FEFE44454641554C54FE4E4730303130303032FEFEFE3230313430373031FE31FEFE3B3B490000000049000000006A62000001733136393834303030323435323030362E303230303031FE3136393834303030323435323030362E303230303031FE3030303030303030303139393431FE4E4730303130303031FE3130383336382E3030FE323133FE434C45415245442044455441494C53FE434C45415245442044455441494C53FEFE31303030FE31FE31303130FE3230313430373031FEFE465431343138323030343633FEFE3230313430373031FE31FE33FE465431343138323030343633FE4654FE3230313430373031FE3136393834303030323435323030362E3032FD312D32FEFEFE31FE31343037303131343236FE4E474EFE3130383336382E3030FEFE435245444954FEFE3230313430373031FE3230313430373031FEFEFEFEFEFEFEFEFEFEFEFEFEFE41432E312E54522E4E474E2E313031302E372E353032302E343930302E4E472E4E472E313030302D312E313030302E2E2E2E2E4E4730303130303031FEFEFE4E4730303130303031FEFEFE3230313430373031FE31FEFE3B3B3B3B5D"
iex> {:data, 5,
[
["163770001352821.010001", "163770001352821.010001", "00000000019739", "NG0010001", "3491.00", "213", "ALLOCATION", "ALLOCATION", "", "1", "1", "5002", "20121101", "", "FT1230600066", "", "20121101", "1",
"3", "FT1230600066", "FT", "20121101", "163770001352821.01;1-2", "", "", "1", "1211011440", "NGN", "3491.00", "", "DEBIT", "", "20121101", "20121101", "", "", "", "", "", "", "", "", "", "", "", "", "",
"AC.1.TR.NGN.5002.1.5020.4900.NG.NG.1000-1.1.....NG0010001", "", "DEFAULT", "NG0010001", "", "", "20121101", "1", "", ""],
["169230000963954.010002", "169230000963954.010002", "NGN1126000010001", "NG0010001", "-494549.01", "213", "", "", "", "", "1", "11260", "20140429", "", "FT1411900006", "", "20140429", "1", "7",
"FT1411900006", "FT", "20140429", "169230000963954.01;1-2", "", "", "1", "1405011745", "NGN", "-494549.01", "", "DEBIT", "", "20140429", "20140429", "", "", "", "", "", "", "", "", "", "", "", "", "",
"AC.1.TR.NGN.11260.7.....1000-1......NG0010001", "", "DEFAULT", "NG0010001", "", "", "20140429", "1", "", ""],
["169840000541985.020002", "169840000541985.020002", "00000000021024", "NG0010001", "-69022.66", "213", "COLLECTION", "COLLECTION", "", "1000", "1", "5003", "20140630", "", "FT1418200142", "", "20140701",
"1", "3", "FT1418200142", "FT", "20140701", "169840000541985.02;1-2", "", "", "1", "1407011139", "NGN", "-69022.66", "", "CREDIT", "", "20140630", "20140630", "", "", "", "", "", "", "", "", "20140701", "",
"", "", "", "AC.1.TR.NGN.5003.7.5020.4900.NG.NG.1000-1.1000.....NG0010001", "", "DEFAULT", "NG0010001", "", "", "20140701", "1", "", ""],
["169840000646177.000001", "169840000646177.000001", "00000000019844", "NG0010002", "205080.88", "213", "", "", "", "2000", "1", "1206", "20140701", "", "FT1418200348", "", "20140701", "1", "7",
"FT1418200348", "FT", "20140701", "169840000646177.00;1;1", "", "", "1", "1407011249", "NGN", "205080.88", "", "CREDIT", "", "20140701", "20140701", "", "", "", "", "", "", "", "", "", "", "", "", "",
"AC.1.TR.NGN.1206.7.5020.4900.NG.NG.2000-1.2000.....NG0010002", "", "DEFAULT", "NG0010002", "", "", "20140701", "1", "", ""],
["169840002452006.020001", "169840002452006.020001", "00000000019941", "NG0010001", "108368.00", "213", "CLEARED DETAILS", "CLEARED DETAILS", "", "1000", "1", "1010", "20140701", "", "FT1418200463", "",
"20140701", "1", "3", "FT1418200463", "FT", "20140701", "169840002452006.02;1-2", "", "", "1", "1407011426", "NGN", "108368.00", "", "CREDIT", "", "20140701", "20140701", "", "", "", "", "", "", "", "", "",
"", "", "", "", "AC.1.TR.NGN.1010.7.5020.4900.NG.NG.1000-1.1000.....NG0010001", "", "", "NG0010001", "", "", "20140701", "1", "", ""]
]}
iex>
1
Popular in Questions
Hi! May someone helps me, please!
I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
In Ruby, I can go:
User.find_by(email: "foobar@email.com").update(email: "hello@email.com")
How can I do something similar in Elixir? ...
New
Hello all!
I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
Hi,
I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
Using vs code and installed ElixirLS: support and debugger.
And I got an error popped up on start up says
Failed to run ‘elixir’ comma...
New
Hey all, just started picking up Elixir last week and am writing a scraper as a learning project.
Baby step #1 is extracting the number ...
New
I have a list say
x = ["23gh", "56kh", "97mh"]
I would like to pass each element to Val in each iteration.
Say, in iteration 1 -------...
New
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
Hi all,
I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage.
I'm trying to use Postg...
New
Other popular topics
Good day to you all.
I have been struggling to get a query involving like and ilike to work.
Can anyone assist me on this, please?
pro...
New
I couldn’t find any guides that worked well with Phoenix 1.6.0 and esbuild. I hope this helps people test the waters and eases you into t...
New
I tried installing
elixir 1.11.2
erlang 23.3.4
via asdf in my zsh shell. Enabled the versions locally and globally.
When I list them ...
New
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum.
...
New
Hi,
I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors:
[WARN] - (starship::utils): Executing command ...
New
Hi everyone!
I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"