dmitriid

dmitriid

Transforming Unzip.file_stream error

I’m trying to parse contents of a file in zip file. My idea is to use Unzip to stream contents (the unpacked file might be too large for my taste), and transform it on the fly.

However, something’s weird:

Unzip.file_stream(handle, entry_name) 
|> Stream.map(fn e -> IO.iodata_to_binary(e) end) 
|> Enum.to_list


[
  <<208, 161, 208, 190, 208, 186, 208, 190, 208, 187, 208, 190, 208, 178, 44,
    208, 161, 208, 181, 209, 128, 208, 179, 208, 181, 208, 185, 44, 208, 144,
    208, 189, 208, 176, 209, 130, 208, 190, 208, 187, 209

This works. Replace it with Stream_flat_map or Enum.take(2) or Stream.transform, and

Unzip.file_stream(handle, entry_name) 
|> Stream.transform([], fn e, acc -> {e |> IO.iodata_to_binary, acc} end ) 
|> Stream.run()


** (ErlangError) Erlang error: :data_error
    :zlib.inflateEnd_nif(#Reference<0.60592435.1843265537.68376>)
    (unzip 0.11.0) lib/unzip.ex:188: anonymous fn/1 in Unzip.decompress/2
    (elixir 1.17.0) lib/stream.ex:1039: Stream.do_transform_inner_list/7
    (elixir 1.17.0) lib/stream.ex:1038: Stream.do_transform_inner_list/7
    (elixir 1.17.0) lib/stream.ex:1055: Stream.do_transform_inner_enum/7
    (elixir 1.17.0) lib/stream.ex:690: Stream.run/1
    iex:58: (file)

Something fails somewhere, and I can’t for the life of me figure out where :slight_smile:

Most Liked

akash-akya

akash-akya

there are three elements in the resulting stream

What is the issue? I am not sure, but different number of elements might be because of internal buffer of streaming operations. Either way it is still stream of iodata. If you want the whole file as binary you can use Enum.into(<<>>) instead of Enum.to_list() or better

Unzip.file_stream(handle, entry_name)  
|> Enum.into(<<>>, &IO.iodata_to_binary/1)

Second issue is for different reason. Unzip checks CRC at the end of file streaming to ensure data integrity, so when you terminate prematurely without reading completely by calling Enum.take(2) it fails because it cannot verify CRC.
For the same reason Enum.take(6) succeeds because there are only 3 chunks and CRC checks out. I explained bit more details here.

I am thinking of adding a flag to skip checking CRC to support cases like this. I haven’t added it sofar since noone requested it, let me know if you need.

dmitriid

dmitriid

Welp. It works as a charm :slight_smile: All I needed to do was just to process the entire stream :slight_smile:

Thank you for helping out and rubber-ducking!

Where Next?

Popular in Questions Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New

Other popular topics Top

sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
977 41022 311
New
sergio
I couldn’t find any guides that worked well with Phoenix 1.6.0 and esbuild. I hope this helps people test the waters and eases you into t...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New

We're in Beta

About us Mission Statement