frumos

frumos

Simple way to stream by line from compressed .gz file

Hello,

I have a text file which has lines with new line separation “\n”

I need to stream this file by lines so I can apply my own functions per line and I am doing like this:

iex(30)> "file.txt" |> File.stream!() |> Stream.map(&(my_whatever_function_here(&1))) |> Enum.take(50)

Now the same file becomes gzip compressed data and I would like to stream it by lines and get the same result as from above code.

I am doing it like:

iex(38)> "file.gz" |> File.stream!([], 2024*2024) |> StreamGzip.gunzip() |> WHAT_I_NEED_TO_PUT_HERE |> Stream.map(&(my_whatever_function_here(&1))) |> Enum.take(50) 

and question is what function(s) i need to put instead of “WHAT_I_NEED_TO_PUT_HERE” placeholder ?

I tried lots of things , splitting by “\n” applying flat_map but I can not get result that I am getting new line after StreamGzip.gunzip()

Seems this simple requirement is not trivial to implement.

Thank you for your time.

Most Liked

frumos

frumos

One of my colleague suggest this:

"file.gz" |> File.stream!([:compressed]) |> Stream.map(&(my_whatever_function_here(&1))) |> Enum.take(50)

I tried with that test file and it seems it works out of the box. Need to do more testing but if it is that simple I can not believe, I struggled so much with gz files in Elixir in past …

Where Next?

Popular in Questions Top

Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
gonzofish
I’m currently trying to understand how to join three tables using Ecto. All the examples I’ve seen use 2, so maybe I’m just missing somet...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New

Other popular topics Top

chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
977 41022 311
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New

We're in Beta

About us Mission Statement