pm100

pm100

Is there a 'next item' call for Stream?

I want to read a file line by line, but I want to pull lines on demand.
I have a GenServer that tokenizes a file. It is called using a call that says ‘give me the next token’. It works but at the moment it reads the whole file into memory at once, doesnt work for big files.

I would like to have the current line in GS state and when I finish it read the next line into state. But I cant see the Stream call needed.

Most Liked

LostKobrakai

LostKobrakai

There’s nothing like next item on Stream, because to elixir it’s all Enumerable data. Elixir just knows that it can reduce over those. There’s no stopping in the middle and continueing later for that model – at least none that you can expose, rather than being an implementation detail.

{item_a, rest} = Stream.next(file_stream)
{item_b, _} = Stream.next(rest)
{item_c, _} = Stream.next(rest)

Given how file streams work – using a file cursor – item_b and item_c would be different, even though the Stream.next function is passed the same value. That’s not great for a functional’ish api.

Instead for your usecase you could use the underlying pieces used to stream lines with File.stream!: File.open/2, IO.read/2 and File.close/1 directly.

krasenyp

krasenyp

Very big tangent, but with Iter.next/1 one can implement a very cool pattern for lazy pagination in LiveView, or elsewhere. Having a resource stream, one can do source = stream |> Iter.from() |> Iter.chunk_every(@page_size), and whenever a new page of items is needed, do {:ok, items, source_rest} = Iter.next(source).

LostKobrakai

LostKobrakai

Iter seems to start a two additional processes for the Enumerable implementation though and using the process mailbox as a temp. storage: iterex/lib/iter/iterable/enumerable.ex at main · ash-project/iterex · GitHub

That’s quite a blunt approach for dealing with that complexity.

krasenyp

krasenyp

You don’t need to use a stream. Regularly opened files are IO devices, themselves processes, and there is an Erlang function which would do exactly what you want - :file.read_line/1. So, you need to open a file, using File.open or File.open!, and keep the result in your process’ state. Then just call the Erlang function I mentioned to read a line.

If you want to learn more on how IO devices work, check out The Erlang I/O Protocol.

nulltree

nulltree

Relatedly you could take a look at Ash’s iterex v0.1.2 — Documentation.

Iterators provide the flexibility of Enum with the laziness of Stream and the ability to pause and resume iteration.

Can’t make a statement about the correctness, stability etc. of the library myself, had no real use case so far.

Where Next?

Popular in Questions Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
Codball
Mix format works fine if run from the cmd. I’ve followed this to facilitate the implementation into VSC which involves downloading an ext...
New

Other popular topics Top

William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement