Consuming a collection (both finite and infinite ones) one-by-one is definitely useful. I have had conversations with @josevalim about this before. The reason Enum (and, by extension, Stream) do not support this by default is twofold:

1. They expose their iteration process as a fold (which is the academic term for Enumerable.reduce), which always consumes the complete collection. For certain kinds of Enumerables, it is impossible or impractical to take elements out one by one. An example are file streams: iterating them one-by-one means that you have to keep the file open for the whole time you take elements out one-by-one.
2. Basing Enumerable on top of a ‘take one per time and keep the collection intact’ interface would be much slower (if I recall correctly, this was tried and benchmarked in some pre-stable version of Elixir. From the top of my head, I believe it was 4-5x slower, which is very significant since so much time in Elixir is spent working with/on collections).

Nevertheless, there are cases like in your use-case where this is very useful. You mentioned the iter repository in your post. That is an old attempt at fleshing out what later became the Extractable library/protocol.
Extractable only comes with out-of-the-box implementations for Lists, Maps and MapSets.
I am fairly certain that you could create a custom struct for your puzzle generating logic that contains a function which returns a {puzzle, new_struct_with_function_for_next_puzzle}.

There is one additional reason why streams can’t be consumed one-by-one and instead need to be folded (Enumerable.reduced) at once: the stream generator may have side-effects (example: consuming File.stream! advances the file position) and therefore must be invoked at most once.

Try this:

{e1, cont1} = MyStream.next(File.stream!("test.txt", [], 1))
{e2, cont2} = MyStream.next(cont1)
{e2a, cont2a} = MyStream.next(cont1)
IO.puts("#{e1} #{e2} #{e2a}")

Assuming test.txt contains 12345, one would expect to see 122 but instead you get 123.

The StreamSplit library suffers from the same problem:

iex(1)> {head, tail} = StreamSplit.take_and_drop(File.stream!("test.txt", [], 1), 1)
{["1"], #Function<55.126435914/2 in Stream.resource/3>}
iex(2)> Enum.take(tail, 2)
["2", "3"]
iex(3)> Enum.take(tail, 2)
["4", "5"]

The documentation for Enumerable repeatedly mentions that “In case a reducer/0 function returns the :suspend accumulator, the :suspended tuple must be explicitly handled by the caller and never leak.” And that’s exactly what your code does — the continuation leaks and can then be called multiple times, messing up the side effects.

If one comes from a Haskell background (like me), it really is somewhat suprising. Stream resembles a lazily evaluated linked list, but the BEAM VM doesn’t have lazy evaluation and thunks, so the side-effectful Stream must be treated with care.

Here’s my stab at a safe-ish stream iterator:

defmodule StreamStepper do
  def stream_stepper(stream) do
    stream
    |> Enumerable.reduce({:cont, nil}, &stream_stepper_suspender/2)
    |> stream_stepper_continuer()
  end

  defp stream_stepper_suspender(head, nil) do
    {:suspend, {head}}
  end

  defp stream_stepper_continuer({done_halted, nil}) when done_halted in [:done, :halted] do
    []
  end

  defp stream_stepper_continuer({done_halted, {head}}) when done_halted in [:done, :halted] do
    tail = fn -> [] end
    [head | tail]
  end

  defp stream_stepper_continuer({:suspended, {head}, tail_cont}) do
    once = callable_once()
    tail = fn -> once.(fn -> tail_cont.({:cont, nil}) |> stream_stepper_continuer() end) end
    [head | tail]
  end

  defp callable_once do
    seen = :atomics.new(1, [])

    fn fun ->
      case :atomics.compare_exchange(seen, 1, 0, 1) do
        :ok -> fun.()
        _ -> raise "protected fun evaluated twice!"
      end
    end
  end
end

It’s just a hack, the continuation kind of leaks as well, but this leak is protected by callable_once which raises whenever the possibly side-effecting fun is invoked more than once.

Personally, I would write a small abstraction around puzzle sequence, with the following operations:

• PuzzleSequence.new(seed)
• PuzzleSequence.next_puzzle(sequence)

From there, I’d write the rest of the system and see how it unfolds. If I notice that there is some significant overlap between what I wrote and Stream/Enum functions, I’d try to see if I can somehow reuse the existing enum logic, e.g. by doing some trickery proposed here. But before that, I’d just stick with this simple interface.

Yep, if I was doing it for work, I wouldn’t think twice about it and went with PuzzleSequence.new(seed) and PuzzleSequence.next_puzzle(sequence). But I am doing it mostly for fun and learning. I am totally aware I am over-engineering but just from those discussions I understood Streams and Enumerables better and learned about :atomics I feel that was worth it

Wow! This callable_once is brilliant! I haven’t heard about :atomics before.

And now I understand better why passing a continuation around might result in shooting myself in the foot

I find this funny: my initial idea for the stream of puzzles was precisely about two processes iterating over the same stream at different speeds

In my case, I could create two identical streams, and that should work.

For infinite streams without side effects like the stream of puzzles or Stream.cycle([1,2,3]) setting atomics references seems wasteful.

BUT, if we would like to introduce Strem.next or Stream.step to standard API, we would have to let the Enumerable decide, if it uses side effects and needs protection or if it is safe to call the same continuation multiple times.

That would require adding protections as callable_once to all side effectful streams in the stdlib and changes in the API of generators like Strem.resource or Strem.iterate to enable such protections. Well, even Stream.map could have side effects that we might want to control.

Suddenly the amount of work to implement it becomes more significant. Next question arises: is it worth it?

The amount of libraries and even GenStage example suggests there is a need for something like that while it is very easy to do wrong.
On the other hand, that would complicate the Stream API and add another thing to think about when implementing Enumberable protocol.
From those two, I think complicating the Stream API is a bigger issue. It is elegant and clean, so adding opts like protect: true seems like a bad idea to me. And of course, allowing continuations to leak is even worse.

Alternatively, we could keep protections for all streams as in your solution, and never allow calling the same continuation twice even when it is safe. That would require information that it might be less efficient than regular processing. Those :atomics instructions should be fast though.

Well, at least I now know why the stepping API is hard to implement

I still don’t want to give up on the idea of my PuzzleStream being a proper Stream, but maybe instead of stepping over it, I’ll implement the game logic inside Stream.transform/3

Thank you all for the discussion! It was enlightening!