Hello!
I have an idea for an Elixir library I’d like to work on, but wanted to get some thoughts from the community first. It would allow you to define data pipelines declaratively. Here’s an example of what I’m imagining the API would look like:

defmodule MyPipeline do
  # Library name tbd
  use Pipeline

  pipeline do

    # This strategy would generate a GenStage pipeline that runs steps in the same stage concurrently
    strategy Pipeline.GenStage

    # This strategy would simply run every step on after the other in the same process.
    # strategy Pipeline.SingleProcess

    stage do
      step Socks, color: :blue
      step Underwear, style: :boxers
      step Shirt
    end

    stage do
      step Pants
      step Tie
    end

    stage do
      step Shoes
      step Belt
    end

    step Jacket
  end

  # The pipeline above would generate a data flow that looks something like the following.
  # All the steps in the same stage will be executed in parallel.
  # A stage will start as soon as all the steps in the previous stage have been ran.
  # Stage 1
  # ┌─────┐┌─────────┐┌─────┐
  # │socks││underwear││shirt│
  # └─────┘└─────────┘└─────┘
  # Stage 2
  # ┌──────────┐┌───────────┐
  # │pants     ││tie        │
  # └──────────┘└───────────┘
  # Stage 3
  # ┌───────────┐┌──────────┐
  # │shoes      ││belt      │
  # └───────────┘└──────────┘
  # Stage 4
  # ┌───────────────────────┐
  # │jacket                 │
  # └───────────────────────┘
end

Each step would look like the following, and could probably also be just a simple function instead of a module:

defmodule Step.Pants do
  @behaviour Pipeline.Step

  @impl Pipeline.Step
  def run(env, _opts) do
    IO.puts("Putting on my pants!")

    updated = Pipeline.Step.put_info(env, pants: :done)

    {:ok, updated}
  end
end

Anyways, I just wanted to get some thoughts before I start working on this. I was unable to find anything that already exists that does this. Broadway and Flow are probably the most similar, and I can even see this using Flow under the hood for the strategy Pipeline.GenStage stuff.

I don’t want to work on it if it has any major flaws that anyone can foresee, or if there’s just a better way to do this already.

Thanks in advance!