Added ability to stream input and output. Now one can do something like this

def audio_stream!(stream) do
  # read from stdin and write to stdout
  proc_stream = ExCmd.stream!("ffmpeg", ~w(-i - -f mp3 -))

  Task.async(fn ->
    Stream.into(stream, proc_stream)
    |> Stream.run()
  end)

  proc_stream
end

File.stream!("music_video.mkv", [], 65535)
|> audio_stream!()
|> Stream.into(File.stream!("music.mp3"))
|> Stream.run()

Along with this there are many changes related to interface and error handling. Please check documentation for more details

v0.1.0

Github

v0.3.0

This is a major release with completely different approach to solve the back-pressure. Ditched named pipes in favor of slightly complicated protocol. And with that ExCmd no longer has scheduler issuers.

I did consider this approach before, but dropped after I hit a blocker. Thanks to @ananthakumaran for clearing it up

Changes

• fixed beam scheduler issues
• demand-driven protocol for back-pressure instead of named pipes
• few breaking changes to options
• many internal changes for maintainability. Such as, ExCmd.Process now uses GenSteteMachine instead of GenServer

The main issue which ExCmd tries to solve, which none of these libraries solve is having proper stream with back-pressure. In these libraries progress of the external command is uncontrolled.

Internally they all use the port and some sort middleware program for IO. We can never have back-pressure (or limit) using ports as the process mailbox is unbounded. ExCmd also uses port and a middleware program (odu) but it only uses port for controlling the external program not for IO. For IO it uses named pipe (FIFO) which is demand-driven.

One can just write the output to a file and read that from to “solve” this. But,

• This involves disk IO for writing and reading which add latency
• We can not control external program speed
• we have to cleanup these files properly which might not be trivial
• and IMHO ergonomics is better with ExCmd approach as the stream is more composable

Apart from these issues,

1. erlexec: It is more focused on orchestrating and linking the external command than on communicating with it. If erlexec accepts os pid for its functionality maybe we can use erlexec with ExCmd
2. porcelain: There are few important issues like zombie process and not having the ability to forcefully kill
3. rambo: similar to porcelain, but rambo does not allow streaming input to stdin so all input must be kept in memory or it has to be passed to the command by writing to a file. It collects output binary in memory by appending it which is not efficient

I think when these libraries were created, they were not trying to solve the issues I mentioned, so I think its not correct to compare them. Please let me if anything is incorrect.

Thanks a lot for the explanation! Makes sense.

Options like :stdin_only, :stdout_only and :stdin_and_stdout (the default) would help a lot. Boilerplate should be hidden away behind options and/or convenience functions like stream_stdin!, stream_stdout! and stream! (which expects both as it is right now).

Not sure about the names, they might not be good.

As for odu itself, it introduces a needlessly complex external dependency installation that some programmers might not be willing to subject themselves to. I’d suggest you write an in-app small Rust library. I can help you integrate Elixir with Rust – the release candidate of Rustler 0.22 has a much nicer and shorter syntax compared to previous versions and is now a joy to use.

I am also interested in your opinion on:

• https://github.com/saleyn/erlexec (Erlang)
• https://github.com/ericentin/exexec (Elixir wrapper around erlexec)
• https://github.com/alco/porcelain (Elixir)
• Rambo: Run your command. Send EOF. Get output (Elixir)