A bunch of useful links:

Multimedia in general

• Codecs comparison
• How does JPEG work?
• How does H261 work?

Membrane

• Membrane Guide
• Membrane Demos

WebRTC

• WebRTC API
• WebRTC W3C Standard
• WebRTC Examples in JS
• WebRTC Examples in JS src code
• WebRTC basics
• WebRTC signaling
• WebRTC signaling 2
• WebRTC tutorial
• webrtcHacks (mentioned by @mat-hek)

Agreed

Playing with media is not as simple as you’d expect, unfortunately

We’re building WebRTC SFU server, that handles both audio and video. We made it work, but It’s still experimental and extensively developed.

That’s mostly true, though Elixir is very convenient for handling protocols and containers too, because of its good support for binaries and bitstrings. Heavy, numerical computations are indeed delegated to low-level, mostly C libraries, but since it’s done via simple NIFs or C nodes, it doesn’t involve

If you deal with latencies around 2ms, then it may be a problem - haven’t tested. But for usual media streaming it’s good enough. People even write streaming apps in Go, that has stop-the-world GC - that one happens to be problematic though, AFAIK.

We didn’t have time for big optimizations of Membrane yet, neither we used it with OTP 24 JIT. Anyway, it’ll probably never be as fast as if we used Rust or C. Membrane focuses rather on reliability, scalability and maintainability.

My understanding of such project is: elixir/erlang is a great fit of “orchestration”… connecting people, setting the pipeline of encoding/decoding/streaming via web/etc. But underneath those encoder/decoder are still delegated to some low level libs.

Though, please note, that I’m not an expert in the the subject and all my experience for realtime audio/video chat is limited by one project where it’s done via WebRTC and Elixir/Phoenix server was just authorizing and connection people.

I think you might be interested in Membrane Framework. Looks like they provide with plugins to use audio codecs and there is a repo with some demos.

Not sure there’s any perfectly justified reason but at least my thinking would be that the BEAM has GC pauses (albeit very short and are per-process which should limit their impact) that can affect latency – which would be a deal-breaker in audio streaming. And as @RudManusachi said, number crunching isn’t the BEAM’s strongest suit so future scaling might suffer.

If you expect no more than 20 people streaming audio at any given time and have done due measurements then Elixir might be a perfect choice. But there are multiple stories I’ve read in blogs about a lot of binaries moved around which eventually made the BEAM’s GC slower.

But from my side I’d opt for an Elixir orchestrator and a Rust implementation beneath just to be sure if the app scales it will still work without lagging.

All that being said, maybe it pays off to ping the creators of Membrane? → Membrane - a new framework for multimedia processing

@mat-hek : Thanks for jumping in and answering all my questions. I feel like I’m finally starting to see the basic building blocks involved.

@dimitarvp : Thanks for mentioning the membrane project. This has turned out to be quite useful.