tomekowal
What Kafka client do you recommend?
We’d like to start using RedPanda at work and I am hunting for a Kafka client library.
There are three options I’ve seen so far:
- kafka_ex. It looks like it is in a transition period between 0.x to 1.x. There is
KafkaEx.New.KafkaExAPI. - brod. Made by Klarna, looks stable
- kaffee. A wrapper around brod that in the README claims to be experimental.
I don’t mind using experimental or unstable code. I could potentially contribute but is there someone who used at least two of them and can compare? Do they differ in architecture? Is one of them easier to use? Do they make different trade-offs?
While searching the forum, I saw @keathley shared a lot about Kafka usage. Thanks for the slides from your presentation about it! Is it available somewhere as a video? I hope you don’t mind the ping 
Marked As Solved
keathley
I prefer to use Brod, but Brod is kinda weird. I ended up writing my own wrapper at B/R that never got open sourced. You want to avoid a lot of the wrappers that use GenStage because they don’t handle partitions correctly and will allow messages to be processed out of order. Obviously, this only matters if you rely on kafka’s partition ordering (we did). But overall Brod will do what you need and have the fewest surprises. Build your own wrapper once you know what features or improvements you need for your use case.
Also Liked
lud
I really need only consuming and I want the minimum amount of boilerplate. That’s going to be my main criteria.
I don’t have the code anymore so I’ll only from my bad memory but I have used spreedly/kaffe for a year and it was a solid basis to setup the project.
For consuming, if you need to customize the listeners architecture you will need to understand brod. And you will need to understand brod and kpro etc. to debug some unpleasant errors.
Before leaving the project I was seriously considering replacing that whole stack with erlkaf and made a quite convincing replacement PoC in a few hours. There is not much boilerplate either but I had problems with dependencies, hopefully fixed by now.
lud
Do you remember how did you work around that?
We did not 
It was not a frequent error. Just that brod is slow to start and to setup the consumer group. Or we did something wrong but erlkaf booted way faster.
For the mocking part I put a private API in front of the kafka libraries to abstract the consuming/producing API and then I implemented and maintained a simple pubsub system mimicking the behaviours of Kafka. And some test helpers on top of that.
This mock was only for tests but it had it’s own unit tests just to be sure. You need to implement a lot of things. Even if you want to only consume, you will need to produce to your mock from the test.
Anyway, a lot of work, not recommended…
dimitarvp
Yep, that’s why I would prefer a library that comes with a Kafka mock out of the box but oh well. In my current employer colleagues maintain a thin wrapper on top of :brod but it only supports producing and I have no free time to contribute to add consuming to it – not yet I don’t have the time anyway, Soon™ I should have it.
Ah well, I guess I can just ignore it then, though I am not keen on the idea of losing data which shouldn’t happen anyway because the Kafka topic will be filled by other apps, and the consumer group I’ll use will remember the offset so technically nothing should be lost.
egze
For what it’s worth - we use :kaffe at work and it’s OK.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Courses that'll move you from confusion to "Aha, now I get it!"
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"