Klife Protocol

link: GitHub - oliveigah/klife_protocol: Elixir implementation of the kafka protocol

I’m thrilled to share that I’ve just completed the initial version of the docs for Klife Protocol. While it’s still early days, I believe the available features could be useful for someone building something with Kafka, so I wanted to share it here with the community.

Klife Protocol already supports all messages and versions currently available with solid performance for most use cases.

My hope is to release a version of Klife Protocol on Hex soon, and I’d love to hear any feedback or suggestions that the community may have. Thanks for your interest.

What is the library about?

Klife Protocol is a library that simplifies the serialization work required for communication with Kafka brokers. It’s important to note that Klife Protocol is not a full Kafka client and is not intended to be used directly in application code. Instead, it’s meant to be used as a component inside a full Kafka client.

Motivation of the library

The motivation behind Klife Protocol was to gain a deeper understanding of Kafka and its inner workings. About a year ago, I decided to explore Kafka and the best way to do that was to build my own client.

However, as development progressed and complexity emerged, I decided to implement the protocol first, as it would provide a better understanding under the hood.

Implementing the protocol also allowed me to use some features, such as tagged fields and compact serialization, which as far as I can tell are not available in other Kafka serialization libraries for the BEAM. This implementation of the protocol will evolve over time, and I plan to use it in a future Kafka client called Klife.

Additionally, the separation of the client and protocol provides benefits to other clients that may want to use this implementation of the Kafka protocol independently.

Motivation of this post

I’m posting this here in order to gather any feedback of anyone that may be intereseted in this problem (that is not only about kafka).

Throughout the development process, various challenges have arisen, and I have done my best to overcome them. However, I am aware that there is always room for improvement, and I believe that feedback from the community can greatly assist in this regard. Some areas I would like to highlight:

• Binary manipulation performance
• API ergonomy and design
• Documentation
• Open source in general

I am open to feedback on any aspect of the project, and I mentioned those areas because they have been the most challenging for me recently, and that’s why they are fresh in my mind.

Closing thoughts

Thank you for taking the time to read through this. I hope that this library can be of use to the community in some way. I am eager to hear any feedback you may have.