Questions & Help>Questions

How to create Apache Kafka consumer groups?

#kafka
kgllev

kgllev

How to create Apache Kafka consumer groups?

I’m quite new to Kafka and trying to implement a simple queue system using 1 topic and 1 consumer group assigned to that topic. However I couldn’t figure out how to create a consumer group. I’m using Kafka_ex library and Kafka version 2.4.1

4 comments
#kafka
4 1667 4

Marked As Solved

sb8244

sb8244

Author of Real-Time Phoenix

I believe this is done fully by Application config, at least it is in apps at work. The consumer_group and auto_offset_reset go into the consumer group setup, and will automatically be created if it doesn’t exist.

We recently switched to brod and have had better luck with it in production.

2
Post #3

Also Liked

tiagodavi

tiagodavi

Hey Kgllev. Why don’t you try to use Rabbitmq… it uses Erlang behind the scenes and it’s simpler than Kafka. I can show you an example if you need help because I have been using it on my projects.

I use Docker to separate things… so my rabbimq is isolated in its own container.

https://www.rabbitmq.com/tutorials/tutorial-one-elixir.html

1
Post #2
Big_chris

Big_chris

To create the consumer groups I would advise you to switch from Kafka to Cassandra. This is much easier there than with Kafka! Also, Cassandra is much bigger than Kafka, which could have a positive effect on your consumer groups!

If you don’t want to change the database completely, there is also the possibility to connect the two databases via a provider like Aiven. Aiven supports both Kafka and Cassandra, so this should not be a problem. Also, the data processing by Aiven is really fast. Could be something for you, right?

1
Post #5

Where Next?

View thread on forum (has 4 responses!)
kafka
Home Questions & Help>Questions
#kafka
4 1668 4
Last post

Popular in Questions Top

shahryarjb
Questions & Help>Questions

How to convert Date to DateTime which Ecto needs
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
#ecto#datetime
15 14989 5
New
joaquinalcerro
Questions & Help>Questions

What is the best approach for fetching large amount of records from postgresql with ecto
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
#ecto
126 24587 27
New
itssasanka
Questions & Help>Questions

Why use `utc_datetime` over `naive_datetime` for Ecto?
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
#ecto#timestamps#datetime#timezones
115 14814 19
New
sacepums
Questions & Help>Questions

Converting date string to Ecto.Date
Hey guys. I'm new to elixir and im really stocked about it. But I ran into a bit of problem - I need to convert a date sting, for examp...
/phoenix#ecto#dates
14 13473 4
New
vertexbuffer
Questions & Help>Questions

Deleting item from a list
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
7 24097 4
New
aadeshere1
Questions & Help>Questions

Write while loop equivalent in elixir
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
#while#loop
60 28200 36
New
rms.mrcs
Questions & Help>Questions

Transform a list into an map with indexes using Enum module
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
35 32607 9
New
stefanluptak
Questions & Help>Questions

Using VSCode on multiple monitors
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
#vscode#vscode-elixir
7 17615 6
New
fireproofsocks
Questions & Help>Questions

Ecto delete a record WITHOUT selecting first
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
#ecto#optimization#delete-method
22 16810 8
New
LegitStack
Questions & Help>Questions

Websocket server in Elixir or phoenix
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
/phoenix#websockets
20 11315 8
New

Other popular topics Top

yurko
Chat & Discussions>Wikis

Elixir apps as systemd services - info & wiki
Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for ...
#wiki#deployment#linux
73 19423 25
New
shahryarjb
Questions & Help>Questions

How to convert Date to DateTime which Ecto needs
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
#ecto#datetime
15 14989 5
New
senggen
Questions & Help>Questions

How to fix *Bad argument in call to erlang:'++'(<<"xxx/crash.log">>, ".3") in lager_rotator_default:rotate_logfile/2 line 84*
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
#production#error#log
2 46314 2
New
yawaramin
Questions & Help>Questions

Dialyzer: suppress warning on a specific function
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
#dialyzer
25 13690 6
New
fireproofsocks
Questions & Help>Questions

Defining Ecto Schemas (PostGres) to use NOT NULL in column definitions
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
#ecto
29 14078 4
New
polypush135
Questions & Help>Questions

Proper use of Task.async?
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
#task
26 15568 8
New
nsuchy
Questions & Help>Questions

IEX in Windows Powershell?
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
#iex#microsoft-windows#windows#powershell
15 29309 4
New
lk-geimfari
Questions & Help>Questions

The most correct way to read a JSON file
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
23 13679 2
New
ashish173
Questions & Help>Questions

Difference in between :utc_datetime and :naive_datetime in Ecto
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
#ecto#timestamps
79 27907 15
New
alice
Questions & Help>Questions

What are the main benefits of Elixir compared to Clojure?
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
#clojure
223 32979 47
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

EEF

Supporting innovation across the BEAM ecosystem.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement