darnahsan
Broadway kafka producer not getting all partition assignment from Upstash Kafka
My understanding is that this is an upstash issue. I have a topic with 3 partitions but I only get assignment for 2 partitions. Below is my broadway code. Is there anything in the config that could be causing this missing partition assignment ?
defmodule Maverick.Broadway.Upstash do
@moduledoc """
The Upstash Kafka context.
"""
use Broadway
alias Maverick.Whatsapp.Message, as: WhatsappMessage
def upstash_kafka_port(), do: Application.fetch_env!(:maverick, :upstash_kafka_port)
def upstash_kafka_endpoint(), do: Application.fetch_env!(:maverick, :upstash_kafka_endpoint)
def upstash_kafka_hosts(), do: [{upstash_kafka_endpoint(), upstash_kafka_port()}]
def upstash_kafka_authentication(),
do:
{Application.fetch_env!(
:maverick,
:upstash_kafka_sasl_mechanism
), Application.fetch_env!(:maverick, :upstash_kafka_username),
Application.fetch_env!(:maverick, :upstash_kafka_password)}
def upstash_kafka_group_id(), do: "localhost-00"
def upstash_kafka_topic(), do: "localhost"
def producer_concurrency(), do: 2
def processors_concurrency(), do: 10
def start_link(_opts) do
Broadway.start_link(__MODULE__,
name: __MODULE__,
producer: [
module:
{BroadwayKafka.Producer,
[
hosts: upstash_kafka_hosts(),
group_id: upstash_kafka_group_id(),
topics: [upstash_kafka_topic()],
offset_reset_policy: :earliest,
client_id_prefix: "localhost-maverick",
client_config: [
sasl: upstash_kafka_authentication(),
ssl: [
# from CAStore package
cacertfile: CAStore.file_path(),
verify_type: :verify_peer,
customize_hostname_check: [
match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
]
]
]
]},
concurrency: producer_concurrency()
],
processors: [
default: [
concurrency: processors_concurrency()
]
]
)
end
First Post!
darnahsan
a new topic with 4 partitions got all partition assignments for now seem odd partition count is the problem, could the broadway config in any way be playing a part in this ?
Popular in Questions
Good day to you all.
I have been struggling to get a query involving like and ilike to work.
Can anyone assist me on this, please?
pro...
New
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
New
i’m a new one to elixir
which editor can i use
vs code? or atom?
Thanks! :smiley:
New
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
I tried installing
elixir 1.11.2
erlang 23.3.4
via asdf in my zsh shell. Enabled the versions locally and globally.
When I list them ...
New
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
Hi all,
I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage.
I'm trying to use Postg...
New
Other popular topics
What is the proper way to load a module from a file in to IEX?
In the python world, doing something like this pretty standard:
from ....
New
Phoenix 1.4.0 released
Phoenix 1.4 is out! This release ships with exciting new features, most notably
with HTTP2 support, improved deve...
New
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
I have followed this StackOverflow post to install the specific version of Erlang.
And When I am running mix ecto.setup then getting fol...
New
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set?
Thanks.
New
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch.
This project took far...
New
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
I tried installing
elixir 1.11.2
erlang 23.3.4
via asdf in my zsh shell. Enabled the versions locally and globally.
When I list them ...
New
Hi guys, i’m new in the Elixir world, and i have to say, that i love it!
i’m having some problem to understand anonymous functions with ...
New










