bluephosphor

bluephosphor

How do I connect to Upstash Kafka with Broadway (ssl issues)

I am trying to connect to my Upstash Kafka cluster using BroadwayKafka. When connecting I get the following error:

[warning] :brod_client [#PID<0.623.0>] MyProject.Data.Pipeline.Broadway.Producer_0.Client is terminating
reason: [
  {{:"project-name.upstash.io", 9092},
   {{:failed_to_upgrade_to_ssl,
     {:tls_alert,
      {:handshake_failure,
       ~c"TLS client: In state wait_cert_cr at ssl_handshake.erl:2140 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,hostname_check_failed}"}}},
    [
      {:kpro_connection, :maybe_upgrade_to_ssl, 5,
...snip

My connection parameters currently look like this:

{BroadwayKafka.Producer,
  [
   hosts: ["project-name.upstash.io": 9092],
   group_id: "group_1",
   topics: ["events"],
   client_config: [
     sasl:
       {:scram_sha_256,
        "username",
        "password"},
     ssl: [
       cacertfile: CAStore.file_path(), # from CAStore package
       verify_type: :verify_peer
     ]
   ]
  ]}

I tried setting ssl: true instead of manual config, it fails with an error indicating that ca certs are missing. I tried setting verify_type: :verify_none but the error is the same. I tried setting server_name_indication: "project-name.upstash.io" because the docs said it was important to set it, but I’m not sure that is the correct value and it gives me a different error:

[warning] :brod_client [#PID<0.456.0>] MyProject.Data.Pipeline.Broadway.Producer_0.Client is terminating
reason: [
  {{:"smooth-grouper-5277-us1-kafka.upstash.io", 9092},
   {{:failed_to_upgrade_to_ssl,
     {:options,
      {:server_name_indication, "project-name.upstash.io"}}},
    [
      {:kpro_connection, :maybe_upgrade_to_ssl, 5,

I feel this is probably a really simple issue, if only I were familiar with the erlang ssl options. Any help would be appreciated!

Marked As Solved

ruslandoga

ruslandoga

:wave: @bluephosphor

You might need to provide :customize_hostname_check option to make :verify_peer work with subdomains / wildcard certs.

iex> Mix.install [:castore]
iex> :ssl.start()
:ok

iex> :ssl.connect(~c"gist.github.com", 443, verify: :verify_peer, cacertfile: CAStore.file_path())
{:error,
 {:tls_alert,
  {:handshake_failure,
   'TLS client: In state wait_cert_cr at ssl_handshake.erl:2113 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,hostname_check_failed}'}}}

iex> :ssl.connect(~c"gist.github.com", 443, verify: :verify_peer, cacertfile: CAStore.file_path(), customize_hostname_check: [match_fun: :public_key.pkix_verify_hostname_match_fun(:https)])
{:ok,
 {:sslsocket, {:gen_tcp, #Port<0.9>, :tls_connection, :undefined},
  [#PID<0.205.0>, #PID<0.204.0>]}}

More info: Erlang standard library: ssl | EEF Security WG

Where Next?

Popular in Questions Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
New
quazar
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs &amp; Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3268 119930 1237
New
William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 45766 226
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New

We're in Beta

About us Mission Statement