larshei

larshei

How to Decode Protobuf/gRPC messages when I have only a JSON Schema available?

Hey everyone!

I’m successfully receiving events from Salesforce’s PubSub API via gRPC, but I’m struggling to decode the actual payload.

Since Salesforce allows schema customization, they don’t provide predefined Protobuf definitions for the event data. Instead, events are wrapped in a generic message, containing:

  • A schema_id (e.g., “23456789asdf”)
  • A binary payload, which needs decoding

I can fetch the corresponding JSON Schema using:

Eventbus.V1.PubSub.Stub.get_schema(
  channel, 
  %Eventbus.V1.SchemaRequest{schema_id: "23456789asdf"}
)

I would love to have a flow like the following:

  1. :white_check_mark: Store known schemas (no problem)
  2. :white_check_mark: Check if a schema is known on receiving an event (no problem)
  3. :white_check_mark: Fetch missing schemas (no problem)
  4. :question: Compile missing schemas
  5. :question:Decode the payload using the schema

I’m stuck on how to go from a JSON Schema to something usable for decoding the Protobuf payload . Any guidance would be greatly appreciated! :+1:

Marked As Solved

larshei

larshei

Salesforce uses Avro Schemas: Event Data Serialization with Apache Avro | Use Pub/Sub API | Pub/Sub API | Salesforce Developers

So this library holds the answer: GitHub - beam-community/avro_ex: An Avro Library that emphasizes testability and ease of use.

{:ok, %{schema_json: json_schema}} = Eventbus.V1.PubSub.Stub.get_schema(
  channel, 
  %Eventbus.V1.SchemaRequest{schema_id: "23456789asdf"}
)

{:ok, avro_schema} = AvroEx.decode_schema(json_schema)

{:ok, payload} = AvroEx.decode(avro_schema, event.payload)

Where Next?

Popular in Questions Top

minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
New
quazar
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
New
sacepums
Hey guys. I'm new to elixir and im really stocked about it. But I ran into a bit of problem - I need to convert a date sting, for examp...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New

Other popular topics Top

SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement