peppy

peppy

How to serialize/unserialize elixir map data for RabbitMQ?

I’ve been working on a RabbitMQ publisher and discovered that you can only publish strings to a RabbitMQ queue. The problem is I’m trying to publish a map to the RabbitMQ queue and it won’t let me.

For example:

my_map = %{message: "test", userid: 234}

I tried using Jason.encode!(my_map) to serialize the map before publishing it to the queue, which seems to work. But when I receive the message out of the queue in a later part of the app, I use Jason.decode!(my_map_from_queue), and I get this:

my_map = %{"message" => "test", "userid" => 234}

I also considered:

:erlang.term_to_binary(my_map)
:erlang.binary_to_term(my_map)

However, this thread mentioned it is unsafe to do on untrusted data: Serialize and unserialize structs, tuples, keyword lists, atoms? - #5 by sezaru

My intention is to save messages from a chat room to the database, so the message text being submitted could be anything and untrusted input coming from the chat room.

The original map being submitted is pre-formatted to a specific database schema. The goal was to gather all chat messages to RabbitMQ and use a producer to collect these messages in batches and use Repo.insert_all([list of maps]). The hope was that anything going into RabbitMQ would come out exactly the same.

What is a good solution for the serialization or “stringifying” and “destringifying” things going into and out of RabbitMQ queues?

Most Liked

SirWerto

SirWerto

I think you can use binary_to_term/2 with the safe option.

:erlang.binary_to_term(my_map, [:safe]).

this only use atoms that have been created earlier, but you have to deal with the possible exception raising.

dimitarvp

dimitarvp

What we do at my current employer is using MessagePack through the Msgpax library. Works well and keeps the door open for having a multi-language stack, if we ever need it.

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Hey @myefomit welcome. Unfortunately I don’t think this is good advice here. Decoding to atoms will create new atoms if the message keys change. Unless the shape of the RabbitMQ messages never changes this will eventually lead to excessive memory use and crash the application.

dimitarvp

dimitarvp

As a bonus you won’t ever get attacked by serialized functions inside the payload. AFAIR the :safe flag does not include functions.

Where Next?

Popular in Questions Top

dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
LegitStack
I’m hoping you guys can give me some general advice and perhaps code examples if you’re feeling up to it. I’m very interested in Elixir,...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
makeitrein
Hey all, just started picking up Elixir last week and am writing a scraper as a learning project. Baby step #1 is extracting the number ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
Fl4m3Ph03n1x
Background Let’s assume I have a typical GenServer that receives messages as requests, does some operation in a DB and returns responses....
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New

Other popular topics Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
sergio
I couldn’t find any guides that worked well with Phoenix 1.6.0 and esbuild. I hope this helps people test the waters and eases you into t...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
273 38985 115
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

We're in Beta

About us Mission Statement