peppy

peppy

How to serialize/unserialize elixir map data for RabbitMQ?

I’ve been working on a RabbitMQ publisher and discovered that you can only publish strings to a RabbitMQ queue. The problem is I’m trying to publish a map to the RabbitMQ queue and it won’t let me.

For example:

my_map = %{message: "test", userid: 234}

I tried using Jason.encode!(my_map) to serialize the map before publishing it to the queue, which seems to work. But when I receive the message out of the queue in a later part of the app, I use Jason.decode!(my_map_from_queue), and I get this:

my_map = %{"message" => "test", "userid" => 234}

I also considered:

:erlang.term_to_binary(my_map)
:erlang.binary_to_term(my_map)

However, this thread mentioned it is unsafe to do on untrusted data: Serialize and unserialize structs, tuples, keyword lists, atoms? - #5 by sezaru

My intention is to save messages from a chat room to the database, so the message text being submitted could be anything and untrusted input coming from the chat room.

The original map being submitted is pre-formatted to a specific database schema. The goal was to gather all chat messages to RabbitMQ and use a producer to collect these messages in batches and use Repo.insert_all([list of maps]). The hope was that anything going into RabbitMQ would come out exactly the same.

What is a good solution for the serialization or “stringifying” and “destringifying” things going into and out of RabbitMQ queues?

Most Liked

SirWerto

SirWerto

I think you can use binary_to_term/2 with the safe option.

:erlang.binary_to_term(my_map, [:safe]).

this only use atoms that have been created earlier, but you have to deal with the possible exception raising.

dimitarvp

dimitarvp

What we do at my current employer is using MessagePack through the Msgpax library. Works well and keeps the door open for having a multi-language stack, if we ever need it.

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

Hey @myefomit welcome. Unfortunately I don’t think this is good advice here. Decoding to atoms will create new atoms if the message keys change. Unless the shape of the RabbitMQ messages never changes this will eventually lead to excessive memory use and crash the application.

dimitarvp

dimitarvp

As a bonus you won’t ever get attacked by serialized functions inside the payload. AFAIR the :safe flag does not include functions.

Where Next?

Popular in Questions Top

pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
Exadra37
Sometimes I want to check if the input into a function is not a blank string. My first approach: defmodule Example do def do_stuff(s...
New
jc00ke
Expanding on this topic: https://forum.elixirforum.net/t/map-typespec-question/19217 Let’s say I have a map with required and optional k...
New
idi527
I’ve been re-reading swift book again and noticed that multiline strings there don’t have a trailing line break, unlike in elixir iex(2)...
New

Other popular topics Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

We're in Beta

About us Mission Statement