roflbobl

roflbobl

Changing module of saved binary data

I run a system, where some structs are saved as binary with :erlang.term_to_binary and then deserialized with :erlang.binary_to_term. The problem is, that if I alter the struct, it does not match the saved binary, and it will then not be deserialized correctly.

I have a test where i add a key, after saving some data, and of course, the deserialized data is missing that key. Is was hoping it will just give the new key the value nil.

Has anyone solved this problem?

Most Liked

al2o3cr

al2o3cr

If you’re storing long-lived term data (versus caching, etc where old versions can be discarded) here are two approaches that can help:

  • explicit migration when changing the underlying struct. You’d write a one-off task to load each term, adjust its structure, then save it back
  • explicit versioning in the data. Add a version field to the struct with a default value. Bump it every time you add/remove fields. The code that says binary_to_term will need to check version and handle converting the old shape into the new shape

IMO the motivation to pick one versus the other is how often “old” data is read; if it’s used constantly then updating it once in a batch will be faster (the first approach) - but if it’s mostly historical and only read occasionally then doing it on-demand (the second approach) will avoid doing computations that might never be used.

NobbZ

NobbZ

Do not serialize the struct, but just the map without the __struct__ key.

Roughly:

data
|> Map.from_struct()
|> :erlang.term_to_binary()

For the deserializing something like the following should do:

binary
|> :erlang.binary_to_term()
|> then(&struct(YourStruct, &1))

This should work, as drafted in the following iex session:

iex(1)> defmodule S do defstruct ~w[a b c d e f g]a end
iex(2)> struct(S, %{a: 1})
%S{a: 1, b: nil, c: nil, d: nil, e: nil, f: nil, g: nil}

On top of that though, I suggest to finally stabilize the struct quickly. Or to version your data format and use explicit conversion or “upcasters”.

gregvaughn

gregvaughn

You could probably even get fancy and use the built-in @vsn Module attribute

Where Next?

Popular in Questions Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

We're in Beta

About us Mission Statement