Menkir

Menkir

Jexon - Lossless encoding to JSON

Hi everyone,

I recently released a small library built on top of Jason that converts Elixir maps and structs to JSON without loss. Atoms and tuples are data types that have no direct representation in JSON. As a result, keys and values of these data types are specially encoded with prefixes. Additionally, structs without the @derive attribute can be directly encoded to JSON.

The library offers three functions:

  • to_map: This function formats structs into maps and stores the __struct__ key.
  • to_json
  • from_json

Feel free to contribute in case you have more edge cases to consider.

Happy Coding!

Most Liked

adamu

adamu

Sorry to be that guy, but if the purpose is to have kind of transport encoding with the objective of eventual decoding back into Elixir, it seems that external term format would be a good choice, unless you need JSON for other interoperability reasons. Maybe some suggested use cases or a mention of this in the docs would be useful :slight_smile:

Menkir

Menkir

I fully understand this. We initially considered this formatting, but we had several concerns:

If you want to transport the state via JSON (for instance, for persistence), it’s necessary to encode it into Base64. Subsequently, you can store it in any DB of your choice. Yet, when you try to retrieve this state, you’re confronted with binary data, which is incomprehensible to both the remote machine and to you.

With Jexon-encoded Elixir structs, it becomes feasible to perform diffs between varied states or carry out migrations when certain keys have changed.

Debugging is more straightforward than when using ETF. I can effortlessly copy and paste parts or the entire state into a remote IEX, then pipe it through functions to diagnose the issue.

Ultimately, the decision boils down to preference: Would you rather store your data in binary or JSON format? In my specific situation, the data was initially stored locally in ETS. However, as is often the case, requirements shifted, necessitating the state’s transfer to a centralized backup service that exclusively accepts JSON.

mayel

mayel

Looks interesting! Some examples of the JSON output would be useful in the readme.

al2o3cr

al2o3cr

One tricky extreme edge-case: Jexon.to_json will produce the same output for these two maps

%{"__atom__:foo" => 1}
# vs
%{foo: 1}

A less edge-case situation: what about maps with other kinds of terms as keys? For instance, a lot of “sparse grid” problems make sense as a map of {x, y} tuples to values:

game_of_life_grid = %{
  {0, 0} => 1,
  {0, 1} => 2,
  # etc
}
Menkir

Menkir

For the extreme edge case: Yes, both lead to the same result because atoms would be encoded as strings in any other Encoder like Jason, Poison etc… If you want to keep the information about the type so that you can parse it back to the original data structure you need to encode the type like __atom__:... otherwise you will never know if it’s a string key or an atom. You can try it oppisite: Take the JSON output and parse it back you will always get

%{foo: 1}`

If you take Jason or Poison you need to pass an option like keys: :atoms so it ends up to the developer and not to the data itself which types it has.

Tuple keys are a good one. I didn’t consider tuples as keys. Will fix it soon. Thanks!

Where Next?

Popular in Libraries Top

praveenperera
FastRSS Parse RSS feeds very quickly: This is rust NIF built using rustler Uses the RSS rust crate to do the actual RSS parsing Speed...
New
arkgil
Hi all! I’m happy to announce that Telemetry v0.3.0 is out! This release marks the conversion from Elixir to Erlang so that all the libr...
New
dbern
I’m excited to announce that TaxJar has developed and open-sourced DateTimeParser. We developed it because we found a need to parse user ...
New
pkrawat1
Presenting Aviacommerce, open source e-commerce platform in Elixir Aviacommerce is an open source e-commerce platform in Elixir. We at...
New
wfgilman
I’ve cleaned up and open sourced three financial libraries I was using for my company. They are bindings for the APIs of these three comp...
New
benlime
I created a new library GitHub - benvp/ex_cva: Class Variance Authority for Elixir which aims to make it very easy to define different va...
New
tfwright
After working on it for a couple of months and using it in production for most of that time, today I’ve released LiveAdmin, a LiveView ba...
New
tmbb
I’ve published the first version of my Makeup library. It’s a syntax highlighter for Elixir in the spirit of Pygments, Currently it highl...
New
wmnnd
Hi there, for my project DBLSQD, I needed a file storage solution that is a bit more flexible than Arc. Because I thought others might f...
New
ericlathrop
I built a silly site for Halloween that uses Phoenix Channels on the backend, and React on the frontend. I had many problems integrating ...
New

Other popular topics Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New

Sub Categories:

We're in Beta

About us Mission Statement