williamthome

williamthome

Exneus - JSON parser, generator and formatter for Elixir built on top of the new OTP json module

Exneus is an incredibly flexible and performant JSON parser, generator and formatter for Elixir. It is a wrapper for Elixir of Euneus, an Erlang library built on the top of the new OTP json module.

Both encoder and decoder fully conform to RFC 8259 and ECMA 404 standards and are tested using JSONTestSuite.

The new OTP json module was introduced in OTP-27, which is compatible with Elixir v1.17, but by installing json_polyfill, the minimum requirement reduces to OTP-24 and Elixir v1.12. I’ve not tested it in other versions, but the CI says it works.

Exneus/Euneus’s goal is to be very flexible and performant.

The first release of Exneus is an Erlangish lib. I just adapted Euneus’s Erlang code to Elixir. This will change :slight_smile:

Further explanation and examples are available at hexdocs.

Features

  • Encode
  • Decode
  • Stream
  • Minify
  • Format

Encode option details

  • codecs - Act like plugins. There are some built-in options, and it is also possible to create custom ones.

    Currently built-in encode codecs:

    • timestamp
    • datetime
    • ipv4
    • ipv6
    • records
  • codec_callback - - Overrides the default codec resolver.

  • nulls - Defines which values should be encoded as null.

  • skip_values - Defines which map values should be ignored. This option permits achieves the same behavior as Javascript, which ignores undefined values of objects.

  • key_to_binary - Overrides the default conversion of map keys to a string.

  • sort_keys - Defines if the object keys should be sorted.

  • keyword_lists - If true, converts keyword_lists into objects.

  • escape - Overrides the default string escaping.

  • encode_integer - Overrides the default integer encoder.

  • encode_float - Overrides the default float encoder.

  • encode_atom - Overrides the default atom encoder.

  • encode_list - Overrides the default list encoder.

  • encode_map - Overrides the default map encoder.

  • encode_tuple - Overrides the default tuple encoder.

  • encode_pid - Overrides the default pid encoder.

  • encode_port - Overrides the default port encoder.

  • encode_reference - Overrides the default reference encoder.

  • encode_term - Overrides the default encoder for unsupported terms, like functions.

Decode option details

  • codecs - Act like plugins. There are some built-in options, and it is also possible to create custom ones.

    Currently built-in decode codecs:

    • timestamp
    • datetime
    • ipv4
    • ipv6
    • pid
    • port
    • reference
  • null - Defines which term should be considered null.

  • binary_to_float - Overrides the default binary to float conversion.

  • binary_to_integer - Overrides the default binary to integer conversion.

  • array_start - Overrides the :json.array_start_fun/0 callback.

  • array_push - Overrides the :json.array_push_fun/0 callback.

  • array_finish - Overrides the :json.array_finish_fun/0 callback.

    In addition to the custom function, there are:

    • ordered
    • reversed
  • object_start - Overrides the :json.object_start_fun/0 callback.

  • object_keys - Transforms JSON objects key into Elixir term.

    In addition to the custom function, there are:

    • binary
    • copy
    • atom
    • existing_atom
  • object_push - Overrides the :json.object_push_fun/0` callback.

  • object_finish - Overrides the :json.object_finish_fun/0 callback.

    In addition to the custom function, there are:

    • map
    • keyword_list
    • reversed_keyword_list

Benchmark

The next step is to improve the performance since no optimization has been done yet. However, there is a simple benchmark comparing it with some known libraries, including the OTP json module.

IMPORTANT!

The benchmark result below just gives an idea of performance since the benchmark runs are very simplistic. Subsequent releases will update those values.

Also, to be an honest benchmark, all libs should produce the same output. I have not checked this.

Encode results:

Name                 ips
euneus           36.95 K
json (OTP)       35.85 K - 1.03x slower +0.83 μs
Exneus           35.71 K - 1.03x slower +0.94 μs
jiffy            34.70 K - 1.06x slower +1.76 μs
Jason            26.75 K - 1.38x slower +10.32 μs
thoas            16.40 K - 2.25x slower +33.92 μs
jsone            15.52 K - 2.38x slower +37.38 μs
jsx               4.21 K - 8.77x slower +210.29 μs
JSON              3.88 K - 9.52x slower +230.47 μs

Decode results:

Name                 ips
json (OTP)       18.79 K
euneus           18.06 K - 1.04x slower +2.16 μs
Exneus           17.68 K - 1.06x slower +3.34 μs
Jason            15.16 K - 1.24x slower +12.74 μs
jsone            13.82 K - 1.36x slower +19.17 μs
jiffy            12.76 K - 1.47x slower +25.15 μs
thoas            11.89 K - 1.58x slower +30.89 μs
jsx               4.93 K - 3.81x slower +149.64 μs
JSON              2.43 K - 7.74x slower +358.48 μs

If you liked those tools, find them helpful, and would like to see them improved, please consider sponsoring me or buying me some coffees :heart:

"Buy Me A Coffee"

Feel free to submit suggestions and to contribute :smiley:


Most Liked

williamthome

williamthome

Sorry, I wasn’t clear.

Using just the OTP :json module without any other lib is fine and is extremely fast! It provides encoding and decoding:

iex(1)> :json.encode(:foo)
[34, "foo", 34]
iex(2)> :json.decode("\"foo\"")
"foo"

However, the OTP :json module is available starting from OTP-27 and provides few options. On the other hand, :euneus is built on top of :json, providing more flexibility and options, including a JSON formatter.

For example, to decode Map keys as atoms:

iex(1)> :json.decode(~s({"foo":"bar"}), [], %{object_push: fn k, v, acc -> [{String.to_atom(k), v} | acc] end})
{%{foo: "bar"}, [], ""}
iex(2)> :euneus.decode(~s({"foo":"bar"}), %{object_keys: :atom})
%{foo: "bar"}

And that’s just one example. Also, I adapted the OTP json module into a lib called json_polyfill to increase the compatibility with OTP and Elixir versions, OTP-24 and Elixir v1.12, respectively.

Please let me know if this answers your question.

D4no0

D4no0

Quick question, why erlang devs need Euneus and can’t use the json library directly from OTP?

williamthome

williamthome

Thiago, I truly appreciate your perspective and respect your comment. Not only for Elixir, but Michał is also an extremely important person to the Erlang community, and his contributions are always great!

I do not know exactly what you mean by “fragmentation.”. Don’t get me wrong too, but I’m not inclined to start a discussion here comparing Exneus to another existing library. The main points of Exneus are in the thread.

Each library has its strengths and weaknesses. If you find a library that addresses your project’s requirements, feel free to consider using it. At the end of the day, libraries are tools, and choosing one over another is entirely optional. Moreover, libraries can always learn from and benefit from one another, and I would be delighted if Exneus could bring value to others. :blush:

williamthome

williamthome

There is no C library involved. The OTP json module is a pure Erlang module.

EDIT

The json_polyfill just adapts some parts of the code to be compatible with other OTP versions.

dimitarvp

dimitarvp

Yes it does, thank you.

Where Next?

Popular in Libraries Top

RobertDober
Earmark is a pure-Elixir Markdown converter. It is intended to be used as a library (just call Earmark.as_html), but can also be used as...
239 11851 134
New
kevinlang
Hey all, We have made an Ecto3 Adapter for SQLite3, ecto_sqlite3! We have successfully on-boarded the full suite of integration tests (...
New
sasajuric
I’d like to announce a small library called boundaries. This is an experimental project which explores the idea of enforcing boundaries ...
New
markmark206
simple_feature_flags is a tiny package that lets you turn features on or off based on which environment (e.g. localhost, staging, product...
New
blatyo
https://www.conduitframework.com/ The best overview for how things are tied together is this presentation. Modules and functions are pre...
New
wfgilman
I’ve cleaned up and open sourced three financial libraries I was using for my company. They are bindings for the APIs of these three comp...
New
danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 27727 240
New
OvermindDL1
I created a new library (rather I pulled out a couple files from my big project), it manages an operating system PID file for the BEAM. ...
New
Azolo
Hey everyone, I just released WebSockex which is a Elixir WebSocket client. WebSockex strives to work as a OTP special process, be RFC6...
New
maltoe
Hello! Came here to announce ChromicPDF, a pet project PDF generator I’ve been working on for the past few months. Why another PDF gener...
New

Other popular topics Top

lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

Sub Categories:

We're in Beta

About us Mission Statement