That’s a really good question but it highly depends on a few factors:

• Do you want data types to be strict for each field?
• Do you want optional fields or is every field mandatory?
• Do you want zero-copy? This means that there is no serialize / unserialize stages: when you get a byte array buffer it can directly be iterated over and read without having to convert it to an in-memory structure.
• Do you want future extensibility without old clients breaking?
• Do you mind having a schema known beforehand?

I’ve been studying data formats lately and to be fair, I can only recommend Google’s FlatBuffers, Ethereum’s RLP (only a byte array encoding though, no concept of types) or maybe MessagePack (although that has the ser/deser stage).

Additionally, a very simplified ASN.1 format that gets ser/deser in BER or DER formats can be very viable and easy to do – but honestly, don’t go down the ASN.1 rabbit hole. You’ve been warned.

Looking through Comparison of data-serialization formats in Wikipedia only shows me one more good candidate – Binn – but haven’t evaluated it yet. Check out that page, it’s not a bad comparison (probably lacking good other formats I’d presume).

Seeing your post’s loosely defined requirements I’d not go for FlatBuffers even though I think it’s a good fit for many scenarios. For your case JSON/BSON might be the least painful option.

Or simply Erlang’s ETF format? (Check out :erlang.binary_to_term and :erlang:term_to_binary)