fireproofsocks
Where did the name "binaries" come from? And how does this relate to Base2
I’ve been diving deep into charlists, encodings, code points, etc… thank you to the community who have humored me in the forums and in Slack.
I have a question about Elixir’s use of the term “binary”. It seems like the term was bogarted… if I showed you a series of 1’s and 0’s, you’d think “Ah, that’s binary.” As in, that data is being represented in a binary format.
But nope, in Elixir, a binary is for all intents and purposes synonymous with a string in other languages. But in Elixir, we don’t take “binary” to mean a series of 1’s and 0’s, but rather, a binary means “a sequence of bytes” (i.e. yes, a sequence of 1’s and 0’s, but specifically, groups of 8 of them).
This has the weird side-effect of making us refer to ACTUAL series of 1’s and 0’s as “Base2” encoding, which isn’t supported in the core, but is available in a little-known package: https://hex.pm/packages/base2
Can someone enlighten me on this?
Thanks!
Marked As Solved
rvirding
I think we called it a binary just to separate from all other Erlang terms. It is just a collection, or array, of raw bytes where we add meaning to it. If the number of bits is not divisible by 8 then it is a bitstring. Again we are the ones who add meaning to it not Erlang/Elixir/BEAM.
Remember that in Erlang we mostly represent strings as lists of integers. Again we put meaning to the bytes, whether they are ascii/latin1/unicode codepoints. We NEVER make lists of utf-8/16/32 encoded characters. What’s the point? There was a good talk on CodeBeamSF on this by Marc Sugiyama Unicode, Charsets, Strings, and Binaries. I don’t know when it will be on line.
Also Liked
al2o3cr
These two are not the same - String.split with an empty string splits on graphemes, which are a further unit bigger than codepoints.
The rules for clustering codepoints into graphemes are defined by the Unicode standards, and the code for handling them is generated from canonical text files.
iex(4)> String.codepoints("🇺🇸")
["🇺", "🇸"]
iex(5)> String.split("🇺🇸", "", trim: true)
["🇺🇸"]
iex(6)> "🇺🇸" <><<0>>
<<240, 159, 135, 186, 240, 159, 135, 184, 0>>
The single displayed character
is a grapheme, composed of two codepoints U+1F1FA and U+1F1F8, represented by 8 bytes.
Another way that codepoints and graphemes can diverge is combining characters; for instance, U+0308 is “Combining Diaresis” which will add ¨ to the preceding character. Example:
iex(9)> s = "ca\u0308t"
"cät"
iex(10)> String.codepoints(s)
["c", "a", "̈", "t"]
iex(11)> String.split(s, "", trim: true)
["c", "ä", "t"]
(note that the combining character prints very oddly when isolated inside ")
NobbZ
- bitstrings are everything what can be represented using
<<>> - binaries are bitstringths which length is a multiple of 8
- Strings are binaries which bytes represent valid utf-8 encoded codepoints.
First 2 terms are from Erlang
josevalim
FWIW, there is nowhere in the docs with that line. If they were, they would indeed be incorrect.
Other than that, you are right, codepoints are integers. String.codepoints returns codepoints as UTF-8 encoded binaries, i.e. codepoints as strings. The precise definition of this would be “code unit” but we wanted to avoid introducing yet another term. I have updated the docs to make it clear that String.codepoints returns an encoded representation, not integers. Thanks.
benwilson512
Just to make sure we’re on the same page, charlists and binaries are entirely different data structures. Charlists are lists of integers, where each integer is a valid code point. A charlist is an actual list through, in a is_list(list) #=> true sense. Binaries are not lists, even a little bit.
A bitstring is a fundamental data type, denoted with the syntax <<>>. It is a contiguous sequence of bits in memory.
A binary is a byte aligned bitstring, which is to say it is a sequence of whole bytes. They can be any bytes at all.
A string is a binary where all the bytes form a valid UTF8 sequence.
fireproofsocks
Thank you all for your input – this has been very educational. I have submitted a PR to the https://elixir-lang.org/getting-started/binaries-strings-and-char-lists.html page that I hope will better explain the concepts at play here.







