pm100
\r\n as a grapheme is it 1 or 2?
I am assuming this is a Windows thing. Regex seems to treat \r\n as 2 characters but String treats them as one
trying to strip whitespace off the front of a string
iex(17)> regex = ~r/^\s+/
~r/^\s+/
iex(18)> junk = "\r\n\r\nabc"
"\r\n\r\nabc"
iex(19)> Regex.run(regex, junk, return: :index)
[{0, 4}]
iex(20)> String.slice(junk, 4..-1//1)
"c"
iex(21)> String.graphemes(juk)
["\r\n", "\r\n", "a", "b", "c"]
iex(22)>
Am I reading this correctly (absolute elixir noob). What the correct way to fix it?
Marked As Solved
sabiwara
As the Regex docs are mentioning, :index “returns byte index and match length”, not the grapheme index.
This works:
iex> binary_slice("\r\n\r\nabc", 4..-1//1)
"abc"
Bytes != codepoints != graphemes.
iex> junk = "\r\n\r\nabc é"
"\r\n\r\nabc é"
iex> for <<byte <- junk>>, do: <<byte>>
["\r", "\n", "\r", "\n", "a", "b", "c", " ", <<195>>, <<169>>]
iex> String.codepoints(junk)
["\r", "\n", "\r", "\n", "a", "b", "c", " ", "é"]
iex> String.graphemes(junk)
["\r\n", "\r\n", "a", "b", "c", " ", "é"]
The String documentation explains this topic well.
Also Liked
sabiwara
Stringis basically a set of helper functions forUTF8binaries
Exactly, it is the same underlying structure (binaries containing valid UTF8) but what changes is if you chose to process it as bytes, codepoints (UTF-8 characters) or graphemes (groups of codepoints which will be displayed as a single character to the human reader).
However this does not change a thing that only
Stringsees\r\nas a single character.
This depends of the function. Many functions of the String module are explicitly working with graphemes, such as String.length, String.at, String.first…
String.codepoints, String.next_codepoints, String.to_charlist, ::utf8 in bitstring patterns all work with codepoints.
It’s a serious gotcha that 2
UTF8characters are seen as 1
Graphemes and codepoints are arguably a gotcha in any language, so I’m not disagreeing, but I wouldn’t pin it on the String library 
LostKobrakai
No. Graphemes are a group construct. Sometimes a grapheme is just a single codepoint. Sometimes a grapheme consists of multiple codepoints – depending on how those codepoints follow up on each other.
iex(1)> str = "🏴☠️"
"🏴\u200D☠️"
iex(2)> String.graphemes(str)
["🏴\u200D☠️"]
iex(3)> String.codepoints(str)
["🏴", "\u200D", "☠", "️"]
iex(4)> str = "🏴"
"🏴"
iex(5)> String.graphemes(str)
["🏴"]
iex(6)> String.codepoints(str)
["🏴"]
And if you forget about “the computer in the back” \r\n and \n look the exact same to someone in notepad. It’s just us programmers still dealing with the fact that all this came from a typewrighter once. To everyone else it’s a newline no matter what.
adamu
FYI flags are multiple codepoints too.
They are the country code offset by 127397.
iex> for <<cp::utf8 <- "UN">>, do: <<(cp + 127397)::utf8>>, into: ""
"🇺🇳"
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"