dogweather

dogweather

Possible bug in String.split/1's handling of Unicode whitespace?

According to the docs, String.split/1

Divides a string into substrings at each Unicode whitespace occurrence…

(emphasis mine)

But I found that it doesn’t do this for U+00A0 “NO-BREAK SPACE” (UTF-8 <<194, 160>>). Maybe others too? No-break space is unicode whitespace and it’s marked as such in the Elixir Unicode property list.

Here’s the unexpected behavior I see (Elixir 1.14):

iex(1)> nbsp = <<194, 160>>
" "
iex(2)> s = Enum.join ["one", "two", "three"], nbsp
"one two three"
iex(3)> String.split s
["one two three"]

I expected it to split on the nbsp, producing the input, ["one", "two", "three"].

I traced the code for String.split/1 to break() here: elixir/lib/elixir/unicode/unicode.ex at main · elixir-lang/elixir · GitHub. It looks ok to me, though.

Background: This caused problems for me parsing some text. I didn’t realize that this is valid Unicode because String.split/1 and Regex.run where failing on it (although for two different reasons).

Marked As Solved

brettbeatty

brettbeatty

The String.split/1 docs also have this line:

Divisions do not occur on non-breaking whitespace.

Also Liked

kip

kip

ex_cldr Core Team

The ‘break’ property and the ‘whitespace’ property are orthogonal in Unicode. Definitely significant overlap when looked at from the perspective of Latin scripts but definitely not universal - especially for languages that do not use whitespace between words (like Japanese, Chinese, Thai, Lao, Khmer, Myanmar scripts).

The elixir default for String.split/1 is “breaking whitespace”. By definition, “non breaking” characters should not break words, lines or sentences.

dogweather

dogweather

Urp! Thank you, I didn’t catch that.

Where Next?

Popular in Questions Top

Tee
can someone please explain to me how Enum.reduce works with maps
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
logicmason
Hi there, I'm working through my first release with elixir/phoenix. I've built a release with distillery and found that it crashes when I...
New
Kagamiiiii
Student &amp; New to elixir. Nice language. I want to convert a english character, e.g. “a”, which is stored in a variable, to it’s asci...
New
dokuzbir
Hello, I am trying to convert my lists to string without losing brackets.For start i have 3 map. They look like these buyer = %{ id: ...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
sabri
Can someone explain the settings of pool_size of Ecto in config file? and what is the recommend size? Thanks
New

Other popular topics Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1140 51847 244
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New

We're in Beta

About us Mission Statement