Matthew

Matthew

Newbie with UnicodeConversionError

Hello, I am new to Elixir and so far finding it awesome. However I have just stumbled across a problem with non ascii strings. Please see below to see the error message I am getting:

iex(4)> "하이"
** (UnicodeConversionError) invalid encoding starting at <<199, 207, 192, 204, 34, 10>>
    (elixir) lib/string.ex:1801: String.to_charlist/1
iex(4)>

I note that I am using Windows 10 and running iex from a Cmder shell (the same error occurs in windows cmd and in powershell).

Any suggestions for how I can fix this? (Besides using developing on a linux system :wink: )

Most Liked

kip

kip

ex_cldr Core Team

Actually you can: String.codepoints/1 will decode what it can and show you the code points it can’t.

iex> x = <<160, 73, 100, 111, 119, 117, 32, 84, 97, 121, 108, 111, 114, 32, 83, 60, 47,
  99, 111, 110, 116, 97, 99, 116, 45, 97, 100, 100, 114, 101, 115, 115, 62, 10>>
iex> String.codepoints x
[
  <<160>>,
  "I",
  "d",
  "o",
  "w",
  "u",
  " ",
  "T",
  "a",
  "y",
  "l",
  "o",
  "r",
  " ",
  "S",
  "<",
  "/",
  "c", 
  "o",
  "n",
  "t",
  "a",
  "c",
  "t",
  "-",
  "a",
  "d",
  "d",
  "r",
  "e",
  "s",
  "s",
  ">",
  "\n"
]
michalmuskala

michalmuskala

While I have very little windows experience, I would expect this error to be caused by shell being not in unicode encoding.

In general any unicode issues on windows related to the shell were solved by using the graphical iex --werl shell.

Marcus

Marcus

You can try to use the raw representation of the string.

iex(30)> i "하이"
...
Raw representation
  <<237, 149, 152, 236, 157, 180>>
...
iex(31)> i <<237, 149, 152, 236, 157, 180>>
Term
  "하이"
...
Raw representation
  <<237, 149, 152, 236, 157, 180>>
...
iex(32)> str = <<237, 149, 152, 236, 157, 180>>
"하이"
iex(33)>

As you can see your string has just the wrong encoding.

iex(43)> String.to_charlist(<<199, 207, 192, 204, 34, 10>>)
** (UnicodeConversionError) invalid encoding starting at <<199, 207, 192, 204, 34, 10>>
    (elixir) lib/string.ex:1801: String.to_charlist/1
iex(43)>

Where Next?

Popular in Questions Top

lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
gonzofish
I’m currently trying to understand how to join three tables using Ecto. All the examples I’ve seen use 2, so maybe I’m just missing somet...
New
kostonstyle
Hi all I want to have a unix time, from the current time plus 1 hour. DateTime.now + 1 hour How to get it in elixir? Thanks
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New

Other popular topics Top

Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New

We're in Beta

About us Mission Statement