cblavier

cblavier

Converting hex chars to unicode equivalent

Hey there :wave:,

I fetch some HTML content with Req and extract plain text with Floki.

The retrieved text contains hex characters such as "Bient\xf4t" where I would expect "Bientôt" or the unicode equivalent "Bient\u00f4t"

I need the following test to pass:

"Bient\xf4t" |> some_magic() |> String.contains?("Bientôt") == true

Marked As Solved

hauleth

hauleth

In this particular case (where it is Latin1) you can use built in function instead of @kip’s code:

:unicode.characters_to_binary("Bient\xf4t", :latin1)

And you are good to go.

Also Liked

kip

kip

ex_cldr Core Team

\xf4 is a byte representation which Elixir understands, but in this example is not UTF8 encoded, which Elixir does expect. This type of representation happens in some languages when they only support ASCII but need to represent other encodings.

Something like this might help:

defmodule XDecode do
  def decode do
    "Bient\xf4t"
    |> String.chunk(:valid)
    |> decode_codepoints()
  end
  
  def decode_codepoints([utf8, codepoints | rest]) do
    utf8 <> List.to_string(:binary.bin_to_list(codepoints)) <> decode_codepoints(rest)
  end
  
  def decode_codepoints([utf8]) do
    utf8
  end
  
  def decode_codepoints([]) do
    ""
  end
end

As you would expect, this will only work if in fact the hex bytes resolve to valid Unicode code points. Otherwise, as @hauleth says, you need a more generalised converter.

hauleth

hauleth

You need to use some library that provides translation between different codepages (you also need to know what is the codepage of the input). If it is Latin1 then there is some stuff in OTP that can help you, but if it is other page, then you need to find `i

Where Next?

Popular in Questions Top

lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New

Other popular topics Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

We're in Beta

About us Mission Statement