DanielRS

DanielRS

How to replace accented letters with ASCII letters?

I’m implementing a blogging system in my website, and I’m trying to generate post slugs from the title, like this:

def slugify(string) do
    string
    |> String.normalize(:nfd)
    |> String.replace(~r/[^A-z\s]/u, "")
    |> String.replace(~r/\s/, "-")
end

If I try something like slugify("árboles más grandes") I get arboles-ms-grandes

Trying slugify("los árboles más grandes") returns los-rboles-ms-grandes

My slugify function seems to only work with accented letters at the start of the string.

Best regards,
Daniel Rivas.

Most Liked

KronicDeth

KronicDeth

I’m not sure what is going on. When I tested it, it works:

iex> "árboles más grandes" |> String.normalize(:nfd) |> String.replace(~r/[^A-z\s]/u, "") |> String.replace(~r/\s/, "-")
"arboles-mas-grandes"
iex> "los árboles más grandes" |> String.normalize(:nfd) |> String.replace(~r/[^A-z\s]/u, "") |> String.replace(~r/\s/, "-")
"los-arboles-mas-grandes"

I got the test string by copying the string from your posting, so I don’t know if that changed the encoding.

hubertlepicki

hubertlepicki

@KronicDeth this does not seem to work on all systems, at least not on mine:

"los árboles más grandes" |> String.normalize(:nfd) |> String.replace(~r/[^A-z\s]/u, "") |> String.replace(~r/\s/, "-")
"los-rboles-ms-grandes" 

There is unix library libiconv that does that. Erlang has a few wrappers, one can be installed from hex and is called iconv.

iex(1) > :application.start(:iconv)
:ok
iex(2) > :iconv.convert "utf-8", "ascii//translit", "Hubert Łępicki"
"Hubert Lepicki"
iex(3) > :iconv.convert "utf-8", "ascii//translit", "árboles más grandes"
"arboles mas grandes"

I think using iconv transliteration also replaces some of the national characters to recognized ascii replacements, I think in German ß is replaced with “ss” etc.

After you transliterated the string to most matching ascii equivalents, you can downcase it and replace whitespace with dashes.

kip

kip

ex_cldr Core Team

I’m late to this party but you might find unicode_transform useful. Its a rules-based transliterator which implements currently just a few of the many CLDR transliterations. Equally it might be both too much and too little for what you need.

The transformation rules for Latin to ASCII are quite comprehensive!

@jayjun’s excellent slugify package would be my general recommendation for slugification.

Examples

iex> Unicode.Transform.LatinAscii.transform "ü ø ß"
"u o ss"

iex> Unicode.Transform.LatinAscii.transform "árboles más grandes" 
"arboles mas grandes"

iex> Unicode.Transform.LatinAscii.transform "Übel wütet der Gürtelwürger"
"Ubel wutet der Gurtelwurger"

iex> Unicode.Transform.LatinAscii.transform "Ł"                  
"L"
KronicDeth

KronicDeth

Is it a requirement that you ascii-ify the slugs? There is support in modern browsers for non-ascii URLs being percent encoded in the HTML source, but displayed as the unicode characters as described on stackoverflow. Certain languages have words that can be confused with other words if you strip accents. I’m aware of it in Polish, I don’t know if it affects any of your target languages. If you’re sure you want to strip the accents, then you can disregard this and we’ll move on to a technical solution to the loss of accented characters.

hubertlepicki

hubertlepicki

@KronicDeth it’s String.normalize. I do not think it does what you think it does, I quite frankly do not understand what it should do. But it does not seem to convert UTF-8 national characters to matching ASCII ones at all on my system:

iex(10)> String.normalize "Łępicki", :nfd
"Łępicki"
iex(11)> "árboles más grandes" |> String.normalize(:nfd)
"árboles más grandes"

(And above is exactly what I see on my IEX terminal). I’m on Linux, en_US.UTF-8 LANG.

Where Next?

Popular in Questions Top

pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
sacepums
Hey guys. I'm new to elixir and im really stocked about it. But I ran into a bit of problem - I need to convert a date sting, for examp...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
mathew4509
I have a list say x = ["23gh", "56kh", "97mh"] I would like to pass each element to Val in each iteration. Say, in iteration 1 -------...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Other popular topics Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 45766 226
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement