DanielRS
How to replace accented letters with ASCII letters?
I’m implementing a blogging system in my website, and I’m trying to generate post slugs from the title, like this:
def slugify(string) do
string
|> String.normalize(:nfd)
|> String.replace(~r/[^A-z\s]/u, "")
|> String.replace(~r/\s/, "-")
end
If I try something like slugify("árboles más grandes") I get arboles-ms-grandes
Trying slugify("los árboles más grandes") returns los-rboles-ms-grandes
My slugify function seems to only work with accented letters at the start of the string.
Best regards,
Daniel Rivas.
Most Liked
KronicDeth
I’m not sure what is going on. When I tested it, it works:
iex> "árboles más grandes" |> String.normalize(:nfd) |> String.replace(~r/[^A-z\s]/u, "") |> String.replace(~r/\s/, "-")
"arboles-mas-grandes"
iex> "los árboles más grandes" |> String.normalize(:nfd) |> String.replace(~r/[^A-z\s]/u, "") |> String.replace(~r/\s/, "-")
"los-arboles-mas-grandes"
I got the test string by copying the string from your posting, so I don’t know if that changed the encoding.
hubertlepicki
@KronicDeth this does not seem to work on all systems, at least not on mine:
"los árboles más grandes" |> String.normalize(:nfd) |> String.replace(~r/[^A-z\s]/u, "") |> String.replace(~r/\s/, "-")
"los-rboles-ms-grandes"
There is unix library libiconv that does that. Erlang has a few wrappers, one can be installed from hex and is called iconv.
iex(1) > :application.start(:iconv)
:ok
iex(2) > :iconv.convert "utf-8", "ascii//translit", "Hubert Łępicki"
"Hubert Lepicki"
iex(3) > :iconv.convert "utf-8", "ascii//translit", "árboles más grandes"
"arboles mas grandes"
I think using iconv transliteration also replaces some of the national characters to recognized ascii replacements, I think in German ß is replaced with “ss” etc.
After you transliterated the string to most matching ascii equivalents, you can downcase it and replace whitespace with dashes.
kip
I’m late to this party but you might find unicode_transform useful. Its a rules-based transliterator which implements currently just a few of the many CLDR transliterations. Equally it might be both too much and too little for what you need.
The transformation rules for Latin to ASCII are quite comprehensive!
@jayjun’s excellent slugify package would be my general recommendation for slugification.
Examples
iex> Unicode.Transform.LatinAscii.transform "ü ø ß"
"u o ss"
iex> Unicode.Transform.LatinAscii.transform "árboles más grandes"
"arboles mas grandes"
iex> Unicode.Transform.LatinAscii.transform "Übel wütet der Gürtelwürger"
"Ubel wutet der Gurtelwurger"
iex> Unicode.Transform.LatinAscii.transform "Ł"
"L"
KronicDeth
Is it a requirement that you ascii-ify the slugs? There is support in modern browsers for non-ascii URLs being percent encoded in the HTML source, but displayed as the unicode characters as described on stackoverflow. Certain languages have words that can be confused with other words if you strip accents. I’m aware of it in Polish, I don’t know if it affects any of your target languages. If you’re sure you want to strip the accents, then you can disregard this and we’ll move on to a technical solution to the loss of accented characters.
hubertlepicki
@KronicDeth it’s String.normalize. I do not think it does what you think it does, I quite frankly do not understand what it should do. But it does not seem to convert UTF-8 national characters to matching ASCII ones at all on my system:
iex(10)> String.normalize "Łępicki", :nfd
"Łępicki"
iex(11)> "árboles más grandes" |> String.normalize(:nfd)
"árboles más grandes"
(And above is exactly what I see on my IEX terminal). I’m on Linux, en_US.UTF-8 LANG.







