mayel

mayel

Mapping a language to a font charset

I use Gettext and ex_cldr for localisation, and would like to load a webfont in the appropriate charset based on the user’s locale (among which cyrillic, cyrillic-ext, devanagari, greek, greek-ext, latin, latin-ext, vietnamese, etc, see for example: arabic and google webfonts helper), rather than having files that include all of them (filesize can be 4x bigger, and also isn’t possible for all languages, some of which need their own font) and was wondering if ex_cldr has a way to map a language to a charset built in? (I couldn’t find it, but may have missed it due to the number of libraries)

Marked As Solved

kip

kip

ex_cldr Core Team

You can get the script for any locale:

iex> {:ok, locale} = Cldr.validate_locale("th")
{:ok, #Cldr.LanguageTag<th [validated]>}
iex> locale.script
:Thai

Which is a starting point. I will add a function Cldr.Locale.script_from_locale/1 in the next release to make this more explicit.

Now to mapping the script to a font name may be out of scope for ex_cldr but I’m curious so will do some more investigating too.

Also Liked

kip

kip

ex_cldr Core Team

After some more digging:

  • I published ex_cldr version 2.31.0 which includes a new function Cldr.Locale.script_from_locale/1 to return the script. For example :Latn or :Thai or :Deva.
  • You can use the function Cldr.Validity.known(:scripts) to get the name of all the 167 scripts known to CLDR. This function parses data so its only used at compile-time in ex_cldr to build functions. Its not public API so use at your own risk (its not going away however). Cldr.Locale.language_data/0 returns a lot of data that can be used to map between languages, scripts and territories its more complex data but more complete.
  • Google Fonts is really sloppy in its terminology using “charset” interchangeably with “subset” when it really means “script”. And it uses “language” when it means “script” too. Disappointing really since Google is the largest sponsor of CLDR.
  • CLDR does have mappings from language to script so a lookup is possible and I’ll add a function to ex_cldr to make that easy. But there’s another problem needs solving first …
  • The language names used in Google Fonts are not canonical and have no alignment with ISO 639 language codes making it difficult to map from locale.script or local.language to the right font “subset”.

Hopefully this gives you something to go on. Its an interesting and useful objective to deliver a typeface to a user that can represent the requested script. So I’m happy to help on the ex_cldr - just reply here or open an issue on GitHub.

kip

kip

ex_cldr Core Team

Maybe you’ll find my text library useful? It has a native Elixir implementation of text detection. You need to install also the text_corpus_udhr library to have a corpus to work with for detection:

iex> Text.detect """
Договоренность о разблокировке украинских портов для возобновления экспорта зерна остается труднодостижимой, потому что Москва использует переговоры для реализации своих военных целей и обеспечения доминирования в Черном море, заявил заместитель министра экономики и торговый представитель Украины Тарас Качка.
"""
{:ok, "ru"}
KungPaoChicken

KungPaoChicken

It is possible to load (subsets of) a webfont dynamically in CSS by Unicode range, assuming that ranges for languages don’t change frequently:

Quote from the page:

For example, a site with many localizations could provide separate font resources for English, Greek and Japanese. For users viewing the English version of a page, the font resources for Greek and Japanese fonts wouldn’t need to be downloaded, saving bandwidth.

KungPaoChicken

KungPaoChicken

I suspect it won’t be that straightforward, awesome library btw!
I tried to find out the length if I had to put all the ranges in CSS, and it is arguably okay (especially with gzip/brotli) if the script isn’t common or arabic.

[
  common: 2363,
  arabic: 781,
  ethiopic: 474,
  latin: 471,
  greek: 450,
  inherited: 385,
  han: 276,
  grantha: 223,
  tamil: 204,
  hangul: 180,
  katakana: 179,
  ...
]

So mapping a language code to the right scripts to the right code points is not a trivial exercise.

Indeed. However, unless OP is creating their own font subsets, it is not necessary to check the relationship between scripts and codepoints (Noto variants did that for us), and the list of glyphs available in each font file can be used instead to generate the list of codepoint ranges, which I imagine could be automated with a FontForge script. Then I remembered where I first see this CSS attribute: Google Fonts. The CSS files they provide already have codepoint ranges for the fonts they offer:

kip

kip

ex_cldr Core Team

Oh, that seems straight forward indeed. Just need a mapping from the ISO 639 language and/or ISO 4217 territory code to the Google Fonts language and it seems good to go!

Was a fun excursion!

Where Next?

Popular in Questions Top

aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
pgiesin
This should be a simple problem but I just can’t seem to figure it out. I have a standalone Elixir app that won’t find the database. Dep...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
sabri
Can someone explain the settings of pool_size of Ecto in config file? and what is the recommend size? Thanks
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
idi527
I’ve been re-reading swift book again and noticed that multiline strings there don’t have a trailing line break, unlike in elixir iex(2)...
New

Other popular topics Top

dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement