mayel
Mapping a language to a font charset
I use Gettext and ex_cldr for localisation, and would like to load a webfont in the appropriate charset based on the user’s locale (among which cyrillic, cyrillic-ext, devanagari, greek, greek-ext, latin, latin-ext, vietnamese, etc, see for example: arabic and google webfonts helper), rather than having files that include all of them (filesize can be 4x bigger, and also isn’t possible for all languages, some of which need their own font) and was wondering if ex_cldr has a way to map a language to a charset built in? (I couldn’t find it, but may have missed it due to the number of libraries)
Marked As Solved
kip
You can get the script for any locale:
iex> {:ok, locale} = Cldr.validate_locale("th")
{:ok, #Cldr.LanguageTag<th [validated]>}
iex> locale.script
:Thai
Which is a starting point. I will add a function Cldr.Locale.script_from_locale/1 in the next release to make this more explicit.
Now to mapping the script to a font name may be out of scope for ex_cldr but I’m curious so will do some more investigating too.
Also Liked
kip
After some more digging:
- I published ex_cldr version 2.31.0 which includes a new function
Cldr.Locale.script_from_locale/1to return the script. For example:Latnor:Thaior:Deva. - You can use the function
Cldr.Validity.known(:scripts)to get the name of all the 167 scripts known to CLDR. This function parses data so its only used at compile-time inex_cldrto build functions. Its not public API so use at your own risk (its not going away however).Cldr.Locale.language_data/0returns a lot of data that can be used to map between languages, scripts and territories its more complex data but more complete. - Google Fonts is really sloppy in its terminology using “charset” interchangeably with “subset” when it really means “script”. And it uses “language” when it means “script” too. Disappointing really since Google is the largest sponsor of CLDR.
- CLDR does have mappings from language to script so a lookup is possible and I’ll add a function to
ex_cldrto make that easy. But there’s another problem needs solving first … - The language names used in Google Fonts are not canonical and have no alignment with ISO 639 language codes making it difficult to map from
locale.scriptorlocal.languageto the right font “subset”.
Hopefully this gives you something to go on. Its an interesting and useful objective to deliver a typeface to a user that can represent the requested script. So I’m happy to help on the ex_cldr - just reply here or open an issue on GitHub.
kip
Maybe you’ll find my text library useful? It has a native Elixir implementation of text detection. You need to install also the text_corpus_udhr library to have a corpus to work with for detection:
iex> Text.detect """
Договоренность о разблокировке украинских портов для возобновления экспорта зерна остается труднодостижимой, потому что Москва использует переговоры для реализации своих военных целей и обеспечения доминирования в Черном море, заявил заместитель министра экономики и торговый представитель Украины Тарас Качка.
"""
{:ok, "ru"}
KungPaoChicken
It is possible to load (subsets of) a webfont dynamically in CSS by Unicode range, assuming that ranges for languages don’t change frequently:
Quote from the page:
For example, a site with many localizations could provide separate font resources for English, Greek and Japanese. For users viewing the English version of a page, the font resources for Greek and Japanese fonts wouldn’t need to be downloaded, saving bandwidth.
KungPaoChicken
I suspect it won’t be that straightforward, awesome library btw!
I tried to find out the length if I had to put all the ranges in CSS, and it is arguably okay (especially with gzip/brotli) if the script isn’t common or arabic.
[
common: 2363,
arabic: 781,
ethiopic: 474,
latin: 471,
greek: 450,
inherited: 385,
han: 276,
grantha: 223,
tamil: 204,
hangul: 180,
katakana: 179,
...
]
So mapping a language code to the right scripts to the right code points is not a trivial exercise.
Indeed. However, unless OP is creating their own font subsets, it is not necessary to check the relationship between scripts and codepoints (Noto variants did that for us), and the list of glyphs available in each font file can be used instead to generate the list of codepoint ranges, which I imagine could be automated with a FontForge script. Then I remembered where I first see this CSS attribute: Google Fonts. The CSS files they provide already have codepoint ranges for the fonts they offer:
kip
Oh, that seems straight forward indeed. Just need a mapping from the ISO 639 language and/or ISO 4217 territory code to the Google Fonts language and it seems good to go!
Was a fun excursion!







