ijdickinson
Detecting non-ASCII characters in a binary
As part of our data validation, I’d like to spot unexpected UTF8 characters in user supplied details. Some non-ASCII characters can be anticipated (é for example), but I’d like to spot unintended changes like UTF-8 single quote mark in place of ASCII ', invisible spaces, etc.
So the question I have is: what’s a good way to scan an Elixir string to detect characters that are in UTF8 but outside the range of ASCII (i.e. which would require File.open/2 to use :utf8 mode when writing)?
Marked As Solved
dwark
I had a use-case to simply detect non-ascii and used:
name != for(<<c <- name>>, c < 128, into: "", do: <<c>>)
which seems to do the trick. Didn’t need to be fast per se.
Also Liked
cevado
I have this library used to transliterate unicode to ascii… it’s based on the pearl and ruby version of it. from your question it seems to be good enough:
https://hexdocs.pm/unidecode/Unidecode.html
edit:
also if you’re going from utf-8 to a specific encoding. maybe using codepagex might work better:
ijdickinson
Well I didn’t expect the Spanish Inquisition. It just happened that way. We’ve had six months of processing bulk customer data, which was all fine until it wasn’t. The IO.write is fixed, obviously, but the interesting part - to me - is that missing the :utf8 flag exposed an error case in the data pipeline that we hadn’t come across before. Production code has bugs sometimes. You find them, you fix them, and move on.
dimitarvp
Heh.
Consider that a good amount of posters arrive here with their variant of the XY problem so we do our best to establish a good foundation for the discussion – which means dispelling potential myths and bad practices from the get go, because they usually prolong the discussion and tend to make it unfocused. Nobody is criticizing you in particular or saying that real code doesn’t have problems – of course it has.
Hindsight, 20/20, and all that.
Hope we were helpful.
dimitarvp
You can just make an allow-list of characters (since you said you are not only after ASCII characters but a few more as well), open a file in :utf8 mode and then work on each character via the in operator and the String.codepoints function. Seems easy.
dwark
In addition to @dimitarvp 's reply, when rolling your own maybe this
String.valid? thread could be helpful.







