ijdickinson

ijdickinson

Detecting non-ASCII characters in a binary

As part of our data validation, I’d like to spot unexpected UTF8 characters in user supplied details. Some non-ASCII characters can be anticipated (é for example), but I’d like to spot unintended changes like UTF-8 single quote mark in place of ASCII ', invisible spaces, etc.

So the question I have is: what’s a good way to scan an Elixir string to detect characters that are in UTF8 but outside the range of ASCII (i.e. which would require File.open/2 to use :utf8 mode when writing)?

Marked As Solved

dwark

dwark

I had a use-case to simply detect non-ascii and used:

name != for(<<c <- name>>, c < 128, into: "", do: <<c>>)

which seems to do the trick. Didn’t need to be fast per se.

Also Liked

cevado

cevado

I have this library used to transliterate unicode to ascii… it’s based on the pearl and ruby version of it. from your question it seems to be good enough:
https://hexdocs.pm/unidecode/Unidecode.html

edit:
also if you’re going from utf-8 to a specific encoding. maybe using codepagex might work better:

ijdickinson

ijdickinson

Well I didn’t expect the Spanish Inquisition. It just happened that way. We’ve had six months of processing bulk customer data, which was all fine until it wasn’t. The IO.write is fixed, obviously, but the interesting part - to me - is that missing the :utf8 flag exposed an error case in the data pipeline that we hadn’t come across before. Production code has bugs sometimes. You find them, you fix them, and move on.

dimitarvp

dimitarvp

Heh. :smiley: Consider that a good amount of posters arrive here with their variant of the XY problem so we do our best to establish a good foundation for the discussion – which means dispelling potential myths and bad practices from the get go, because they usually prolong the discussion and tend to make it unfocused. Nobody is criticizing you in particular or saying that real code doesn’t have problems – of course it has.

Hindsight, 20/20, and all that.

Hope we were helpful.

dimitarvp

dimitarvp

You can just make an allow-list of characters (since you said you are not only after ASCII characters but a few more as well), open a file in :utf8 mode and then work on each character via the in operator and the String.codepoints function. Seems easy.

dwark

dwark

In addition to @dimitarvp 's reply, when rolling your own maybe this
String.valid? thread could be helpful.

Where Next?

Popular in Questions Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement