scouten

scouten

Unicode-savvy `letter_or_digit?` function?

I’m parsing a file format by recursing on a charlist. One of the challenges I have in parsing this format is determining whether the current Unicode codepoint is a letter or digit. So far I’ve written this:

# HELP: This is not Unicode-savvy. Is there such a thing?
defp letter_or_digit?(c) when c >= ?0 and c <= ?9, do: true
defp letter_or_digit?(c) when c >= ?A and c <= ?Z, do: true
defp letter_or_digit?(c) when c >= ?a and c <= ?z, do: true
defp letter_or_digit?(_), do: false

This is called via Enum.split_while(&letter_or_digit?/1) and similar constructions.

What I’ve written is obviously not savvy about non-ASCII letters or digits. Is there such a thing handy? My Google-fu has failed me so far.

The obvious answer would be to parse via Regex match. Unfortunately, some other parts of the file format are far better consumed via charlist recursion and I don’t want to pay the cost of bouncing back and forth between strings and charlists.

Most Liked

Nicd

Nicd

Try Erlang regular expressions for charlists instead, I think this is what hauleth meant: http://erlang.org/doc/man/re.html

You probably want to use regex to match for character property Number.

kip

kip

ex_cldr Core Team

I’ve just published the first version of ex_cldr_unicode that might be helpful to you. It builds functions at compile time using data from the Unicode database.

Theres a bunch of fun stuff you can do, but to your use case it includes some guards that you may find useful. For example:

defmodule MyModule do
  require Cldr.Unicode.Guards
  alias Cldr.Unicode.Guards

  def my_function(codepoint) when Guards.is_upper(codepoint) do
    IO.puts "Its Uppercase!"
  end

See hex docs for further info.

I’ve defined only the following guards so far but it’s trivial to add more so let me know if its useful:

  • is_upper
  • is_lower
  • is_digit
  • is_currency_symbol

Oh, and it’s more than twice as fast as using a regex for this kind of matching.

There are a bunch of classifier functions as well. From my understanding of your use case the following may also apply:

iex> Cldr.Unicode.Property.alphanumeric? "1234"
true
iex> Cldr.Unicode.Property.alphanumeric? "KeyserSöze1995"
true
iex> Cldr.Unicode.Property.alphanumeric? "3段"
true
iex> Cldr.Unicode.Property.alphanumeric? "dragon@example.com"
false
kip

kip

ex_cldr Core Team

Updated to version 0.2.0. Main changes are:

  • Moves the public API to the Cldr.Unicode module.

  • Updates and adds documentation to all public functions.

  • Removes the text annotations from the compiled functions which materially reduces the size of the beam files.

Feedback welcome as are feature requests and PRs.

hauleth

hauleth

  1. You absolutely can run regexes on charlists as in Erlang charlists are the strings.
  2. If you want to check if code point is digit, then here you have full list, you need to check all these groups. For rest of the character groups check here.

BTW you do not need to change binary (as assume that you get binary from the file) to iterate over it, you can pattern match on binary fragments as well by <<byte, rest :: binary>> = "abba", byte == ?a and rest == "bba".

scouten

scouten

I think you’re mistaken. Strings and charlists are not the same thing. Just to confirm:

iex(1)> x = '1234'  # note: single-quote to make it a charlist, not a binary / string
'1234'
iex(2)> Regex.match?(~r/23/, x)
** (FunctionClauseError) no function clause matching in Regex.match?/2    
    
    The following arguments were given to Regex.match?/2:
    
        # 1
        ~r/23/
    
        # 2
        '1234'
    
    Attempted function clauses (showing 1 out of 1):
    
        def match?(%Regex{re_pattern: compiled}, string) when is_binary(string)
    
    (elixir) lib/regex.ex:231: Regex.match?/2

Where Next?

Popular in Questions Top

Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
jc00ke
Expanding on this topic: https://forum.elixirforum.net/t/map-typespec-question/19217 Let’s say I have a map with required and optional k...
New

Other popular topics Top

rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement