makeitrein

makeitrein

Extracting numbers from a string

Hey all, just started picking up Elixir last week and am writing a scraper as a learning project.

Baby step #1 is extracting the number from a URL on the target web page… here’s what I’ve written:

# url is in "https://xxyyzz.com/xxyyzz.383254/" format... goal is to extract 383254
  def get_id_from_url(url), do: Regex.run(~r"\d+\/", url) |> Enum.at(0) |> Integer.parse |> elem(0)

This seems a bit clunky of a function to me for a simple integer extraction… is there a better way of going about this?

Marked As Solved

dimitarvp

dimitarvp

(EDIT 1: Account for invalid values.)
(EDIT 2: Trim empty strings when splitting.)
(EDIT 3: Included explanations.)

defmodule Test do
  def extract_id(url) when is_binary(url) do
    url
    |> String.split(~w(. /), parts: 1000, trim: true)
    |> List.last
    |> parse_id
    |> fetch_id
  end

  defp parse_id(nil), do: :error
  defp parse_id(x) when is_binary(x), do: Integer.parse(x)

  defp fetch_id({number, ""}) when is_integer(number), do: number
  defp fetch_id(:error), do: :error
end

Test it:

iex> urls = ["https://foster.com/death-pancake.1468/", "https://hkd33.net/mr-rogers101.690153/", "whatever_dude", "https://space-force911.gov/sauce-master.13257777/"]

iex> urls |> Enum.map(&Test.extract_id/1)
[1468, 690153, :error, 13257777]

Breaking it down:

  • ~w(. /) equals [".", "/"] (so String.split is called with multiple separators).

  • parts: 1000 is used to prevent denial-of-service attacks, in case somebody manages to smuggle huge strings to your code. trim: true removes empty strings from the result. Check String.split docs.

  • "https://foster.com/death-pancake.1468/" |> String.split(~w(. /), parts: 1000, trim: true) yields this:

["https:", "foster", "com", "death-pancake", "1468"]

…so we are calling List.last on it to give us the desirable piece of data.

  • Our internal function parse_id has to also handle invalid data:
    • If String.split returns [], List.last would return nil.
    • If String.split returns ["single_invalid_url"], List.last would return "single_invalid_url".

Both cases would make our internal function parse_id to return :error. (Integer.parse will return :error if you supply it a string that does NOT start with an integer.)

  • The fetch_id internal function uses function heads instead of if or case to extract successful integer parsing and return it, or react to an :error return value and just pass it down the line to your consumer code.

  • One caveat: notice that fetch_id matches on {number, ""} when is_integer(number) which means the function will be called only if a full integer string is passed, namely “123” or “456” will succeed but “123xyz” will not. If you expect URLs like “https://whatever.man/1234abcd”, this code won’t work.

Also Liked

hassan

hassan

How about

iex(10)> "https://xxyyzz.com/xxyyzz.383254/" |> String.replace(~r/[^\d]/, "")
"383254"
iex(11)>
dimitarvp

dimitarvp

defmodule Test do
  def match_string("https://xxyyzz.com/xxyyzz." <> suffix) do
    case Integer.parse(suffix) do
      {number, "/"} when is_integer(number) ->
        IO.puts "suffix is #{number}"

      _ ->
        IO.puts "cannot parse suffix: #{suffix}"
    end
  end
end

Test it in iex:

iex> Test.match_string "https://xxyyzz.com/xxyyzz.383254/"
suffix is 383254
:ok
iex> Test.match_string "https://xxyyzz.com/xxyyzz.383254/!"
cannot parse suffix: 383254/!
:ok

You can abuse Elixir’s allowed syntax of pattern matching on a string suffix (you cannot pattern-match strings in the middle of the bigger string though, have that in mind). Not sure if I am not taking your example too literally but if I understood you correctly, that’s how I would approach the problem.

NobbZ

NobbZ

Now as we have more information, I have an alternative version which I prefer over @dimitarvp, because it is much more explicit about what we want.

  • It says that we want an URL and verifies we get one (by parsing it) and that we are only interested in the path,
  • it says that we are searching for dot, followed by at least one digit and ending with a slash as the last character of the path, but we are only interested in the actual digits (the call to Regex.named_captures/3),
  • we want those digits to cleanly parse into a number.

If all succeed, we return an :ok-tuple, and simply :error otherwise.

But which version to choose is probably a matter of taste, I have not benchmarked them.

defmodule M do
  def extract(url) do
    with %URI{path: path} when is_binary(path) <- URI.parse(url),
         %{"num" => num_str} <- Regex.named_captures(~r[\.(?<num>\d+)/$], path),
         {num, ""} <- Integer.parse(num_str) do
      {:ok, num}
    else
      _ -> :error
    end
  end
end

IO.inspect M.extract("https://foster.com/death-pancake.1468/")
IO.inspect M.extract("https://hkd33.net/mr-rogers101.690153/")
IO.inspect M.extract("https://space-force911.gov/sauce-master.13257777/")
NobbZ

NobbZ

You can use hd instead of Enum.at. that saves you some characters.

Also you can use String.to_integer instead of piping through Integer.parse and elem.

But that’s the way to go.

dimitarvp

dimitarvp

Nope! We have enough borked gov’t systems. Let’s do better.

If you give us a few examples and/or explain the whole URL schemata then I can help you better.

Where Next?

Popular in Questions Top

lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
sacepums
Hey guys. I'm new to elixir and im really stocked about it. But I ran into a bit of problem - I need to convert a date sting, for examp...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New

Other popular topics Top

chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New

We're in Beta

About us Mission Statement