cosmicrose

cosmicrose

How do you write nested and recursive NimbleParsec parsers?

I’m having trouble implementing a parser. Either it enters an infinite recursive loop, or it only parses the first part of the input and drops the rest.

For context, I’m writing a parser for a query language, and it contains the ability to nest boolean expressions. Here’s a stripped-down version of what I’m working with:

defmodule CESQL.Parsec do
  import NimbleParsec

  value_identifier = ascii_string([?a..?z], min: 1, max: 20)

  boolean_operation =
    parsec(:expr)
    |> ignore(string(" "))
    |> choice([
      string("AND"),
      string("OR"),
      string("XOR")
    ])
    |> ignore(string(" "))
    |> parsec(:expr)

  expr =
    choice([
      value_identifier,
      boolean_operation
    ])

  defparsec :expr, expr
end

My end goal is to parse a boolean expression, and I intend to allow for complex nested expressions like (a AND b) OR c. But for starters, this is what I want out of the above code:

iex> CESQL.Parsec.expr("a AND b")
{:ok, ["a", "AND", "b"], "", _, _, _}

However, when I place value_identifier first in expr’s argument to choice/2, the parser takes the first value and drops the rest of the string, which looks like this:

iex> CESQL.Parsec.expr("a AND b")
{:ok, ["a"], " AND b", _, _, _}

Alternatively, when I place boolean_operation as the first choice, I believe the parser enters an infinite loop trying to find the start of an expression, because my test times out.

How can I get this working the way I want it to? I’ve tried using NimbleParsec.lookahead/2 every way I can think of, but I might be misunderstanding how it works, because I’ve had no luck.

Most Liked

jakemorrison

jakemorrison

I did a lot of parsing of SQL in this project: GitHub - cogini/ecto_extract_migrations: Elixir library to generate Ecto migrations from a PostgreSQL schema SQL file. Uses NimbleParsec and macro-style code generation.

It converts a Postgres database schema dump file into Ecto migrations.

kip

kip

ex_cldr Core Team

Using Leex and Yecc is very workable (and what I used for the fundamentals of ex_cldr). But because parsers are fun, here’s a reasonable attempt at parsing your logical expressions in nimble_parsec using a common approach to de-structuring such parsers:

defmodule CESQL.Parsec do
  @moduledoc """
  Based upon the simple grammar of:

    Expression ⇒ Term {AND Term}
    Term ⇒ Factor {OR Factor}
    Factor ⇒ Item | "-" Factor
    Item ⇒ Identifier | "(" Expression ")"

  """
  import NimbleParsec

  whitespace = times(ascii_char([?\s, ?\t]), min: 1)

  # An expression
  defparsec(
    :expr,
    ignore(optional(whitespace))
    |> choice([
      parsec(:term) |> parsec(:op_and) |> parsec(:expr) |> reduce(:postfix),
      parsec(:term)
    ])
  )

  # A term
  defparsec(
    :term,
    choice([
      parsec(:factor) |> parsec(:op_or) |> parsec(:term) |> reduce(:postfix),
      parsec(:factor)
    ])
  )

  # A factor
  defparsec(
    :factor,
    choice([
      parsec(:identifier),
      ignore(ascii_char([?(]))
      |> ignore(optional(whitespace))
      |> parsec(:expr)
      |> ignore(optional(whitespace))
      |> ignore(ascii_char([?)]))
    ])
  )

  # OR operation
  defparsec(
    :op_or,
    ignore(whitespace) |> string("OR") |> ignore(whitespace)
  )

  # AND operation
  defparsec(
    :op_and,
    ignore(whitespace) |> string("AND") |> ignore(whitespace)
  )

  # An identifier (lower case letters)
  defparsec(
    :identifier,
    times(ascii_char([?a..?z]), min: 1) |> reduce({List, :to_string, []})
  )

  # Convert infix list to postfix for more regular "AST"
  def postfix([term_1, op, term_2]) do
    [op, term_1, term_2]
  end

  # Just pattern matching some examples
  def test do
    {:ok, [["AND", "a", "b"]], "", %{}, _, _} = expr("a AND b")
    {:ok, [["OR", "a", "b"]], "", %{}, _, _} = expr("a OR b")

    # precedence
    {:ok, [["AND", "a", ["OR", "b", "c"]]], "", %{}, _, _} = expr("a AND b OR c")
    {:ok, [["AND", ["OR", "a", "b"], "c"]], "", %{}, _, _} = expr("a OR b AND c")

    # nesting
    {:ok, [["OR", "a", ["AND", "b", "c"]]], "", %{}, _, _} = expr("a OR (b AND c)")

    :ok
  end
end
ityonemo

ityonemo

/Self-promotion but I also have pegasus: Pegasus — pegasus v0.2.2 if you like grammars that actually look like grammars

Where Next?

Popular in Questions Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
makeitrein
Hey all, just started picking up Elixir last week and am writing a scraper as a learning project. Baby step #1 is extracting the number ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
kostonstyle
Hi all I want to have a unix time, from the current time plus 1 hour. DateTime.now + 1 hour How to get it in elixir? Thanks
New
Exadra37
Sometimes I want to check if the input into a function is not a blank string. My first approach: defmodule Example do def do_stuff(s...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
Codball
Mix format works fine if run from the cmd. I’ve followed this to facilitate the implementation into VSC which involves downloading an ext...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
idi527
I’ve been re-reading swift book again and noticed that multiline strings there don’t have a trailing line break, unlike in elixir iex(2)...
New

Other popular topics Top

chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
yurko
Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for ...
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New

We're in Beta

About us Mission Statement