linusdm

linusdm

How to use NimbleParsec to split text with a separator sequence?

I’m trying to parse a file with nimble_parsec but I got stuck quite early. The input is a file with many lines, and the content is divided with separators. A simplified example:

some text, may also contain character "=", which is part of the separator
can be any number of lines long until the first separator line
==========
next block of text
==========
and another one
==========

I can’t come up with a good combination of building blocks to get to the output:

["some text, may also contain some character \"=\", which is...", "next block of text", "and another one"]

I’m trying this:

defmodule MyParser do
  import NimbleParsec

  separator = string("==========")
  block = utf8_string([], min: 1) |> lookahead(separator)

  defparsec(:parse, block |> times(min: 1))
end

But this yields an error:

{:error, "expected string \"==========\"", "", %{}, {8, 210}, 210}

I think the key part of my problem is that the utf8_string parser is eagerly matching beyond the first separator. Fiddling with the accepted codepoints and excluding = does help when the text blocks do not contain the =, but that’s not how I want to use it (the text should be able to contain =, only a newline with a succession of = characters should break the text up).
I’m probably using lookahead wrong here, but I’m out of ideas :person_shrugging:

Marked As Solved

kip

kip

ex_cldr Core Team

First thing to note is that your separator is really \n=======\n - that is it is inclusive of the newlines. My instinct would be to reach for String.split/2, for example:

iex(1)> String.split("""
...(1)> some text, may also contain character "=", which is part of the separator
...(1)> can be any number of lines long until the first separator line
...(1)> ==========
...(1)> next block of text
...(1)> ==========
...(1)> and another one
...(1)> ==========
...(1)> """, ~r/\n=*\n/)
["some text, may also contain character \"=\", which is part of the separator\ncan be any number of lines long until the first separator line",
 "next block of text", "and another one", ""]

If you’re doing this as an exercise in NimbleParsec here’s a few thoughts:

  • the separator includes the surrounding newlines so its actually "\n==========\n"
  • logically a block is made up of one or more lines
  • a line is a sequence of characters bounded by a newline

Using these ideas we can build a parser which I think does what you want. I suspect this is slower that then String.split/2 version above.

  separator =
    string("\n==========\n")

  line =
    repeat(utf8_char([{:not, ?\n}]))

  block =
    line
    |> repeat(lookahead_not(separator) |> ascii_char([?\n]) |> concat(line))
    |> reduce({List, :to_string, []})

  defparsec(:parse, repeat(block |> ignore(separator)))

Also Liked

kip

kip

ex_cldr Core Team

I’ve added a gist for future reference with some comments (and also handling the case where the last block is not followed by a separator).

Where Next?

Popular in Questions Top

dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
New
mathew4509
I have a list say x = ["23gh", "56kh", "97mh"] I would like to pass each element to Val in each iteration. Say, in iteration 1 -------...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
chewm
Hi guys, nice to meet you to the whole forum, I’m new here, I’m trying to configure visual studio code for elixir, right now the intellis...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

sergio
I couldn’t find any guides that worked well with Phoenix 1.6.0 and esbuild. I hope this helps people test the waters and eases you into t...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
quazar
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New

We're in Beta

About us Mission Statement