dogweather

Code critique: group_after/2 for parsing flat HTML

My problem

When parsing old government webpages, my input is often just like that one:

<p><b>Section 1, name, and text</b></p>
<p>Section 1 more text</p>
<p>Section 1 more text</p>
<p><b>Section 2, name, and text</b></p>
<p>Section 2 more text</p>
<p><b>Section 3,  name, and text</b></p>
// etc.

I’d really like feedback about the approach I came up with last night:

This looks like a take/while/scan kind of problem. But I couldn’t find an Enum or Floki function that seemed to handle this kind of repeated pattern.

I decided to write a function that would generically group items and following items using a predicate. In the case of the HTML above, the predicate would be “Does the element contain a <b>?” So, abstractly:

input =  [1,2,2,2,2,1,1,1,2]
output = [[1,2,2,2,2], [1], [1], [1, 2]]

I realized that’s not too hard with Enum.reduce:

  def group_after(list, predicate) do
    reduce(list, [], fn e, acc ->
      case predicate.(e) do
        true ->
          [[e]] ++ acc

        false ->
          {curr, rest} = List.pop_at(acc, 0)
          [curr ++ [e] | rest]
      end
    end)
  end

It works fine. Although, the reduce function’s code is very procedural and not expressive. What do you all think? Is there another approach I’m not considering?

An alternate idea: Consider a string "tfffftttf" as an isomorph of map(list, predicate). Then use an expressive regex like ~r/tf*/ to group the true & false — instead of the procedural reduce. Finally, undo the mapping back into the original list elements.

9 comments

#floki

5 894 9

2022-09-25 22:35:13 UTC

Most Liked

derek-zhou

This is not too bad; at least all the <p> have closing </p>. You know they don’t have to; and I;ve seen html that freely mix the 2 styles, with or without closing </p>

Post #8

Eiji

Slow load (possibly timeout). I know what you feel. Hope those are not ASP.net pages with invalid HTML code.

Anyway, here is my solution:

Mix.install([:floki])

defmodule Example do
  def sample(list, acc \\ [])

  # for empty input after parsing
  def sample([], []), do: []

  # when all p elements are passed
  # reverse last section texts and wrap them into list
  # as otherwise a resulting list would be added
  # to a main list where each element contains a list of sections texts
  def sample([], acc), do: [:lists.reverse(acc)]

  # in case of first bold text simply add text to
  # as the only element in new acc
  # and call function recursively
  def sample([{"p", _, [{"b", [], [text]}]} | tail], []) when is_binary(text) do
    sample(tail, [text])
  end

  # however if there is some data in acc
  # reverse its contents and return it as a list of
  # last section texts and recursive call
  def sample([{"p", _, [{"b", [], [text]}]} | tail], acc) when is_binary(text) do
    [:lists.reverse(acc) | sample(tail, [text])]
  end

  # when we got a normal text simply add it to acc
  # and call function recursively
  def sample([{"p", _, [text]} | tail], acc) when is_binary(text) do
    sample(tail, [text | acc])
  end
end

"""
<p><b>Section 1, name, and text</b></p>
<p>Section 1 more text</p>
<p>Section 1 more text</p>
<p><b>Section 2, name, and text</b></p>
<p>Section 2 more text</p>
<p><b>Section 3,  name, and text</b></p>
"""
|> Floki.parse_fragment!()
|> Example.sample()
|> dbg()

Pattern matching is a fastest solution. You can take a look at this post to see possible alternative solutions.

Post #3

dogweather

Thanks! That’s pretty interesting. This meta tag is pretty chilling.

<meta name=Generator content="Microsoft Word 15 (filtered)">

It seems they’re somehow scripting MS Word to “Save as HTML”.

Post #5

Eiji

Since you did not give a struct definition I wrote it myself:

Mix.install([:floki])

defmodule Section do
  defstruct ~w[contents id title]a

  def add_text(%__MODULE__{contents: contents} = section, {"span", _, [text]})
      when is_binary(text) do
    %{section | contents: [String.trim(text) | contents]}
  end

  def first_paragraph({"b", _, [{"span", _, [id_title]}]}, {"span", _, [text]})
      when is_binary(id_title) and is_binary(text) do
    [id, title] = id_title |> String.trim() |> String.split("\n", parts: 2)
    %Section{contents: [String.trim(text)], id: id, title: title}
  end

  def reverse_contents(%__MODULE__{contents: contents} = section) do
    %{section | contents: :lists.reverse(contents)}
  end
end

defmodule Example do
  def sample(list, section \\ nil)

  def sample([], []), do: []

  def sample([], section), do: [Section.reverse_contents(section)]

  def sample([{"p", _, [title, text]} | tail], nil) do
    sample(tail, Section.first_paragraph(title, text))
  end

  def sample([{"p", _, [title, text]} | tail], section) do
    new_section = Section.first_paragraph(title, text)
    [Section.reverse_contents(section) | sample(tail, new_section)]
  end

  def sample([{"p", _, [text]} | tail], section) do
    sample(tail, Section.add_text(section, text))
  end
end

"""
<p class=MsoNormal style='margin-bottom:0in;line-height:normal;text-autospace:
none'><b><span style='font-size:12.0pt;font-family:"Times New Roman",serif'>      838.025
Election laws apply.</span></b><span style='font-size:12.0pt;font-family:"Times New Roman",serif'>
(1) ORS chapter 255 governs the following:</span></p>

<p class=MsoNormal style='margin-bottom:0in;line-height:normal;text-autospace:
none'><span style='font-size:12.0pt;font-family:"Times New Roman",serif'>      (a)
The nomination and election of district board members.</span></p>

<p class=MsoNormal style='margin-bottom:0in;line-height:normal;text-autospace:
none'><span style='font-size:12.0pt;font-family:"Times New Roman",serif'>      (b)
The conduct of district elections.</span></p>

<p class=MsoNormal style='margin-bottom:0in;line-height:normal;text-autospace:
none'><span style='font-size:12.0pt;font-family:"Times New Roman",serif'>      (2)
The electors of a district may exercise the powers of the initiative and
referendum regarding a district measure, in accordance with ORS 255.135 to
255.205. [Formerly 494.043]</span></p>
"""
|> Floki.parse_fragment!()
|> Example.sample()
|> dbg()

What do you think about it?

Post #7