neuone

neuone

Floki parse and group by heading

The HTML has no attributes it’s very vanilla and plain

<!doctype html>
<html>
<body>
  <section class="body-copy">
  <h2>Topic 1</h2>
  <p>data-a</p>
  <p>data-b</p>
  <p>data-c</p>
  <h2>Topic 2</h2>
  <p>data-d</p>
  <p>data-e</p>
  <p>data-f</p>
  <h2>Topic 3</h2>
  <p>data-g</p>
  <p>data-h</p>
  <p>data-i</p>
  </section>
</body>
</html>

I’m using Floki and I’m trying to parse it so I can create a List of maps like so.

%{ topic: "Topic 1", data: "data-a" }
%{ topic: "Topic 1", data: "data-b" }
%{ topic: "Topic 1", data: "data-c" }
%{ topic: "Topic 2", data: "data-d" }
%{ topic: "Topic 2", data: "data-e" }
%{ topic: "Topic 2", data: "data-f" }

I’m struggling to get all the P tags under each H2 with Floki.

# Try loading this html
path = "/Users/Foo/Desktop/test.html"
{_, local_file } = File.read(path)

# This will return me all the h2
Floki.find(local_file, "h2")               
[{"h2", [], ["Topic 1"]}, {"h2", [], ["Topic 2"]}, {"h2", [], ["Topic 3"]}]

# This will return me the first p from a specific h2. But not all of them
Floki.find(local_file, "h2:nth-of-type(1) + p")
[{"p", [], ["data-a"]}]

# This return this first p for Topic 2 but I need 2 more p tags (data-e, data-f)
Floki.find(local_file, "h2:nth-of-type(2) + p")
[{"p", [], ["data-d"]}]

# This return this first p for Topic 3 but I need 2 more p tags (data-h, data-i)
Floki.find(local_file, "h2:nth-of-type(3) + p")
[{"p", [], ["data-g"]}]

Question
I cannot figure out how to get ONLY the P tags for each H2.

Marked As Solved

mischov

mischov

This is brittle as all get-out because it makes some serious assumptions about the shape of the data, but:

def extract_topic_maps(html) do
  nodes = Floki.find(html, "section.body-copy > *")

  {_, topic_maps} =
    Enum.reduce(nodes, {nil, []}, fn
      {"p", _, _}, {nil, _} -> raise "Invalid state: no topic"
      # New topic, set in accumulator
      {"h2", _, [topic]}, {_, topic_maps} -> {topic, topic_maps}
      # New value in topic, add appropriate topic_map to topic_maps in accumulator
      {"p", _, [data]}, {topic, topic_maps} -> {topic, [%{topic: topic, data: data} | topic_maps]}
      node, _ -> raise "Invalid state: unexpected node #{inspect(node)}"
    end)

  Enum.reverse(topic_maps)
end

With your input that returns something like

[
  %{data: "data-a", topic: "Topic 1"},
  %{data: "data-b", topic: "Topic 1"}, 
  %{data: "data-c", topic: "Topic 1"},
  %{data: "data-d", topic: "Topic 2"},
  ...
]

Also Liked

mischov

mischov

The problem is that the data is not actually grouped by anything other intent and order, which CSS selectors don’t do well with.

If each item looked like <div><h2>Header</h2><p>Item 1</p><p>Item 2</p></div> then trying to select with CSS selectors would be a lot easier because that’s the type of data those selectors work best with.

Here are a couple ways you could solve the problem (based on the example data):

  1. If there are an identical number of p tags after each h2 tag, you could get a list of all the children of the <section> and do a Enum.chunk_by to group the nodes that belong together and go from there.

  2. If there aren’t an identical number of p tags after each h2 tag or there are other unwanted children, you could get a list of all the children of the <section> and write a reducer that builds up a map of h2 element to a list of the sections that occur before the next h2, and go from there.

There might be better solutions I’m not thinking of, but those are places where you could start.

idi527

idi527

Maybe

def grouped_by_topics(body_nodes) do
  # nil here is a hack, you can run this function in two phases instead
  # 1. find the first `h1` with a topic
  # 2. then start this function with the rest of `body_nodes`
  grouped_by_topics(body_nodes, nil, [], [])
end

# when we meet a `h1` tag, start a new `inner_acc` for collecting the data for the topic in `h1`
defp grouped_by_topics([{"h1", [], ["Topic" <> _ = next_topic]} | rest], prev_topic, prev_inner_acc, outer_acc) do
  grouped_by_topics(rest, next_topic, [], [%{prev_topic => prev_inner_acc} | outer_acc])
end 

# when we meet a new `p` tag, add it to the `inner_acc` for the current topic
defp grouped_by_topics([{"p", [], ["data" <> _ = new_data]} | rest], current_topic, inner_acc, outer_acc) do
  grouped_by_topics(rest, current_topic, [new_data | inner_acc], outer_acc)
end

# neither a `p` nor an `h1` tag -- skip
defp grouped_by_topics([_other | rest], current_topic, inner_acc, outer_acc) do
  grouped_by_topics(rest, current_topic, inner_acc, outer_acc)
end

# no more html nodes -- finish
defp grouped_by_topics([], last_topic, inner_acc, outer_acc) do
  [%{last_topic => inner_acc} | outer_acc]
end

inner_acc is for collecting data-* in p tags
outer_acc is for collecting %{topic => data (aka final_inner_acc)} maps
current_topic is for keeping the topic from the last h2 tag

mischov

mischov

Simply, reducers are functions that take 1) the current item from the collection being reduced over, and 2) the current accumulator, and that return a value that will be used as the accumulator in the next call of the reducer (for the next item in the collection being reduced over).

For example, fn n, sum -> sum + n end is a reducer that take a number, and adds it to the running sum. It could be used like

iex> Enum.reduce([1, 2, 3], 0, fn n, sum -> sum + n end)
6

In the solution, my accumulator takes the shape of a tuple that hold the current topic (or nil) as the first element, and a list of topic maps as the second element.

The reducer has four clauses:

  1. If it encounters a <p> element before it has a topic, it raises.
  2. If it encounters a <h2> element, it assumes that that element has one child, which will be the topic, and sets the topic in the accumulator to that topic.
  3. If it encounters a <p> element and has a topic, it creates a topic map with the data from the <p> element (again assuming the element only has one child) and the topic, and adds that topic map to the topic maps in the accumulator.
  4. If it encounters any other shape of node (whether it’s a <span>, or a <p> element with no or multiple children, or anything else), it raises.

If you have any other questions about how some part of that works feel free to ask.

Where Next?

Popular in Questions Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
gonzofish
I’m currently trying to understand how to join three tables using Ecto. All the examples I’ve seen use 2, so maybe I’m just missing somet...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
quazar
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID&lt;0.412.0&gt; terminating ** (Postgrex.Error) FATAL...
New

We're in Beta

About us Mission Statement