Newxan

Newxan

Help with using sweet_xml and nested xpath

Hi, I’ve got a database XML dump from an old wordpress site that I want to import into a project.
I’musing sweet_xml and it’s working fine for the simple structures.
However when I encounter lists of nested fields such as wordpress’s metadata fields I can’t get sweet_xml to pick up the values. I’ve reproduced my problem in a minimal way:
import.ex

defmodule Mix.Tasks.Import do
  use Mix.Task

  import SweetXml

  @target_file "./data.xml"

  def run(_args) do
    Mix.Task.run("app.start")
    import_dump()
  end

  defp import_dump do
    tree = File.stream!(@target_file)

    rows =
      SweetXml.xpath(
        tree,
        ~x"/rss/channel/item"l,
        slug: ~x"./wp:post_name/text()"s,
        company_logo: ~x"./wp:postmeta[wp:meta_key = '_thumbnail_id']/wp:meta_value/text()"s,
        edited_at: ~x"./wp:postmeta[wp:meta_key = '_edit_last']/wp:meta_value/text()"s
      )

    IO.inspect(rows)
  end
end

data.xml

<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0"
xmlns:excerpt="http://wordpress.org/export/1.2/excerpt/"
xmlns:content="http://purl.org/rss/1.0/modules/content/"
xmlns:wfw="http://wellformedweb.org/CommentAPI/"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:wp="http://wordpress.org/export/1.2/">
  <channel>
    <item>
      <wp:post_id>7068</wp:post_id>
      <wp:post_name>
<![CDATA[a-post-name]]>
      </wp:post_name>
      <wp:postmeta>
        <wp:meta_key>
<![CDATA[_edit_last]]>
        </wp:meta_key>
        <wp:meta_value>
<![CDATA[1]]>
        </wp:meta_value>
      </wp:postmeta>
      <wp:postmeta>
        <wp:meta_key>
<![CDATA[_thumbnail_id]]>
        </wp:meta_key>
        <wp:meta_value>
<![CDATA[11111]]>
        </wp:meta_value>
      </wp:postmeta>
    </item>
  </channel>
</rss>

The output But the output I’m getting no matter what I try is:

[%{company_logo: "", edited_at: "", slug: "\na-post-name\n      "}]

As you can see the company_logo and edited_at fields are always empty.

I’m not really sure where to go from here, as far as I can tell the xpath syntax itself is correct. But I am unsure about how you’re supposed to use the sweet_xml sigil modifiers when there is nested data such as this.

Marked As Solved

srowley

srowley

There is a hint in your output - the CDATA elements are wrapped in newline characters and have trailing spaces, i.e. the value for slug in your example is "\napost-name\n ".

That means that your Xpath query is not finding anything for the slug or company logo nodes because, for example, it is looking for the key _edit_last when that key is \n_edit_last\n .

Your query would work like this:

SweetXml.xpath(
      tree,
      ~x"/rss/channel/item"l,
      slug: ~x"./wp:post_name/text()"s,
      company_logo: ~x"./wp:postmeta[contains(wp:meta_key, '_thumbnail_id')]/wp:meta_value/text()"s,
      edited_at: ~x"./wp:postmeta[contains(wp:meta_key, '_edit_last')]/wp:meta_value/text()"s
    )

I don’t know enough about XML to understand why the newlines/spaces are there; if there is some rule associated with that I suppose you could also just apply it and keep using = instead of contains, akin to (I didn’t test this one):

company_logo: ~x"./wp:postmeta[wp:meta_key = '\n_thumbnail_id\n     ')]/wp:meta_value/text()"s,

Also Liked

rjk

rjk

I can confirm that the suggestion from @srowley works, this code is tested in an iex repl:

    clean_newlines_and_ws = fn x -> String.replace(x, ~r/\r|\n|\W/, "") end
    rows =
      SweetXml.xpath(
        tree,
        ~x"/rss/channel/item"l,
        slug: ~x"./wp:post_name/text()"s |> transform_by(clean_newlines_and_ws),
        company_logo:
          ~x"./wp:postmeta[contains(wp:meta_key, '_thumbnail_id')]/wp:meta_value/text()"s
          |> transform_by(clean_newlines_and_ws),
        edited_at:
          ~x"./wp:postmeta[contains(wp:meta_key, '_edit_last')]/wp:meta_value/text()"s
          |> transform_by(clean_newlines_and_ws)
      )

i also added a transform_by function from sweet_xml that cleans up those newlines and extra whitespaces.
Also not sure why those are added in the first place as I can’t see them in the XML itself, seems like the underlying xmerl library (that sweet_xml uses) or sweet_xml itself does something weird to those cdata nodes?

But at least with this code you could go on with the task at hand :slight_smile:

srowley

srowley

As a postscript, it occurred to me later that all that is happening here is that the whitespace within the markup is being preserved - looking at the sample within the offending tags there is a newline, the CDATA node, some spaces, and then another newline. The :xmerl documentation confirms this and provides an example very similar to this one.

Where Next?

Popular in Questions Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
quazar
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
mathew4509
I have a list say x = ["23gh", "56kh", "97mh"] I would like to pass each element to Val in each iteration. Say, in iteration 1 -------...
New
Codball
Mix format works fine if run from the cmd. I’ve followed this to facilitate the implementation into VSC which involves downloading an ext...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New

Other popular topics Top

senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

We're in Beta

About us Mission Statement