bettio

bettio

Streaming XML to file

Hello :slight_smile:

We are trying to export a “huge” amount of data from a database using XML.
Our XML exporter service is running on a cluster and it has a limited amount of memory (e.g. 4GiB of RAM) so we cannot keep in memory the whole XML document.
Does anyone knows any good Elixir (or erlang) library which allows to stream data so we don’t have to keep in memory the whole document before writing it?

Most Liked

dimitarvp

dimitarvp

I hate to bring this suffering upon you but for excellent performance you should definitely just use Erlang’s :xmerl. It’s not exactly easy to work with, ask away if you get stuck.

It does support streaming, both reading and writing, so RAM will not be a problem.

ananthakumaran

ananthakumaran

We had a similar use case the difference being we need to generate xlsx files (which is just a zip file with multiple xml files). Apparently neither google nor hex.pm seems to show this library for xml stream

bettio

bettio

This does not help:

Fast Expat based Erlang XML parsing library. I’m looking for a writer, not a parser.

dimitarvp

dimitarvp

I have a few scattered links that can help you get started. xmerl is not very intuitive for a newcomer so I see some cursing in your future. :003:


Notes on xmerl and processing XML in Erlang. This one can help you understand how can you construct :xmerl XML nodes programmatically (as opposed to receiving them when stream-parsing a file) – and then stream them to a destination file. You’ll likely have to learn to import and use Erlang records in Elixir at some point – but mind you, this is only for reading Erlang records from Elixir, not for writing them.

It’s quite simple:

defmodule Meh.Xmerl
  Record.defrecord(
    :xmlElement,
    Record.extract(:xmlElement, from_lib: "xmerl/include/xmerl.hrl")
  )
end

Read more about it:


Two blog articles on stream-reading XML files. You’ll need these even if you only write XML:

  1. SAX Xml parsing in Erlang – Part 1
  2. A Simple XML State Machine Accepting SAX Events to Build xmerl Compitable XML Tree: icalendar demo

xmerl’s docs on the various hook functions that you can pass to its processors (trust me, you’ll need some of these as well, depending on the complexity of your task):


Basically, xmerl works by accumulating XML elements / texts / namespaces etc. in a reverse-accumulator manner (prepending elements to a list; at least that’s how it works when stream-parsing and processing-as-you-go – haven’t checked the writing functionality for that idiom), using its own in-memory representation of XML. Which you can then pass to its export functions if you want a file (or only a singular XML element serialised to a charlist).

As I said before, it’s going to be a ride.

Where Next?

Popular in Questions Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
Werner
Hi, I’m using Ubuntu 18.04 and after updating to OTP-24.0 yesterday i have this warning when I run “mix local.hex”: 14:57:30.512 [warn] ...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New

Other popular topics Top

Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3268 119930 1237
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
sergio
I couldn’t find any guides that worked well with Phoenix 1.6.0 and esbuild. I hope this helps people test the waters and eases you into t...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New

We're in Beta

About us Mission Statement