bettio
Streaming XML to file
Hello 
We are trying to export a “huge” amount of data from a database using XML.
Our XML exporter service is running on a cluster and it has a limited amount of memory (e.g. 4GiB of RAM) so we cannot keep in memory the whole XML document.
Does anyone knows any good Elixir (or erlang) library which allows to stream data so we don’t have to keep in memory the whole document before writing it?
Most Liked
dimitarvp
ananthakumaran
We had a similar use case the difference being we need to generate xlsx files (which is just a zip file with multiple xml files). Apparently neither google nor hex.pm seems to show this library for xml stream
bettio
This does not help:
Fast Expat based Erlang XML parsing library. I’m looking for a writer, not a parser.
dimitarvp
I have a few scattered links that can help you get started. xmerl is not very intuitive for a newcomer so I see some cursing in your future. ![]()
Notes on xmerl and processing XML in Erlang. This one can help you understand how can you construct :xmerl XML nodes programmatically (as opposed to receiving them when stream-parsing a file) – and then stream them to a destination file. You’ll likely have to learn to import and use Erlang records in Elixir at some point – but mind you, this is only for reading Erlang records from Elixir, not for writing them.
It’s quite simple:
defmodule Meh.Xmerl
Record.defrecord(
:xmlElement,
Record.extract(:xmlElement, from_lib: "xmerl/include/xmerl.hrl")
)
end
Read more about it:
- XML, SweetXml, Xmerl, and Erlang records from Elixir
- Parsing XML using Elixir (Mostly)
- Processing XML with Elixir and xmerl
Two blog articles on stream-reading XML files. You’ll need these even if you only write XML:
- SAX Xml parsing in Erlang – Part 1
- A Simple XML State Machine Accepting SAX Events to Build xmerl Compitable XML Tree: icalendar demo
xmerl’s docs on the various hook functions that you can pass to its processors (trust me, you’ll need some of these as well, depending on the complexity of your task):
Basically, xmerl works by accumulating XML elements / texts / namespaces etc. in a reverse-accumulator manner (prepending elements to a list; at least that’s how it works when stream-parsing and processing-as-you-go – haven’t checked the writing functionality for that idiom), using its own in-memory representation of XML. Which you can then pass to its export functions if you want a file (or only a singular XML element serialised to a charlist).
As I said before, it’s going to be a ride.








