njonsson

njonsson

StructuredIO, an Elixir API for consuming structured input streams

StructuredIO is a Hex package. What problem does it solve? In a nutshell, it simplifies working with structured input (such as markup or binary data) that streams from another process or computer. In such situations your application needs to tolerate input arriving piecemeal rather than in complete, well-formed data elements. Input fragments may even split multibyte characters and thereby cause encoding errors.

How do you tame this incidental complexity so that your application can just focus on using data elements instead of the error-prone tedium of reassembling them from fragments? StructuredIO can help.

There are two main features of this library.

  1. It provides a stateful process with a writer function for writing binary data. (If it sounds to you like IO that is part of the Elixir standard library, you’re right.)
  2. It provides a variety of reader functions for conditional reading according to a specified data structure. If a complete data structure has not (yet) been written to the process, nothing is read.

The combination of these two features makes it easy for your application to allow data to trickle into a StructuredIO process, and to consume data elements only as each arrives in its entirety.

You’ll find detailed examples in the project readme as well as in the project API reference.

Most Liked

njonsson

njonsson

Thanks! Yes, I’m now working on adding parsers that use the building blocks of the existing API. So there will be one-liner streaming access to well-known wire formats such as JSON, XML, CSV, BER-TLV, DER, and ASN.1.

CptnKirk

CptnKirk

But more generically aren’t you interested in turning a stream of TCP bytes into a stream of structured content? Structure packing is at the core of nearly every streaming protocol.

Unfortunately, I don’t see Flow as providing a convenient API for this. Although, this would still have value as a GenStage. Or at least it would be nice if there was a way for this to assume GenStage behavior, even if it is available outside of GenStage by default.

OvermindDL1

OvermindDL1

Huh, that’s a cool design, kind of like gen_tcp with a parsing step it sounds like (and no tcp). :slight_smile:

OvermindDL1

OvermindDL1

Whooo cool. :slight_smile:

axelson

axelson

Scenic Core Team

Just wanted to give my +1 to the idea of the library. Sounds well thought out and useful and I’ll keep it in mind for the future :slight_smile:

Where Next?

Popular in Libraries Top

josevalim
EDIT: since Ecto 3.0 final version is out, this post was amended to use the final versions in the instructions below. Hi everyone, We a...
New
woutdp
Hi! I wanted to introduce my latest project LiveSvelte. It allows you to render Svelte inside LiveView with end-to-end reactivity. It’s ...
New
blatyo
https://www.conduitframework.com/ The best overview for how things are tied together is this presentation. Modules and functions are pre...
New
asiniy
Hey there! I wrote a download elixir package which does exactly what its name about - an easy way to download files. I saw solutions ...
New
Crowdhailer
I have been updating a library that allows you to pipe between functions that use the erlang result tuple convention. Assuming you have...
New
benlime
LiveMotion enables high performance animations declared on the server and run on the client. As a follow up to my previous thread A libr...
New
OvermindDL1
I created a new library (rather I pulled out a couple files from my big project), it manages an operating system PID file for the BEAM. ...
New
mtrudel
Bandit is an HTTP server for Plug and WebSock apps. Bandit is written entirely in Elixir and is built atop Thousand Island. It can serve...
New
gjaldon
As the title states, EctoEnum has just been updated after some time of hardly any activity in the repo. Here’s the latest release: https:...
New
Hal9000
Here is my first stab at this. README pasted below. https://github.com/Hal9000/elixir_random Comments and critiques are welcome. Th...
New

Other popular topics Top

chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Sub Categories:

We're in Beta

About us Mission Statement