njonsson

njonsson

StructuredIO, an Elixir API for consuming structured input streams

StructuredIO is a Hex package. What problem does it solve? In a nutshell, it simplifies working with structured input (such as markup or binary data) that streams from another process or computer. In such situations your application needs to tolerate input arriving piecemeal rather than in complete, well-formed data elements. Input fragments may even split multibyte characters and thereby cause encoding errors.

How do you tame this incidental complexity so that your application can just focus on using data elements instead of the error-prone tedium of reassembling them from fragments? StructuredIO can help.

There are two main features of this library.

  1. It provides a stateful process with a writer function for writing binary data. (If it sounds to you like IO that is part of the Elixir standard library, you’re right.)
  2. It provides a variety of reader functions for conditional reading according to a specified data structure. If a complete data structure has not (yet) been written to the process, nothing is read.

The combination of these two features makes it easy for your application to allow data to trickle into a StructuredIO process, and to consume data elements only as each arrives in its entirety.

You’ll find detailed examples in the project readme as well as in the project API reference.

Most Liked

njonsson

njonsson

Thanks! Yes, I’m now working on adding parsers that use the building blocks of the existing API. So there will be one-liner streaming access to well-known wire formats such as JSON, XML, CSV, BER-TLV, DER, and ASN.1.

CptnKirk

CptnKirk

But more generically aren’t you interested in turning a stream of TCP bytes into a stream of structured content? Structure packing is at the core of nearly every streaming protocol.

Unfortunately, I don’t see Flow as providing a convenient API for this. Although, this would still have value as a GenStage. Or at least it would be nice if there was a way for this to assume GenStage behavior, even if it is available outside of GenStage by default.

OvermindDL1

OvermindDL1

Huh, that’s a cool design, kind of like gen_tcp with a parsing step it sounds like (and no tcp). :slight_smile:

OvermindDL1

OvermindDL1

Whooo cool. :slight_smile:

axelson

axelson

Scenic Core Team

Just wanted to give my +1 to the idea of the library. Sounds well thought out and useful and I’ll keep it in mind for the future :slight_smile:

Where Next?

Popular in Libraries Top

marcuslankenau
I feel kind of stuck with the absence of a proper xml library for Elixir. Currently I use SweetXML which was ok for me more or less to pa...
New
cjen07
parameterized pipe in elixir: |n> edit: negative index in |n> and mixed usage with |> are supported example: use ParamP...
New
mbuhot
Leverage Open Api 3.0 (Swagger) to document, test, validate and explore your Plug and Phoenix APIs. Generate and serve a JSON Open API ...
New
archan937
It is a well-know topic within the Elixir community: “To mock or not to mock? :)” Every alchemist probably has his / her own opinion con...
New
wfgilman
I’ve cleaned up and open sourced three financial libraries I was using for my company. They are bindings for the APIs of these three comp...
New
benlime
I created a new library GitHub - benvp/ex_cva: Class Variance Authority for Elixir which aims to make it very easy to define different va...
New
Qqwy
Solution is a library to help you with working with ok/error-tuples in case and with-expressions by exposing special matching macros, as ...
New
Jskalc
Hi! Today, after a couple weeks of development I’ve released v0.1 of LiveVue. It’s a seamless integration of Vue and Phoenix LiveView, i...
New
Crowdhailer
Experimenting with this code. OK.try do user <- fetch_user(1) cart <- fetch_cart(1) order = checkout(cart, user) save_or...
New
KallDrexx
For a good number of months I've been working on creating a very basic RTMP live video streaming server. Now that I have a very, very ba...
New

Other popular topics Top

gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New

Sub Categories:

We're in Beta

About us Mission Statement