mbklein

mbklein

SQNS (or OffBroadway.SQNS or Sequins) - SQS/SNS data processor using Broadway

I’m not here to announce a release (yet). Instead, I’m here for some pre-release opinions.

I’ve created a library tentatively named SQNS, for Simple Queuing and Notification Service. It’s not a typical Broadway library, as it uses BroadwaySQS as its producer and acts instead as an abstract processor and batcher.

The processor’s handle_message/3 takes a message from SQS, pre-processes it a little, hands it off to a process/2 callback, then post-processes the result before sending it back to Broadway. The batcher forwards the resulting message to an SNS topic, which can then be subscribed to by any number of consumers (including one or more SQS queues, each with their own SQNS handlers). It also provides some convenient utility modules to set up the queues and topics and to establish the subscriptions among them. We’re using it as the basis for a really tidy data processing / content ingestion engine.

Here’s where my questions come in:

  1. Has the basic description piqued your interest enough to want more details? I’m trying to decide whether to keep this (for now) as an internal tool or make it a community library, and early interest would help me demonstrate (to my managers) the value of the extra time spent making it releasable.
  2. (Primarily for @josevalim) Does something like this belong in the OffBroadway namespace, or is it too much of an abstraction? Here in my office we pronounce SQNS as “Sequins,” which kind of fits the whole Broadway theme, so releasing it simply as Sequins is an attractive option.

I’ll keep updating this thread with info and examples as I continue to work towards a release.

Thanks!
MBK

Most Liked

josevalim

josevalim

Creator of Elixir

Hi MBK! I was originally thinking the OffBroadway namespace would be mainly for producers. And I do think this is an abstraction on top, rather than something used by Broadway itself, I like the Sequins route. :slight_smile: Thanks for asking!

Where Next?

Popular in Libraries Top

RobertDober
Earmark is a pure-Elixir Markdown converter. It is intended to be used as a library (just call Earmark.as_html), but can also be used as...
239 11851 134
New
sasajuric
I’d like to announce a small library called boundaries. This is an experimental project which explores the idea of enforcing boundaries ...
New
cjen07
parameterized pipe in elixir: |n> edit: negative index in |n> and mixed usage with |> are supported example: use ParamP...
New
mcrumm
If you would like to migrate away from node/npm/webpack while still using sass, the dart_sass package provides a installer and runner for...
New
New
wfgilman
I’ve cleaned up and open sourced three financial libraries I was using for my company. They are bindings for the APIs of these three comp...
New
Crowdhailer
I have been updating a library that allows you to pipe between functions that use the erlang result tuple convention. Assuming you have...
New
woylie
Flop is an Elixir library that applies filtering, ordering and pagination parameters to your Ecto queries. offset-based pagination with...
New
Azolo
Hey everyone, I just released WebSockex which is a Elixir WebSocket client. WebSockex strives to work as a OTP special process, be RFC6...
New
bluzky
You may know https://ui.shadcn.com/, a UI component library for React. I really love it’s design style and components. I’ve built some co...
381 12391 119
New

Other popular topics Top

Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3268 119930 1237
New
chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 27727 240
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New

Sub Categories:

We're in Beta

About us Mission Statement