marcuslankenau

marcuslankenau

How to use a set of worker to handle sqs messages?

Hi there,

we are implementing a client for AWS EQS queuing (that we already use in other [ruby] projects). Sqs is providing a http entpoint that is returning a list of messages. In case of an empty queue it is possible to let that call block for a couple of seconds (to save http calls and money). We would like to have one process polling sqs and handing the messages to a pool of worker, that do the real work. The number of worker should be limited to avoid sucking all sqs messages into elixir and dying an overload death.

We have a kind of hacked solution using honeydew, but there we need busy waiting loops to wait for a free worker. That we would like to avoid.

It would be really nice to have a worker pool that we can pass a message and it blocks (the caller) in case all worker are busy. I think we would be able to handcraft that, but prefer to use otp or standard libraries.

Solutions and ideas are highly appreciated. :slight_smile:

Cheers
Marcus

Most Liked

OvermindDL1

OvermindDL1

Actually, instead of poolboy, this seems like something that the new GenStage module that is being added to Elixir (and currently available via the above link on hex.pm) is explicitly and specifically designed for. Hard to get more ‘standard built-in library’ than that (at least until it is added to, likely, the next Elixir version base). :slight_smile:

EDIT: /me is really loving the GenStage module

StevenXL

StevenXL

Hi @marcuslankenau,

I would love to hear what the folks that have actually worked with pools would do in this case, but pool management is a (elegantly) solved problem in Elixir / Erlang. The most common package I’ve seen used for this is poolboy. It’s a dependency of Ecto, in case that increases your confidence in the library. I found a very quick tutorial / example on using poolboy - I don’t know how much the library has changed since that post was written, but it will certainly help you understand the intent of the library.

Does it seem to fit your need?

StevenXL

StevenXL

I’ll second @OvermindDL1’s request for a follow-up, if it’s not to much trouble. Also, @OvermindDL1 thanks for pointing out GenStage - I wasn’t aware of it.

Where Next?

Popular in Questions Top

sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
makeitrein
Hey all, just started picking up Elixir last week and am writing a scraper as a learning project. Baby step #1 is extracting the number ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
mathew4509
I have a list say x = ["23gh", "56kh", "97mh"] I would like to pass each element to Val in each iteration. Say, in iteration 1 -------...
New
gonzofish
I’m currently trying to understand how to join three tables using Ecto. All the examples I’ve seen use 2, so maybe I’m just missing somet...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New

Other popular topics Top

JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1140 51847 244
New
William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New

We're in Beta

About us Mission Statement