alexcastano

alexcastano

Nice shutdown for a worker pool

I’ve been working on a pool of remote sessions, where each session is created and closed with HTTP requests. Sessions are limited in number, so I figured, why not implement a pool? An additional requirement is that these sessions must be refreshed every 15 minutes if idle.

I decided to use a GenServer to handle the sessions. When it starts, it makes a request to create a session and stores the secret. On termination, it makes another request to close the session. It has a public API that makes an HTTP request using the session, parses the response, and sends it back to the client. And if it’s not used for 15 minutes, it’ll refresh the session itself.

To manage the pool, I’ve got a Supervisor looking after two children:

  1. Finch HTTP pool
  2. Poolboy worker pool

So when a Poolboy worker needs to make a request, it grabs a connection from the Finch HTTP pool. However, things start to go south when it’s time to shut everything down.

When the app needs to shut down, the Supervisor sends a shutdown signal to Poolboy. But it seems Poolboy isn’t that patient and doesn’t wait for its children to stop. From what I can tell, the exit/2 function works asynchronously. So, the children get the shutdown signal, but the Poolboy worker pool doesn’t stick around.

Then the Finch pool shuts down, the Supervisor follows suit, and finally, the workers. But, here’s the twist: the workers need the Finch pool to send out a final request to close the remote session. So, they crash without closing the remote sessions.

What’s throwing me for a loop is why a library as popular as Poolboy would behave like this. Am I missing something here?

I also thought about using NimblePool, but decided against it because:

NimblePool may not be a good option to manage processes. After all, the goal of NimblePool is to avoid creating processes for resources. If you already have a process, using a process-based pool such as poolboy will provide a better abstraction.

Any ideas on this? Maybe a different approach?

PD: I opened an issue in Poolex :slight_smile:

Most Liked

mpope

mpope

If you need a solution before a fix is released from the libraries you could use a process that each worker links with called the WorkerManager. This process can add each registerd worker to a set for tracking, and listen for their deaths. Once all of the processes that were linked die, then this process can terminate itsself. You can use a supervisor is has a rest_for_one strategy, then I think if you have the order: [FinchPool, WorkerManager, WorkerPool]. The WorkerPool can shutdown, the manger will wait for all the workers, then terminate once the final worker dies, and then finally the FinchPool can shutdown. This ordering should allow for a worker that is shutting down to grab a Finch connection.

cmo

cmo

pooler exists too

alexcastano

alexcastano

Thank you for your proposal. Last commit in Poolboy repo was more than 4 years ago, so I’m hopeless. I’ll try your solution if I cannot find a good alternative to Poolboy.

I don’t like that the worker has to know about the manager to link, but I’m sure it will eradicate the bug :slight_smile:

Where Next?

Popular in Questions Top

pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
dokuzbir
Hello, I am trying to convert my lists to string without losing brackets.For start i have 3 map. They look like these buyer = %{ id: ...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

Other popular topics Top

vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 45766 226
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New

We're in Beta

About us Mission Statement