Hey

I’m building a system in which users dynamically will be allocated a number computation ‘slots’. Each slot is basically a single process doing heavy IO-bound work like downloading a file.

Furthermore, I want these processes to be supervised and spread out across nodes, so that the same user can have dedicated processes on multiple machines. I also want the number of computation slots to always (with brief interuptions) to be equal to the number of currently running processes for that particular user.

My initial idea was to use something like swarm to handle the registration and distribution of a ‘coordinator’ process for each user, which would then be responsible for starting and supervising the pooled processes for that same user.

For the pooling I was looking into using something like poolboy, but to my knowledge poolboy and any other pooling libraries I’ve found only allows pooling processes on the same node, and not across nodes in a distributed fashion

Any ideas on how to accomplish the distributed pooling?