Fl4m3Ph03n1x
Why should I uses Tasks instead of Pools?
Background
A book I am reading makes the following statement:
Tasks will allow us to do one-off jobs and still rely on the rich OTP library. Connection pooling libraries let us share long-running connections across processes.
So, my understanding of this is as follows:
- if I have a one off task, I should use tasks
- if I need persistent connections to something, I should use pools
Why not always use pools?
And this is my questions. Even if I have one-off tasks, I still think pools are superior because they don’t have to create a new process to do the work, everything is already created.
I could understand the argument here if using pools was significantly harder than using one-off Tasks, but today with libraries like poolboy and books like Elixir in Action I do feel it is easy to understand, create and use pools these days.
Questions
- What are the major benefits of using one-off Tasks instead of a Pool?
- In what scenario would I be better of using a one-off Task instead of a Pool?
Marked As Solved
peerreynders
For me the whole point of the BEAM is that processes are lightweight and creating processes is inexpensive.
This property makes it very attractive to view processes a disposable - much in the same way as memory is “disposable” in a garbage collected language.
This is also helpful when it comes to resilience. If a process is disposed of after a single use, it is much less likely that you’ll end up with a situation where some weird edge case is corrupting your process state over time.
If the process lifetime is short enough, memory may never have to be garbage collected and is simply returned wholesale to the system after process termination.
So, personally I would favour short lived processes over long lived ones - until there is a good reason for making it long lived.
if I need persistent connections to something, I should use pools
More generally, if your process works with resources that are expensive to create and reusable then it should be a worker in a pool. Connection pools are built around that concept because database connections are expensive to create and there is an effective, finite limit to the quantity of simultaneous database connections.
The typical tradeoff is that long-lived processes should be as simple as possible to minimize the danger of state corruption over the long term. So it wouldn’t be that unusual for a long-lived process to delegate a lot of its work to short-lived processes while the long-lived process only manages simple state transitions.
Furthermore, from the resilience perspective, it may be worthwhile making worker processes “perishable” - terminate them once they have been used too often or aged too much and replace them with a fresh worker (and resources).
Also Liked
NobbZ
The only 2 reason I could spontaniously come up are those:
- Pools also always introduce some kind of rate limit or upper bound for concurrent processing, while you can spawn as many tasks as you want. Of course there are pools which can handle this by dynamically spawn processes when exhausted, but if this occurs often, then you are basically back to even slower spawning tasks.
-
Taskis part of the standard library, pools aren’t.
lud
Well for some reasons people could use the process dictionary, for random numbers related stuff for instance. Or some people could register the process to a registry, use a library that would link/monitor the process from another process, create ETS tables that would not be destroyed.
hauleth
Because often we pool other resources as well. Sometimes these resources are limited (RAM when you do image processing, you want to limit amount of the images processed at once to keep RAM usage in reasonable boundaries), sometimes setup can be long and it is better to do it once and then keep it alive for longer time (HTTP keep-alive connections), and sometimes there is mix of both of those (for example DB connections).
hauleth
You can do, but this will be applied globally (by reducing amount of schedulers). Alternatively you could use :max_children option in Task.Supervisor or DynamicSupervisor, but that will have problem as there is no queuing mechanism and it will simply return error when trying to start new job. So it all depends on your use case, in the end we have DynamicSupervisor which in most cases will be more than enough, and pools are similar to that (when :max_children applied), but with additional features (like queuing and ending tasks that take too long).
lud
Also pooled processes must have a cleanup step if they will perform many, independant, unrelated tasks.
A dedicated Task process will have everything cleanup up upon completion.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Build Elixir applications with speed and confidence.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"