shahryarjb
Help to understand PartitionSupervisor in elixir 1.14
Hi, I have read the PartitionSupervisor documents and I think I need help to understand it.
For example:
defmodule MishkaInstaller.Application do
use Application
@impl true
def start(_type, _args) do
plugin_runner_config = [
strategy: :one_for_one,
name: PluginStateOtpRunner
]
children = [
{DynamicSupervisor, plugin_runner_config},
]
opts = [strategy: :one_for_one, name: MishkaInstaller.Supervisor]
Supervisor.start_link(children, opts)
end
end
So I started it in a file named like plugin_state_dynamic_supervisor.ex this:
def start_job(args) do
DynamicSupervisor.start_child(PluginStateOtpRunner, {MishkaInstaller.PluginState, args})
end
and finally, in my Genserver module:
def start_link(args) do
GenServer.start_link(__MODULE__, [], name: via(id, type))
end
defp via(id, value) do
{:via, Registry, {PluginStateRegistry, id, value}}
end
So every time I want to select a Genserver I am using a registry like this:
def get_plugin_pid(module_name) do
case Registry.lookup(PluginStateRegistry, module_name) do
[] -> {:error, :get_plugin_pid}
[{pid, _type}] -> {:ok, :get_plugin_pid, pid}
end
end
Now where I can use PartitionSupervisor to improve it? And the second question is with this ways If 1000 user call get_plugin_pid function and load a data from a specific state in a same time, I am going to have a bottleneck?
Thank you in advance
Genserver, and the Supervisor
Marked As Solved
msimonborg
No problem!
If you have many users concurrently starting dynamic processes then the PartitionSupervisor may help reduce bottlenecks in that process. At least it won’t hurt : )
Registry is the right tool for this, it uses ETS tables which provide good read concurrency when being accessed my many processes. Consider partitioning your registry for better concurrency.
You may experience bottlenecking with this, you can run some basic load tests on your own to find out. If you have many processes concurrently accessing the same state then this could be a good use case for an ETS table with the read_concurrency: true option.
Also a good use case for wanting this state in an ETS table. I personally find it helpful to think of genserver state as something that the genserver uses to do its work, maybe a few processes will access it. If you plan to have many concurrent processes accessing shared state then it’s better to store it in ETS rather than a single process
If the state is only read and never updated, then it could be a good use case for persistent_term, which is optimized for reads at the expense of writes. Just be sure to read the performance tradeoffs in the module documentation very carefully.
It doesn’t seem like you’d have many issues with this if the state is only being read by 5 processes that all belong to the same user.
Also consider whether all or any of this state should be persisted into a Postgres database instead of in memory : ) Some of the complexity may disappear
Ultimately the PartitionSupervisor will probably only help you by reducing bottlenecks at your DynamicSupervisor, which seems like it could be a good fit for your app and it’s a very simple migration so it can’t hurt. I don’t think it will help with your other needs.
Also Liked
msimonborg
In this case you would use PartitionSupervisor to partition your DynamicSupervisor and spread out the work load from one supervisor to many, with the default number of partitions being equal to the number of cores i.e. System.schedulers_online/0. This would reduce bottlenecks at your DynamicSupervisor for the work it’s doing, like starting, stopping, and restarting your plug-in processes. If you have a lot of concurrency with starting and stopping these processes then this may help you. The PartitionSupervisor would not prevent your get_plugin_id function from bottlenecking, since that is going through a Registry, not your DynamicSupervisor. To maximize concurrency and reduce bottlenecks with your Registry you can manually increase the number of partitions with the :partitions option, e.g. {Registry, keys: :unique, name: PluginStateRegistry, partitions: System.schedulers_online()}.
In my app there are many processes that may be started concurrently by many users. Each process does some clean up in the terminate/2 callback at shutdown. During deployments and scale-downs I need nodes to shutdown gracefully, and these processes must store their state to perform an eventual handoff to a new process that might start on another node and continue its work. I noticed in load testing that when the number of these processes under a single DynamicSupervisor reached a certain level, shutdowns took a really long time. When I tested out replacing the dynamic supervisor with PartitionSupervisor on my 8-core machine the shutdown time decreased dramatically, far more than 8x. This was so helpful for my use case that I copied the PartitionSupervisor module directly into my app so I can start using it without depending on pre-release 1.14.
If you have any particular questions about migrating to PartitionSupervisor I’d be happy to try and help.
msimonborg
This was my try at answering your second question. If your get_plugin_pid function is using Registry then no part of the call is hitting your DynamicSupervisor, so optimizing with a PartitionSupervisor will not help. Optimizing Registry with more partitions might be helpful. Or, are you more concerned with this piece of code GenServer.call(pid, {:pop, :module})? Sorry if I’m misunderstanding your question.
shahryarjb
I am very grateful to you, and thank you for your time, I try to change my code to PartitionSupervisor after releasing elixir 1.14 Official
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Producing high quality Elixir screencasts since 2017.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"