PJUllrich
How to spawn many, many processes really fast?
I’m currently playing around with GenServers and for a totally unrealistic, but still interesting problem, I wondered whether there is a faster way of spawing 40.000+ processes than this:
pids = for id <- 1..40_000 do
{:ok, pid} = GenServer.start_link(MyGenServerModule, id: id)
pid
end
Since this for-loop runs synchronously, it takes around 10 to 12 seconds on my macBook to spawn all 40k processes. I wondered, whether this could somehow be achieved asynchronously?
Marked As Solved
gregvaughn
Well, there’s your bottleneck. The LiveView has to handle 50K messages.
Also Liked
PJUllrich
Thank you! That indeed decreased the execution time from ~12s to ~1.2s, but unfortunately, I believe the processes weren’t linked to the original process correctly. Maybe because the GenServer.start_link/2 was called inside the Task.async_stream? Will those processes then correctly be linked to the “parent process” in which I used to call the GenServer.start_link/2 function originally?
PS: I’m currently playing around with simulating the Game of Life with 50.000 erlang processes. Because, why not? ![]()
rvirding
Just to be a bit picky here, starting a GenServer does more than just spawn a process. So the GenServer.start_link will spawn the process and then sit and wait until the new process does the necessary initialisation to become an OTP process and then calls the behaviour’s init callback. When that has completed the GenServer.start_link will return {:ok,pid}. This is a synchronous operation which is well defined in how a GenServer behaviour works. Can you get around it? NO, all you can do is try and make the init callback as fast as possible. Check the documentation for tips for doing that.
If you want to check how fast you can spawn processes just do a spawn instead, of course then you won’t get a GenServer. 
lud
There is a feature to do just that:
@impl true
def init(state) do
{:ok, state, {:continue, :after_start}}
end
@impl true
def handle_continue(:after_start, state) do
# start living
end
princemaple
Hi, That’s pretty cool.
To link the process to your main process, you can do this:
1..40_000 |> Task.async_stream(fn _ -> YOUR_CODE end) |> Enum.map(fn {:ok, pid} -> Process.link(pid) end)
PJUllrich
I wrote down my findings in a blog post 









