PJUllrich

PJUllrich

Author of Building Table Views with Phoenix LiveView

How to spawn many, many processes really fast?

I’m currently playing around with GenServers and for a totally unrealistic, but still interesting problem, I wondered whether there is a faster way of spawing 40.000+ processes than this:

pids = for id <- 1..40_000 do
  {:ok, pid} = GenServer.start_link(MyGenServerModule, id: id)
  pid
end

Since this for-loop runs synchronously, it takes around 10 to 12 seconds on my macBook to spawn all 40k processes. I wondered, whether this could somehow be achieved asynchronously?

Marked As Solved

gregvaughn

gregvaughn

Well, there’s your bottleneck. The LiveView has to handle 50K messages.

Also Liked

PJUllrich

PJUllrich

Author of Building Table Views with Phoenix LiveView

Thank you! That indeed decreased the execution time from ~12s to ~1.2s, but unfortunately, I believe the processes weren’t linked to the original process correctly. Maybe because the GenServer.start_link/2 was called inside the Task.async_stream? Will those processes then correctly be linked to the “parent process” in which I used to call the GenServer.start_link/2 function originally?

PS: I’m currently playing around with simulating the Game of Life with 50.000 erlang processes. Because, why not? :smiley:

10
Post #5
rvirding

rvirding

Creator of Erlang

Just to be a bit picky here, starting a GenServer does more than just spawn a process. So the GenServer.start_link will spawn the process and then sit and wait until the new process does the necessary initialisation to become an OTP process and then calls the behaviour’s init callback. When that has completed the GenServer.start_link will return {:ok,pid}. This is a synchronous operation which is well defined in how a GenServer behaviour works. Can you get around it? NO, all you can do is try and make the init callback as fast as possible. Check the documentation for tips for doing that.

If you want to check how fast you can spawn processes just do a spawn instead, of course then you won’t get a GenServer. :smile:

lud

lud

There is a feature to do just that:

  @impl true
  def init(state) do
    {:ok, state, {:continue, :after_start}}
  end

  @impl true
  def handle_continue(:after_start, state) do
    # start living
  end
princemaple

princemaple

Hi, That’s pretty cool.

To link the process to your main process, you can do this:
1..40_000 |> Task.async_stream(fn _ -> YOUR_CODE end) |> Enum.map(fn {:ok, pid} -> Process.link(pid) end)

PJUllrich

PJUllrich

Author of Building Table Views with Phoenix LiveView

I wrote down my findings in a blog post :slight_smile:

Where Next?

Popular in Questions Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

We're in Beta

About us Mission Statement