ahferroin7

Most efficient way for a process to voluntarily yield scheduling priority?

Background

I’m working on a Discord bot in Elixir that runs as two separate applications, the bot itself, and all the backend logic and data handling that needs to be done on behalf of the bot. The back end application has to do a lot of data processing on startup which it’s currently handling asynchronously (curretly, each component in the backend is a GenServer, and the init callback just immediately returns a continue instruction that triggers the actual data processing needed for initialization). This is working rather well overall, except for one specific component which has a long and computationally intensive initialization sequence after a change in either the code or the initialization data for that component.

The component in question needs to process a very large list (roughly 2900 items) by computing an SQL transaction for each item and then running that against a database. The amount of processing here is time-prohibitive if it needs to be done serially (each item in the list takes about 50-100ms to process and then run the SQL transaction, so the full list takes almost 5 minutes if run one-by-one), but there are a handful of computations that can be shared across all the items, so my current code is using Stream.chunk_every/1 and Task.async_stream/3 to run the initialization in a number of parallel chunks equal to the number of online schedulers like so:

items
|> Stream.chunk_every(div(length(items), System.schedulers_online()) + 1)
|> Task.async_stream(&process_chunk/1, ordered: false)
|> Enum.to_list()

This is working in terms of actually processing things correctly and making the initialization fast enough to be useful, but causing a completely different issue in that it’s blocking scheduling of other processes for a long time, which is causing the bot itself to fail initialization because it can’t finish starting up before this starts running.

The question

My first instinct here based on experience elsewhere is to have the process_chunk/1 function voluntarily yield scheduling priority (I suppose this translates to voluntarily moving to the end of the run-queue for the scheduler in BEAM terms) before it processes each individual item. Right now, I’m doing this by running :timer.sleep(1) at the beginning of each iteration within process_chunk/1, which seems to be working to ensure that other things can run but feels like a bit of a hack TBH and also adds to the overall initialization time for this component (it’s only ~91ms of extra time on my development box, but translates to ~734ms on the production system it will be running on due to a much lower scheduler count).

Is there some more efficient way to voluntarily yield scheduling priority in Elixir or Erlang? Or is there perhaps some other approach I could take here that still lets other things run without significantly impacting the initialization times for the component in question?

12 comments

#async_stream

10 2595 12

2022-03-16 14:35:36 UTC

Most Liked

sasajuric

Author of Elixir In Action

This is a curious problem

First, I’ll echo the sentiment of others that GenServer shouldn’t be blocking for a long time, because that might cause the rest of the system to block. This can be handled in a couple of ways:

Process chunks synchronously during the app or server boot (e.g. in the init callback).
Start a separate task which will start the async_stream, await for the results, and then do something with them (e.g. send them to other processes).
Instead of waiting for all the tasks to finish in GenServer, handle task results as they arrive in handle_info.

However, given you description I’m not sure that this would solve the issue. It’s interesting that including :timer.sleep(1) in process_chunk removes the problem. This could indeed mean that schedulers are blocked, or alternatively that some locking takes place at the SQL level.

If the schedulers are blocked, a likely reason would be a custom native code (NIF). As mentioned by others, the scheduler does frequent preemptive context switching. Due to the functional nature of BEAM languages, functions are frequently invoked, while a single longer-running BIF will bump the reduction count by more than 1. Furthermore, in recent OTP versions BIFs also yield. E.g. since OTP 22, length/1 will yield when called with long lists (source).

To check this I’d try to reproduce the problem using a single scheduler thread. I’d write a test function, e.g. process_big_chunk/0 which processes a larger amount of data sequentially (i.e. no tasks). I’d also comment out the startup processing code, i.e. I’d make the app start the required processes without doing anything else (like starting some activity).

Then I’d manually start a single-scheduler-threaded BEAM with ELIXIR_ERL_OPTIONS="+S 1" iex -S mix. From the iex session I’d first start the oberver (:observer.start), and then spawn an infinite processing loop as:


spawn(fn -> Stream.repeatedly(&process_big_chunk/0) |> Stream.run() end)`

If the observer is responsive (you can click on it and it refreshes data), it means that the scheduler is not blocked. OTOH if the observer is blocked, or very laggy, it would be an indication that something is indeed blocking the scheduler. You could then proceed by sprinkling IO.inspects to see where the blocking takes place (i.e. which operations take long to finish). Alternatively you could start the system with more schedulers, and use observer or Erlang tracing to deduce the same thing.

If the single thread processing doesn’t block the scheduler, the problem could be in how the library and/or SQLite handle concurrent operations. You could try the same experiment using two scheduler threads and two infinite processing loops. Again, sprinkling some IO.inspect for debugging purposes might help discover where the process is blocking.

In any case I feel that the sleep hack is not a reliable fix, and that the issue might still occasionally resurface, so I’d personally spend some time trying to understand the issue. It’s hard to tell exactly where the problem is, but based on your description, it might be caused by the NIF implementation of the 3rd party library, or by the concurrent behaviour of SQLite. Of course it’s also possible that you stumbled upon some bug/deficiency in Erlang, but I don’t think this is likely.

Either way, some further analysis & debugging is required to properly understand this. Best of luck and keep us posted

Post #9

tcoopman

What do you mean exactly with that it is blocking scheduling of other processes?

I’m not an expert at this, but the BEAM has preemptive scheduling which means that nothing should be able to block the scheduler for a long time

Post #2

al2o3cr

The scheduler’s designed to equitably share CPU between all the runnable processes; if it’s blocking that means something else is going wrong.

How many database connections are in Ecto’s pool? If there aren’t more than System.schedulers_online, that could cause the Tasks to hold all of them and then block forward progress from other processes.

Post #3

ahferroin7

Ecto isn’t involved here, I’m just poking at a local SQLite3 database using Sqlitex, serialized through a single GenServer instance (it’s largely a case of using SQL here because it’s moderately easier and rather surprisingly significantly faster than trying to do the same thing with ETS or mnesia), and there’s no concurrent access from any other processes except those that are trying to initialize the database (because all other access would be through the GenServer that’s waiting for the initialization code to finish).

Looking a bit deeper, I think what’s going on here is that something in the third-party code being used where things are failing to initialize is expecting some sort of ordering constraint in the scheduling that is not holding true when the scheduling code has more runnable tasks than it has online schedulers.

Post #5

josevalim

Creator of Elixir

You shouldn’t have to worry about doing such scheduling because of the BEAM. Either a process has work to do, and then it will do it, or it doesn’t have work to do, which means another process gets to run instead.

A scenario where a process doesn’t have many functions to run just means it doesn’t have work, so it will be schedule out. It won’t block or wait until work becomes available.

Maybe there is a shared resource or something else they are all trying to reach out, but from a glance it doesn’t look like an issue with the VM scheduling.

Post #6