I think you’re confusing the practice of having a specific process that owns the :ets table with the practice of serializing access to that table through the genserver. The two are orthogonal. You can setup a named table with read / write concurrency that is accessed directly, but still is owned by a supervised process. It is generally a best practice to put :ets tables under such managed processes because it allows them to be started / stopped / restarted effectively within your application’s supervision tree.

Maybe you should have used ETS table instead of list? If there are random updates it can make it slower even further.

It should probably be said that an entire book chapter could be dedicated to these situations (I believe there are a few in existence!). I’d maybe go as far as to say it’s one of the main quirks with immutable data, since you can find yourself copying the data over and over instead of what is a simple pointer update in mutable languages.

Also, a thousand IDs should not cause what you are describing; which makes me think you are replicating the data. If the ETS option doesn’t solve your problem, I’ve picked up a few tricks when dealing with this type of scenario; which might be applicable for you:

1. you can inspect the process’ memory usage using Observer. Runaway increases in memory may be a sign that old data is not being cleaned up as you update the ID list.

2. if you’re processing the IDs each in a dedicated process, you can “monitor” the process, listen for the :DOWN message, and then force garbage collection (:erlang.garbage_collect()). Not ideal, but it can work. You can target particular processes as well (eg. parent process).

3. state (memory) can come along for the ride when using Task.Supervisor with anonymous functions. Here’s a nice explanation (not a bug). Short answer, MFA is preferred. If you’re doing any supervised processing in a loop, you may be inadvertently passing a lot of data around (locking it up).

As @hauleth notes, :ets may be the right call here. It has the further perk of not creating garbage collector pressure, since its items are stored in its own managed memory space instead of within the process memory.

When it get slow. However, as other has noted, what you are doing should not be slow; and even if it does get slow, there is other ways to make it faster, such as ets.