This is my go to reference for when terminate/2 is called: https://gist.github.com/mrallen1/806fe5506132260574af33e99dadd499

This is a very good question. I personally mostly avoid terminate, because it won’t be invoked if the process crashes, or if it’s forcefully terminated (killed) from the outside. Thus, if some cleanup code must be executed, I prefer having another process to do it.

However, using a cleanup process isn’t synchronous, since the cleanup code will run after the “main” process has terminated. Therefore, there are some special cases where terminate works better. For example, supervisor terminates children from the terminate callback. This ensures that when the supervisor goes down, its complete subtree is already down. I can’t think of a way to ensure such synchronism by using a separate cleanup process.

However, this approach suffers from potential theoretical issues. If a supervisor process is brutally killed or if it crashes, then this guarantee doesn’t hold. The child processes will still be taken down eventually, but not immediately. If some descendant is trapping exits and ends up in an infinite loop, it might never happen. This in turn could prevent the restart of the crashed supervisor, which could take down the entire system, or it could lead to duplicate processes running, which could cause some strange behaviour of the system.

However, we can assume that the supervisor process is thoroughly tested and hopefully free from unexpected crashes. In addition, in a properly constructed supervision tree, a supervisor is never brutally killed (because :shutdown of a supervisor is :infinity by default), so I’d say that these issues are theoretical, and very unlikely to occur in practice.

So my take would be to use terminate to implement a synchronous cleanup (things are cleaned up before the process terminates). For example, I use it in Parent to terminate children, similarly to supervisor. In such cases, you probably want to keep the logic of the process simple, to reduce the chance of it crashing. You may also consider setting it’s shutdown option to :infinity to prevent its parent from brutally killing it.

If the synchronism is not required, and a cleanup code must be executed when a process goes down, regardless of how/why it goes down, I’d suggest using a separate process.

A distributed counter is one of the use cases, we’re thinking about when introducing the Firenest.ReplicatedState abstraction in the firnest project. As an example, with the interface we have planned right now, a distributed counter could look like the following.

Each process can register itself for tracking and increment/decrement the counter. When the process goes down, its data is removed (when a node goes down other nodes remove data for all processes from the dead node).

defmodule DistributedCounter do
  alias Firenest.ReplicatedState

  @behaviour ReplicatedState

  def child_spec(opts) do
    Firenest.ReplicatedState.child_spec(topology: MyApp.FirenestTopology, name: opts[:name], handler: __MODULE__)
  end

  def track(server, key) do
    ReplicatedState.put(server, key, self(), 0)
    # calls local_put as the callback inside the server
  end

  def increment(server, key, by) when is_integer(by) do
    ReplicatedState.update(server, key, self(), {:increment, by})
    # calls local_update as the callback inside the server
  end

  def untrack(server, key) do
    ReplicatedState.delete(server, key, self())
    # calls local_delete as the callback inside the server
  end

  def get(server, key) do
    # list returns a value for each process tracking the state - both local and remote, 
    # we just sum them to get the final value
    Enum.sum(ReplicatedState.list(server, key))
  end

  @impl true
  def init(_opts) do
    {0, _config = %{}}
  end

  @impl true
  def local_put(state, _config) do
    {:ok, state}
  end

  @impl true
  def local_update({:increment, by}, _delta, state, _config) do
    # we don't use precise data tracking, so we just use the state as out delta that will be 
    # propagated to remote servers in the handle_remote_delta callback
    {_state = state + by, _delta = state + by}
  end

  @impl true
  def handle_remote_delta(remote_delta, _old_state, _config) do
    # since the remote delta is just the remote state, we don't need to do any 
    # state mutation and the remote delta is the new state
    remote_delta
  end
end

The same abstraction will be used to re-implement Phoenix.Presence and possibly other things - it seems quite flexible. I’d recommend reading the docs in the linked PR - while the implementation is not ready, the docs should be close to the final thing we want. There’s also a mechanism for precise tracking of state changes (with the observe_remote_deltas callback which is not shown here).

+1 that terminate callback is not the appropriate place to cleanup counters of processes which are terminated. You need another GenServer to monitor these processes and perform the cleanup.

Instead of rolling your own, you could use gproc aggregated counters.

To make that work, you need to register the aggregated counter, e.g. in your application start callback, or in some top-level process:

:gproc.add_local_aggr_counter(:my_counter)

Now, in every process, you initialize the local counter when the process starts:

# invoke in each actor process
:gproc.add_local_counter(:my_counter, 0)

Where 0 is the initial count for that process.

When you want to change the counter value, you can use update_counter:

# invoke in each actor process
:gproc.update_counter({:c, :l, :my_counter}, increment)

The :c and :l indicate that you’re updating a local counter which is tied to the current process. If the process terminates, its count will be removed from the aggregated count.

To get the aggregate value (sum of all counters), you need to invoke:

:gproc.lookup_local_aggr_counter(:my_counter)

Demo:

:gproc.add_local_aggr_counter(:my_counter)

:gproc.lookup_local_aggr_counter(:my_counter)
# 0

# start one agent and bump its count by 1
{:ok, agent1} = Agent.start_link(fn -> :gproc.add_local_counter(:my_counter, 0) end)
Agent.update(agent1, fn _ -> :gproc.update_counter({:c, :l, :my_counter}, 1) end)

# The aggregated count is now 1
:gproc.lookup_local_aggr_counter(:my_counter)
# 1

# start another agent and bump its count by 2
{:ok, agent2} = Agent.start_link(fn -> :gproc.add_local_counter(:my_counter, 0) end)
Agent.update(agent2, fn _ -> :gproc.update_counter({:c, :l, :my_counter}, 2) end)

# the aggregated count is now 3 (1 from agent1 and 2 from agent2)
:gproc.lookup_local_aggr_counter(:my_counter)
# 3

# stop agent2
Agent.stop(agent2)

# The aggregated count is now 1 (1 from agent1)
:gproc.lookup_local_aggr_counter(:my_counter)

With the warning that the terminate callback is not always called and cannot be relied upon to run.