Hi everyone. I have an interesting case at work. I’m developing a service which ingest data from an external data provider via an exclusive AMQP queue. Since it’s an exclusive queue, when the consumer disconnects, the queue gets destroyed and having more than one queues means I have to de-duplicate messages. My task is to minimize the chance of missing messages in case of flaky connection or if something happens to the container the service is running in.

By using supervision, I can minimize the missed messages when the connection is flaky but how to handle containers crashing? I’ve identified two ways to tackle the problem both of them presume a cluster of at least two instances of the services. One approach is to register the AMQP connection process as a global process by using :global or horde. The second approach consists of creating a process group with :pg and synchronize the connection and channel where the queue is declared. I couldn’t manage to make it working reliably when nodes are started at the same time.

Are there any alternatives and am I missing something? What approach would you advise me to use? Thanks in advance.