would also not suffer if something was a bit delayed or if the same message was received more then once.

The concern is not a “delay” or “duplicates” but actually the loss of the messages.

with my current payload size and load I easily get 3.5GB of transferred data from the main node in just 24 hours.

why do you care about that number?

1. GC takes care of it
2. large binaries are optimized to not be copied in the same node between processes (I think >64kb on 64bit machine)

Now since you are dealing with LiveView, @lud gave a very valuable point about cutting down the number of messages sent to client.
That part actually can cause your server to get OOMKilled!

Watch this part of Marlus’s talk “Optimizing LiveView for Realtime Applications” in that last section he demonstrates how memory of Phoenix app can grow unbound if client can’t receive messages with the same rate they are published! (and overall the whole talk is a gem)

Can you add the origin node in the message, publish only to chmmon topic and filter locally ?

This will divide your bandwith by like 1.8 at least.

You can also use the compression options of term_to_binary/2

You could write the latest values to a database table and use Postgres pubsub to listen for changes, removing the need for Phoenix PubSub (and maybe Cachex). Or at least just publish a small value on Phoenix pubsub when the database has changed.

Also 10 million messages is a lot for a view. It is 115/s on average (or is it not ? I suck at math). Could you just send the last value every 1 second ?

Hi, @mortenlund!
I can’t give you definitive answer, but there are some things come to my mind to consider:

Phoenix.PubSub uses adapters to deliver messages, by default its :pg2. However, it could be Redis, Postgres, or even RabbitMQ or anything else, that can act as a message bus…

Now, it’s worth mentioning that default :pg2 uses distributed erlang… In general most of the available out of box tools in distributed erlang provide with eventual consistency. And they don’t provide with any “at least 1 delivery guarantee”.
If you need some guarantees - you would most likely need to opt in to 3rd party (RabbitMQ is a very strong candidate)

I’ll address just this because the answer is “Yes”.

I built a pusher clone that was pushing payloads between 100 bytes and (too many) MB in size. We pushed easily > 300,000,000 messages per day. I don’t know the aggregated size of the messages. We had ~5 servers so each message was broadcast to all of the other servers.

I never saw performance problems from pushing too much data here. It was almost entirely hands off except for unrelated problems over time.

It’s important to note we pushed using built-in node-node connections. We did not use PubSubRedis connection.