How many sensors are we talking about? Emitting once every minute is slow enough that you should only worry about performance and partitioning if there are enough sensors.

Also what should happen if something fails? Is it ok if you drop messages? What should happen if after 6 minutes your accumulation logic crashes. Is it ok to just restart from the current message or do you need to restart from the first message?

Do you need to store all incoming data for later usage? Or can you just drop it. What about the accumulation, do you need to store that or just emit it somewhere?

These would be the kind of questions I’d ask myself before making a decision on anything. Because they will affect how you want to design your solution.

Depending on some of the answers to the above question. I’d probably start with a MQTT client and handle the logic myself. I don’t know flow so I can’t comment if it would be a good choice for you.

If you do not need the actual values the time series database can do the aggregation. You can then pull from the DB, which will simplify the architecture.

Can you save your accumulation after every message or only after 10 messages? Because that will make a big difference if you can let the broker handle unacknowledged messages or not. Basically if you save accumulation after every message then you can acknowledge right after saving. In the other case you’d need to wait 10 minutes. that’s something the message broker probably doesn’t support…

A time series database might definitely be worth looking at, but for me it would depend if the added complexity of the extra database (although there are postgres extensions as well) is worth it.

I’ve built something similarly in the past. With rabbitMq and postgres. I could easily handle 1000s msg/sec on a fairly regular server. Don’t remember the exact details but nothing very beefy was needed.

Use Shared Subscription

Shared subscriptions are an MQTT v5 feature that allows MQTT clients to share the same subscription on the broker. In standard MQTT subscriptions, each subscribing client receives a copy of each message that is sent to that topic. In a shared subscription, all clients that share the same subscription in the same subscription group receive messages in an alternating fashion. This mechanism is sometimes called client load balancing , since the message load of a single topic is distributed across all subscribers.

image720×504 19.8 KB

Along with HiveMQ

Reliable Data Movement

for Connected Devices

HiveMQ’s MQTT broker makes it easy to move data to and from connected devices in an efficient, fast and reliable manner.

image420×250 19.2 KB

HiveMQ is architected for scale and reliability. MQTT broker scales up to 10 million connected devices and uses industry standards to ensure data is not lost.

Followed by Genstage based Flow

To intercept and perform business logic before storage.

Use Clickhouse for Timeseries storage

It’s an OLAP + Time series db, faster than Timescale & Influx DB.

See: Benchmark

Definitely use Broadway

They solved the problems that arose after people started building complex GenStage applications.

1. Rate Limiting
2. Batching
3. Metrics
4. Graceful shutdown
5. Automatic acknowledgement at the end of the pipeline
6. Fault tolerance with minimal data loss
7. Backpressure
8. Concurrency

Instead of reinventing the wheel, write a MQTT connector for Broadway if existing ones fall short.

See why Broadway is needed:

Learning material: