I’m researching a project where I need to talk to a lot of embedded devices (let’s say it’s 10s of thousands of devices). We need to be able to send/receive commands, state, configurations, see online/offline status etc.
We are currently discussing two options that I’d like some advice on, if you have experience.

Option 1: MQTT
Basically, we’d have

Device <-- MQTT --> MQTT Broker <-- MQTT --> Elixir App <-- HTTPS --> Other services

Here all devices connect to the MQTT Broker, and also the Elixir app. Other services can all the HTTP endpoints to send commands or query devices without having to deal with MQTT.

Pros:

• Devices does not drop connection on deploys etc of the Elixir App
• MQTT is a battle tested and performant protocol with good brokers around
• Built in QoS (exactly-once, at-least-once delivery etc)
• Elixir app would be a bit simpler

Cons:

• Another service to configure and maintain
• Expensive to pay for hosted/managed options (especially if you want persisted messages etc)

Option 2: Pure Elixir

Here we’d skip MQTT and just connect directly, like:

Device <-- Websocket? --> Elixir App <-- HTTPS --> Other services

The idea here is that we skip MQTT and all devices connect directly to the Elixir app (maybe using Phoenix Channels if possible). We’d run processes for each device which handles and routes commands and requests appropriately.

Pros:

• Only one service to maintain
• Can do whatever we want in Elixir compared to only features available in the MQTT broker

Cons:

• All devices will drop their connection on deploys/restarts and will have to reconnect
• A more complex Elixir application

For me, I’m leaning more towards option 1 since I see a lot of value in not having to deal with the long running connections, they are instead managed by the broker. But, I also haven’t really used MQTT before (other in some toy projects)…

Has anyone done anything similar and have some advice?