Hi,

I have both Python and Elixir experience. Other languages greatly influenced me with their ideas. Beam VM and OTP inspires me with architecture decisions. Its documentation is very proactive and helps to see the problems from other perspective.

I have never worked with Pykka directly, only when debugged code in Python music player. This experience is not enough, so will not speculate on it.

As I understand, you have a server, that collects telemetry data. This server also needs to display interface for the user. It would be helpful, if you could give more context about the task. For example:

• Do you need to aggregate data before showing it? You can discard raw data after aggregation.
• Do you need a raw data? You can run other aggregations on it later.
• How fast the data comes in?
• How do you plan to store this data?
• What will be if server crashes? Is it OK, if it stops responding?
• Do you really have to implement both desktop and browser UI?
• Which web framework do you plan to use?
• Does a server fetch the data from these devices, or the devices send telemetry to the server?
• How much history do you plan to store? Do you have enough disk space for it (if everything will be on the single server)?
• Do you need to have separate permissions for people? I mean, do you need to hide some data from specific users?
• Do you need to show interface remotely? I mean, if it will be deployed to a public server? If yes, then it is better to protect it.
• How do you plan to scale the application, if you will have 2 servers instead of one?

I would split the task into 3 main parts. Receiving the data, processing it and presentation.

Receive (devices send data to a server)

If I expect large amounts of data, then I possibly need to store it raw before processing.
There are some ways to do it. For example, the data can be saved directly to a database (many options here), added to a queue (RabbitMQ, Redis, Kafka, etc) or possibly some other solution.

It is better to have these endpoints as fast as possible, so they can handle much data. Also I would load-test different solutions here. Possibly web servers with async features will work fine.

We can compare it to process mailbox in Elixir. A process receives many messages, but can handle them as soon as gets some processor time. So, we try to emulate a mailbox here.

Receive (server polls devices for data)

If server fetches data from many devices over the network, then this part should be as concurrent, as it can be. Running multiple Python processes will introduce overhead here, so possibly you will want to go with threads, or even asyncio-related libraries. Idea is the same - you need to fetch as many data as possible with as little overhead, as possible.

You can compare this to launching multiple processes in Elixir to get the data, but we try to emulate the same behaviour with features from Python.

Process the data

If you need to show a raw data, then you can omit this step at all.

If you need to process it somehow, then you can look at something like Celery. I have also worked with Apache Airflow, but it can be overkill here. So, the basic idea is to get a chunk (batch) to process and store the results somewhere. The details greatly depend on the use-case.

Present data

It depends on the type of data, its amounts etc. Possibly you can just generate pages with charts via Flask or Django templates. Besides, you can just refresh a page every N seconds instead of using websockets. It can be just refreshing the whole page, only a part of it, or fetching data from API. Example with Stimulus, example with unpoly.js, example with HTMX. I would like to have only one UI. If you do really need a desktop application - possibly there is a way to show the same UI in a web-view.

Also there is Plotly Dash - have used it to render dashboards. It also supports open-source version.

General advices

I would recommend Designing Data-Intensive Applications book. It gives nice overview of tradeoffs, architecture details and approaches to work with data. It does not dive too deep into details, but gives many ideas (at least for me). Enjoyed reading it (and watching nice pictures there).

Another book with useful tips for overloaded system is Erlang in anger. It describes some typical issues with overloaded system in Beam and how to fight them. But you can apply ideas in it in other languages. Actually you can use actor model to design the system, but it will consist in Operating System processes instead of BEAM VM ones.

I would also recommend not to make the system too complicated - you will be grateful for it later. Possibly the simplest solution will work fine for you. The more moving parts you have - the more of them you need to support.

Hope it will help you, or at least give some ideas.

Hi thanks for a reply! I am glad that it helped a little bit

So, as I understand you have something like this:

• Multiple remote servers. They connect via TCP/IP to a central one.
• Central server gathers information, converts it to the required format and stores it to the database.
• You need to send alerts if the data is out of bounds.
• You need to display data.

According to your description, it is not an ordinary HTTP server. I will be able to provide only general advice in this case. Possibly it will give you some insights. You know the requirements better and will be able to decide how to structure the system.

Regarding multiprocessing.

Here is a nice article about different ways to speedup Python code.

Types of tasks:

• CPU-bound tasks - working with numbers, compressing images, transcoding video, etc. In simple words - the faster PC you have, the faster you get results. You can speed up processing by running multiple parallel tasks.
• IO-bound tasks - writing/reading files, network requests, etc. You need to wait the response. Faster PC does not mean that you will get results faster. You can speed up this by multiple concurrent tasks.

Difference between processes and threads in Python:

• Process - OS process. Has one thread by default (you can launch more later). Each Python process has a single GIL (at least now). GIL allows to execute Python code in only one thread at a time.
• Thread - part of a process. All threads share the same GIL. One thread does some CPU work - other threads wait. If one thread does too much CPU-bound work it blocks other threads in the process, and it “freezes”.
• Asyncio loop - Runs inside a single thread. “Event-based”. If you run too much CPU-bound tasks in a coroutine - it may block the whole loop (and thread).

Types of IO:

• blocking - You opened a TCP socket. When you read from socket, the thread waits until you receive a message.
• non-blocking - You opened a TCP socket. You can periodically poll a socket if it has some data. Or you can set timeout on a read/write operation. The code will throw error if received no data before timeout.
• async - You opened a TCP socket. You want to read some data and “subscribe” (or “register a callback”) to “data available event”. Event loop does other jobs. When it sees a “data available event” from your socket, it runs your callback.

Major part of Python code and libraries use blocking IO (at least now).

So, here is a difference between between multithreading, multiple processes and asyncio:

• Multiple processes - Each process is a separate OS process with a single thread (you can launch more threads later). It loads full Python (takes more RAM). Sharing memory is limited. Best for running multiple parallel CPU-bound tasks (Python GIL will not interfere here).
• Multiple threads - They share the same process. Sharing memory is easier, but may lead to race conditions (use it with caution). You can use it to run multiple concurrent IO-bound tasks (many Python libraries support only this kind of operation).
• Asyncio - coroutines share the same process. Has same memory and GIL issues as multiple threads. Best for async concurrent IO-bound tasks. Not all Python libraries support async features, use them with caution (they may block the whole event loop).

I gave a very high-level overview of how Python system works. If you have only one process, and some part takes too much CPU time - the whole system may freeze and stop responding. I would not recommend to run many background tasks. Also each process has a single GC - it can pause the whole system if it consumed too much memory. Sometimes Python leaks a memory, so GC is possible for active long-running processes.

Regarding Celery

Celery runs background tasks or . It starts a couple of “worker processes”. Each worker process subscribes to a “broker” - message queue (example brokers are Redis or RabbitMQ). Worker reads a task from the queue and processes it. You can put the task to the queue at any part of your application (even from another task). “Celery beat” may run scheduled tasks (like cronjob).

Regarding message passing between processes

I am not sure that I understand the reason to pass some datatypes between processes. Possibly there is a simpler approach.

Regarding Plotly Dash

I think it will be better to use Grafana here, if it fits you. Plotly Dash allows you to write dashboards in Python code. Used it for small-medium datasets, so have no experience if it scales well and fits your needs. Suggested it as an option (not as a strong recommendation), so you can try it.

My thoughts about possible architecture

Warning - this is not a call to action, but only a point of view.

As I understand, the task is to:

• connect to the remote servers
• periodically get some data from them
• process a data and put it to the database
• periodically check for outliers and send alerts

I think the system may consist of this parts:

• “Connectors” - Each connector connects to a remote server. It gets a data, processes it and puts it into the database.
• Periodic or background tasks - If you don’t need fast alerts, then possibly, you can periodically query the most recent from the database for outliers. This may be once-a-minute periodic job. If you do some heavy processing (it seems not in your case), then you may offload this to background jobs. General recommendation about background jobs - it is better to put the data into the database (possibly into other table) and launch specific job with record ID, instead of serialising the whole structure.
• Alert system - Optional, if you need to report alerts as soon as possible.
• GUI - You know the requirements better and can pick the one that fits your needs better. You can get the data directly from DB, instead of serialising objects between processes.

If you need a realtime updates, then you may create a message queue (possibly consider external one). If a connector sees the outlier, it publishes the id of the outlier into the “outliers” topic. Alert system reads this topic and takes does relevant actions.

So the algorithm like this:

• Connector fetches the data.
• Connector processes data and stores it in the database.
• If the data is wrong, connector raises alert (possibly it can just put the data to another database table).
• GUI takes the data from the database and displays it.

As I understand, you have a very similar solution:

• You have asyncio connectors, that connect to the remote servers, fetch the data and put it into the database.
• You can send commands to these connectors via asyncio.Queue.
• Your future GUI shows the data from the database.

Again, you know your system, requirements and constraints better than me, so please, do not take any action unless it makes sense to you. My message is just an opinion, not an order or call to action. I can not read minds or debug systems/architecture without seeing a code.

I wish you to succeed with your system. Possibly this message was helpful for you.

Hi Roman,

I apologize for my delayed response. Thank you very much for such a detailed response and helpful pointers! I read it at the time you replied and kept doing some research. To provide a little more detail, here is a kind of architecture I’ve landed upon:

image1114×499 39.5 KB

However, even that is a little bit of an old design, because as of yesterday, we no longer have a need to separate the GUI over a network, so it’s likely that the WebSocket server is no longer needed and can become some sort of internal worker process.

If server fetches data from many devices over the network, then this part should be as concurrent, as it can be. Running multiple Python processes will introduce overhead here, so possibly you will want to go with threads, or even asyncio-related libraries.

There are indeed a lot of external devices, primarily TCP/IP servers. So I have been landing on the idea of using asyncio for all (or at least most of) the TCP/IP requests with asyncio.Queues for message passing, treating the various asyncio streams clients as little workers. I think you are right that creating many Python processes introduces a lot of overhead (because of the additional network calls between the processes) and orchestration (because of having to rely on the OS to manage these) needed. And it also requires the need for serialization/deserialization between the processes, instead of using native Python datatypes and of course custom ones using dataclasses and typing.NamedTuple.

Between threading, asyncio, multiprocessing, and separate Python processes, I mainly landed on asyncio because of the very nice APIs it has (such as asyncio streams). For a threading solution, I probably would lean on Pykka since it is the actor model on top of Python’s threading, but I worry about the context switching performance hit. For multiprocessing, I considered a queue-based solution there, but I am unfamiliar with multiprocessing for IO-bound applications, and asyncio seemed to be the recommendation there. And as mentioned, the multiple, separate Python processes feels unwieldy, slower, and requires serialization/deserialization for datatypes.

Does that sound accurate to you? Or a valid approach and decision criteria?

If you need to show a raw data, then you can omit this step at all.

There is some processing, but it is mainly parsing the text-based responses into datatypes and then passing them around to a timeseries database and the GUI. There may be one or two procedures that we need with heavier calculations, but even there it shouldn’t be too much to handle. If so, then perhaps multiprocessing is the way to offload the calculation. The aggregation is primarily simply collating the data and then displaying and saving it. There will be some watchers that look for values out of bounds that then provide alerts. But such calculations are like “if outside of a given numeric range, then alert”.

I think in general, I have ruled out the need for and complexity of things like RabbitMQ, Celery, etc.

Also there is Plotly Dash - have used it to render dashboards. It also supports open-source version.

Thank you for mentioning that. I am aware of Plotly, having used it in F#, but I haven’t ever used Dash or looked into it much. Right now, the plan was to write data to a timeseries database (such as InfluxDB, QuestDB, or TimescaleDB), and then view that data with Grafana. How does Dash compare to that approach?

Thank you again, Roman! I greatly appreciate the time you put into your original post.

Hi Roman. Thanks again for your messages and advice. I’m just now getting back to your latest message, and it’s very interesting because it looks like we independently came up with some very similar things, which I suppose is not a surprise given the familiarity of messaging systems that Elixir/Erlang brings. Here’s a prototype I recently built that creates the concept of a worker (I avoided the term actor because of some non-technical reasons, although I like the term actor) which contain inboxes that wrap asyncio.Queues for messaging. Here’s the Python discussion forum post where I describe the prototype in more detail: Request for review of PySide6 (Qt for Python) and asyncio task prototype - Async-SIG - Discussions on Python.org

Hi,
The Elixir/Erlang systems are very nice - you can apply their ideas in other languages too.
I have recently realised, that the author of Python gunicorn web server is very closely related to Erlang ecosystem and has popular libraries here too. Gunicorn uses the analog of supervisor to increase a number of workers and process more requests. Supervisor watches the workers and restarts them once they crashed, or after a certain number of requests (--max_requests setting). This allows to reduce memory leaks of long-running Python processes.

By the way, you can consider using asyncio.PriorityQueue instead of asyncio.Queue. This can help you to add “signals” to your workers. Signal is a more urgent message. If you don’t have priorities, the urgent message will wait until all previous messages are processed. With priority, it will wait only the completion of processing of your current message.

I have looked into your example - it is pretty nice!
Regarding the scaling:

• Side note. I think you will need to adjust a linux file limits (ulimit -n), if you want to open thousands of connections from the single process. Otherwise you will get something like Too many files open error.
• It is better to create a benchmark and test by yourself if it scales well. You can create a simple socket server in Elixir. It can use Mix.install (examples) together with ThousandIsland. You can deploy this server to other PC and open as many connections as you wish (or as your system allows). This will be the best proof if your approach scales well and will allow to find issues with the design.
• You will may want to split workers into multiple processes (for example according to the number of CPU cores). You can have a main “supervisor” that will launch workers and distribute clients between them. If some worker is stuck - it will affect only it. “Supervisor” may exchange heartbeat messages with workers and restart/notify/send event to Sentry about unresponsive workers. Possibly each worker will also have separate ulimit -n limits (but it is better to verify it during a test). Each worker possibly will need to speak only to the “supervisor” and not to other workers. You can partition data between workers as mix test (a sole link in my the previous message).
• Theoretically this approach should scale well, unless you do too much sync computations in a coroutine. It is better to simulate it with a real-world like data. It is better to run simulation for some time (possibly for an hour) and measure the metrics of the system. You will be able to see CPU utilisation in a real-time with htop or similar application. You can periodically measure amounts of RAM (as well as other useful metrics) and print them into stdout in a CSV format (just separate them with commas or other delimiters). You can store the results in a file with simulation.py > outputs.csv or simulation.py | tee output.csv (if you want to see the logs in the shell too). You can load the results in Excel, and create charts from it. By the way, you can print different data to stderr and stdout. For example, you can use stderr for debugging information, and stdout for metrics, or vice versa.