I started exploring this library a few hours ago and it is extremely good. I have used horde in the past and pogo more recently. Both of those libs are great but process_hub is truly a joy to use.

There is a lot to love and I have not even started using the hot-swapping and hooks stuff. Reading the list of hook events blew my mind. My imagination was piqued.

I love the simplicity of the hub_id atom. This plays well with Fly because the fly region can be used to build the hub_id, for simple per-region clustering.

I also love the way you use local ETS tables for fast lookup. Perfect use-case for ets.

I am still doing a refactor on a feature branch to see if my own hacky :pogo-based attempts at distributed process management can be replaced with :process_hub. So far I refactored a cache for oauth “state” and a per-kitchen “global singleton” order number generator process. I have way less LOC already and it is far simpler to read and understand.

I am in awe. Congratulations.

Thanks for another update!

In my current project I use :syn (GitHub - ostinelli/syn: A scalable global Process Registry and Process Group manager for Erlang and Elixir.) for process registry for a pub/sub mechanism and :libcluster (GitHub - bitwalker/libcluster: Automatic cluster formation/healing for Elixir applications) as as an automated clustering mechanism as well as ProcessHub for process distribution and control. It does seem like there is overlap (there is!) but each does a good job for what i use it for i.e. i am using selected bits of each one as they all have there strengths and weaknesses.

I was thinking is it worth making the process tracking method selectable so you can use/make different implementations if you wish? i.e. yours, pg, syn, custom etc as each has trade offs.

My project is just a hobby project but its intended to make a changing cluster of machines look like a single machine with a single memory space and have the system behave as if its a single machine with automated handling of nodes joining and leaving the cluster and auto migration of processes to keep the system complete and running.

Its a play thing and speed and memory constraints are not of any concern and i intend to use a lot of processes like objects (similar to GitHub - wojtekmach/oop: OOP in Elixir!) but with all the genserver stuff hidden behind a macro DSL. You could then submit a program to the cluster without knowing or caring about where it is running.

Well enough rambling (sorry!), the joys of retirement is you can play with all these interesting techs!!

Best wishes

Shifters

New version 0.3.0-alpha released!
This release focuses on performance and stability improvements.

Changelog:

### Breaking changes
- `Hook.post_children_start()` now returns: `[{child_id(), result(), pid(), [node()]}]`
- Fixed error handling on process startups and stoppings. Returning more information about the failures and partial successes. Example:`ProcessHub.start_children/3`, `ProcessHub.stop_children/3` with `async_wait: true` now on failures return: `{:error, list_of_failures(), list_of_partial_successes()}`. This does not affect successful operations.

### Added
- Added new option `on_failure: :continue | :rollback` to the `ProcessHub.start_children/3`, `ProcessHub.stop_children/3` functions. This option allows the user to specify what should happen if the operation fails.

### Fixed
- `ProcessHub.Strategy.Redundancy.Replication` was not properly updating the redundancy_signal value on some occasions
due to race condition.
- Fixed issue with `ProcessHub.Janitor` not purging the cache properly when using Gossip protocol.
- State passing on hotswap migration with graceful shutdown fixed.
- Timeout option was not properly used in some cases.

### Changed
- Improved typespecs across the codebase.
- Improved overall performance of start/stop/hotswap migration operations which involved synchronization and large
amount of message passing by using bulk operations.

I also conducted performance tests, and the results show that synchronous operations and migrations are now multiple orders of magnitude faster.
Many potential issues have also been resolved.

There’s still a lot to improve, but here are the results for spawning X distributed processes across a 10-node cluster and stopping them synchronously:

This includes all distribution logic, registry updates, hooks, event dispatching, and result messages for startups and shutdowns etc.

Up until 10k distributed processes in a 10 node cluster the library does okay imo. On average 2.6 seconds to start and stop everything synchronously. After 20k processes the jump is pretty big.

Operating System: Linux
CPU Information: Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
Number of Available Cores: 8
Available memory: 31.23 GB
Elixir 1.17.1
Erlang 27.0
JIT enabled: true

• 10 nodes, 10 children

Name                                         ips        average  deviation         median         99th %
start_&_stop_processes-0.2.10-alpha        27.43       36.46 ms    ±67.09%       33.20 ms       84.64 ms
start_&_stop_processes-0.3.0-alpha        290.82        3.44 ms    ±46.01%        3.26 ms        6.51 ms


Memory usage statistics:

Name                                       average  deviation         median         99th %
start_&_stop_processes-0.2.10-alpha        2.31 MB    ±31.30%        2.83 MB        2.83 MB
start_&_stop_processes-0.3.0-alpha         2.08 MB    ±36.49%        2.82 MB        2.82 MB

• 10 nodes, 100 children

Name                                         ips        average  deviation         median         99th %
start_&_stop_processes-0.2.10-alpha         2.50      400.57 ms    ±62.90%      421.35 ms      863.76 ms
start_&_stop_processes-0.3.0-alpha         57.59       17.36 ms    ±47.88%       18.34 ms       35.52 ms

Memory usage statistics:

Name                                       average  deviation         median         99th %
start_&_stop_processes-0.2.10-alpha       31.32 MB    ±12.02%       32.24 MB       32.24 MB
start_&_stop_processes-0.3.0-alpha        24.02 MB    ±33.93%       32.13 MB       32.13 MB

• 10 nodes, 1000 children

Name                                         ips        average  deviation         median         99th %
start_&_stop_processes-0.2.10-alpha         0.22         4.49 s    ±27.37%         4.42 s         5.76 s
start_&_stop_processes-0.3.0-alpha          7.00      142.89 ms    ±50.88%      172.62 ms      297.17 ms

Memory usage statistics:

Name                                       average  deviation         median         99th %
start_&_stop_processes-0.2.10-alpha      326.14 MB     ±0.01%      326.14 MB      326.34 MB
start_&_stop_processes-0.3.0-alpha       248.12 MB    ±32.89%      324.90 MB      324.94 MB

• 10 nodes, 10000 children

Name                                         ips        average  deviation         median         99th %
start_&_stop_processes-0.2.10-alpha       0.0116       1.44 min     ±0.00%       1.44 min       1.44 min
start_&_stop_processes-0.3.0-alpha          0.38         2.60 s    ±40.86%         3.09 s         3.11 s

Memory usage statistics:

Name                                       average  deviation         median         99th %
start_&_stop_processes-0.2.10-alpha        3.48 GB
start_&_stop_processes-0.3.0-alpha         3.04 GB    ±28.03%        3.47 GB        3.47 GB

At this point the testing the old version took too much time and I just tested the newer version.

• 10 nodes, 20000 children

Name                                         ips        average  deviation         median         99th %
start_&_stop_processes-0.3.0-alpha         0.193         5.18 s   ±111.63%         5.18 s         9.27 s

Memory usage statistics:

Name                                       average  deviation         median         99th %
start_&_stop_processes-0.3.0-alpha         5.75 GB    ±46.80%        5.75 GB        7.65 GB

• 10 nodes, 30000 children

Name                                         ips        average  deviation         median         99th %
start_&_stop_processes-0.3.0-alpha        0.0277        36.09 s     ±0.00%        36.09 s        36.09 s

Memory usage statistics:

Name                                  Memory usage
start_&_stop_processes-0.3.0-alpha        11.47 GB

• 10 nodes, 40000 children

Name                                         ips        average  deviation         median         99th %
start_&_stop_processes-0.3.0-alpha       0.00925       1.80 min     ±0.00%       1.80 min       1.80 min

Memory usage statistics:

Name                                  Memory usage
start_&_stop_processes-0.3.0-alpha        15.95 GB

Future releases will address the memory allocations.

I’m pleased to know that there’s interest in the development progress. I’ll continue to update any future releases here.

I’d like to share some plans regarding the library. One of the primary upcoming goals is to optimize the library by identifying bottlenecks and areas where improvements can be made. The most recent release introduced configurable timeouts, which are essential for larger systems with a significant number of nodes and processes, and which can tolerate higher latencies. This enhancement has also laid the groundwork for performance testing and fine-tuning. I’ll update the documentation later by including general guidelines for configuring timeouts and other parameters based on the cluster size and the number of processes.

Additionally, I’ll need to expand the documentation and provide more guides because various configurations and methods are undiscovered in the current documentation.

Prior to optimizing and improving the documentation, I’ll enhance the hook system to make it more robust. This will provide additional ways to extend or modify the system’s behavior. At this point, it should be relatively straightforward to replace built-in strategies with custom implementations that can hook into the system at different points as needed and execute code.

Hopefully 0.x.x-beta soon

Good catch! Thanks for the pr!
First pull request for the project