With 3k services this would result in 3k nodes, which would mean roughly 9M node-interconnections, since the BEAM wants to interconnect all nodes with each other. This won’t work well. You need to replace the messaging by another mechanism that does not require the full mesh.

Also you’d load the BEAM 3k times, while by putting some if not all of those modules alltogether on a single bare machine without docker, you can still update the modules one by one…

Running 3000 services in a single BEAM node should not be a problem. You basically start one (or more) processes per each service, and that’s it.

The second part of your requirement is indeed tricky. If you can afford to restart everything, your life will be much simpler. If not, then you must enter the realm of code reloading. Some basic instructions are available here.

In some simpler cases, this might work out of the box, If you actually have 3000 different modules (which I somehow doubt, but who knows ¯\(ツ)/¯ ), and cache previous release builds on the build server, then I think (not 100% sure though) that distillery will be able to detect the change and generate a correct appup automatically.

In more complicated cases, you might need to hand code an appup file. You can find some basic examples here. As far as I understand, appup is quite flexible. Among the low-level instructions there is apply which allows you to invoke a series of functions in an arbitrary order, so I it should be possible to do perform any kind of upgrade logic, no matter how complex it is.

I think that article was triggered by an earlier StackOverflow topic - where I found that one particular passage stood out:

So far I haven’t talked about microservices. That’s because, up to this point, they don’t really matter.

i.e. building (potentially distributed) applications with Elixir/OTP can leverage some of the benefits associated with microservices architecture without having to accept the overhead of supporting microservices architecture - but that doesn’t mean Elixir/OTP solutions necessarily would classify as realizations of microservices architecture - and that’s OK because microservices architecture isn’t supposed to be an end in itself. Meanwhile there are lots of principles that microservices architecture is based on that can also benefit Elixir/OTP solutions.

Of your post, to me, this immediately stood out. Because this kind of implies that at least on a conceptual level you currently have 3000 “beefy single boxes” processing your sources and yet:

An App per seems out of place,

… which kind of sounds like what you are doing right now. From my current research, still incomplete and ongoing - so I’m sure somebody will correct me if I’m wrong:

• Node: An executing Erlang run-time system (ERTS) which can communicate with other Erlang run-time systems.
• While a single hardware server (or virtualized OS) can run multiple Nodes, a Node itself cannot be distributed.
• Given that a Node executes an ERTS and that a Release includes an ERTS it seems that a Node can at most run one Release (???)
• While a Release can include multiple “applications” it seems to only support one primary application while the remainder are simply included applications. Therefore a Release seems to only support one supervision tree (???). So far my search for a release supporting multiple primary applications and supervision trees has come up empty.
• The point being is that a Node is executing one single primary application and it’s supervision tree.
• It would be tempting to equate (one Node == one microservice) - but that overlooks the level of isolation and decoupling that is possible within the same supervision tree.
• While the Node itself can’t be distributed, the application executing within it can spawn processes on other Nodes - though design-wise I would prefer an explicit (primary) application spawning and managing processes on that second Node on behalf of the first Node.

Essentially it seems the current starting point of your “Windows service” (per source) is roughly equivalent to a single primary application on a dedicated Node.

Now I suspect it’s unlikely that cramming all 3000 sources into one single Node/Application is that great an idea either - given that a single Node doesn’t distribute - so it’s more likely you are looking for an application that can be deployed on to multiple distinct Nodes, each Node listening to any number of sources via configuration - which all forward their pre-processed/normalized results to yet another Node which executes the application that concentrates/aggregates the results.

We are building an API which talks to around 50 services which is not huge, but they are all different providers which understand SOAP, XML, JSON, Rest etc,. And, we have had good success with just putting them in different modules. I think just having 3000 genservers with supervisors may work out without a lot of complications. Like @NobbZ mentioned creating 3K nodes is definitely not a good idea. You will also have to tune your :hackney settings (if you are using HTTPoison) as it sets the max connection limit to a default value of 50.

I would personally do the simplest thing possible, by building out some common utilities which the 3000 services can use. And run them on a single node.

This is the potential misunderstanding I wanted to address in my first reply. As far as I can tell a single node runs exactly one “primary application” - the other included applications act as libraries to the primary application by becoming part of the primary application’s supervision tree.

It’s a peculiarity in the OTP naming convention - a library application does not implement the Application callbacks and therefore cannot be started or stopped (as an application). So for example Poison is an “OTP application” but it’s a library application for use by a primary application.

$ mix new app_name

creates a library application - not a primary application. For a primary application

$ mix new app_name --sup

is required - that will include the application callback to create the supervision tree.

An umbrella project still contains just a single primary application - one of the “applications” implements the application callbacks and starts the supervision tree while the other (included) “applications” simply become part of the first application’s supervision tree.

but the question would be how to stop, start, pause and upgrade each service individually. Each service seems it would be a GenServer, but maybe an application?

In this situation the term service could cause some confusion/ambiguity. Handling a single source may require a number of processes, possibly even a small library application that could be designed to have it’s processes handling the source started, paused, and stopped. Upgrades could be trickier. In general hot code reloading targets code at the module level - and in many use cases the recommendation is to avoid using/supporting it because it tends to make the overall design much more challenging.

The work does not have to be distributed, so all 3000 on a single node is no problem.

The issue is

• it’s not exactly clear how severe a workload handling a single source is
• how capable the physical/virtual CPU is that the node resides on.

While a node will spread all its processes over all the cores of the CPU - it can’t scale by utilizing additional CPU’s (short of spawning processes on other nodes that reside on a different CPU). So while the total number of processes shouldn’t be a problem for a single node, the actual workload of handling 3000 concurrent sources could potentially be too much work for the CPU the node is executing on. If that is the case the solution design will have to account for the eventuality of distributing the workload over separate nodes each executing on their own CPU.

Taking service oriented design principles into account it may make sense to avoid sharing a node configuration database across multiple nodes but instead have a separate “configuration node” which supplies the other nodes with their configuration information when they start up (which could also route start, pause and stop requests to the appropriate node).

Ultimately the design details are affected by the expected workload and expected capability of your deployment platform.