TomGrozev

TomGrozev

Microservice node communication with Phoenix

I am trying to setup elixir in a microservice architecture and have each of them (i.e. nodes) communicating efficiently. For context, there is a product microservice and a phoenix frontend microservice (there are more but these are relevant). I have got them connected through some custom module that is a dependency on both. This code basically spawns a task locally to spawn a task on another service to do whatever (i.e. create a product), this works great using a call similar to Connector.call(ProductMicroService.Products, :create, [some_product]). The downfall of this is when it comes to using Ecto schemas in Phoenix since it will try to call the schema module functions (e.g. struct) locally and not in the products microservice where the schema actually exists.

Now I have a couple possible solutions but none of them seem to appetizing and all seem very ‘hacky’. Some ideas I had are:

  1. Create copies of the schema modules on the frontend microservice
  2. Create fake schema modules on the frontend microservice that has all the functions but redirects calls the the real schema module
  3. Override the relevant functions in Phoenix (there are a lot)
  4. Some alternative?

For the more synchronous calls I believe the direct node communication is best and for asynchronous things a message queue (RabbitMQ, etc.) but is there a better way that I am missing? Anyway this is done, Phoenix still needs to be able to access these methods on the schema. It doesn’t seem to be an issue with anything other than phoenix.

Most Liked

Laetitia

Laetitia

In microservices architecture, the microservices share “raw data”, i.e the interfaces can expose maps, not struct. Each microservice must have its “view” of the model. For instance, a product in microservice A may have some different attributes from a product in microservice B. The output of microservice A will be a map, and microservice B will receive this map and convert it to its struct or schema.
Phoenix offers a backend-template application, so you don’t need microservices and you can use the schema along all your application but if you really need microservices, it’s a good way to use Erlang node communication and to apply decoupling between them.

al2o3cr

al2o3cr

Option 4: follow the first rule of distributed objects and don’t divide the system up this way.

AFAIK there aren’t any common patterns that would recommend turning context or view functions into remote procedure calls. Code in the “frontend” should interact with the “backend” via a clearly-defined API.

One other thing worth thinking about: the remote Task.Supervisor is single-threaded, so unless you add a lot of complexity that can be a bottleneck at high concurrency.

al2o3cr

al2o3cr

The phrasing is a little flippant, but the message is serious. Imposing a network boundary where there was only a module boundary before has significant costs:

  • duplication (for things like data definitions)
  • latency (network calls take time)
  • fallibility (network calls can fail)
  • deploy complexity (network calls may interact with servers that are a different version)

The biggest hazard in building a distributed system is that you get all of those headaches automatically, but you only get the good stuff (independent scaling, independent deploy, etc) with the right implementation.

For instance, an Elixir novice who confuses GenServer with “a server” and makes each piece of their application a single GenServer will produce a single-threaded performance tarpit.

The “rule” is a reminder to carefully consider the consequences of your decisions - what are you getting in exchange for all the additional complexity?

olivermt

olivermt

Could you elaborate a bit on why you want it this way? It sounds like you are adding a lot of complexity for gains I don’t see.

To use built in node communication in a somewhat easy manner you need codebases that are either the same or very aware of eachother, which sort of brings you back to a monorepo just split across more servers.

Why not use a http layer for the synchronous parts? Build a authority of sorts on the web node and pass it along with each request to keep the microservice acl etc.

mpope

mpope

I have seen this solved before in a Java microservice cluster and it might work well for your case. Services that communicate had a shared library of DTO class definitions. They weren’t the entire schema, but it was more of a request level object. The same could work for your schema definitions for task spawning. Two services could share a hex package (or even a git submodule) that has the shared definitions. That might not be a very ‘erlangy’ way of solving things, however it seemed clean enough in the Java environment. The downside was that both services needed a redeploy for schema changes, and the receiving serviced needed code around to handle the previous ‘version’ of message format that could be cut away after a full deploy.

Where Next?

Popular in Questions Top

gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
hpopp
To simplify some tasks at work, I wrote and published this package yesterday. It’s a simple macro that enables Access behaviour on struct...
New
sabri
Can someone explain the settings of pool_size of Ecto in config file? and what is the recommend size? Thanks
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

Other popular topics Top

yurko
Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for ...
New
peerreynders
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
326 29600 154
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

We're in Beta

About us Mission Statement