heathen

heathen

What is the best way to open external connections in a CQRS\ES application?

I’m working on a project one part on which requires information (for example, external systems resource consumption info) from one or many external sources of the same type. These external systems don’t have push or subscription options so I need to poll them periodically. The tricky part is that users must be able to add new or delete already added sources at any time as well as to change connection properties (like url, auth info, polling interval and so on). It is possible that in the future it will be possible to get information on subscription scheme.

This particular application’s part is a kind of connector\translator events from the external systems to the core domain. I’m going to use Commanded on the whole project.

I want to put this part into a Connectors subdomain and describe the Connection aggregate (names are debatable). I want to start polling right after this aggregate creation and stop on deletion.

So the question is what is the best way to start the polling processes?

In general it could be a GenServer process (as I need to keep some service information like the last success call timestamp, prevent parallel connections and so on) with poll request scheduling. I evaluate Commanded’s process manager but is it a good idea to make external calls right from it (through ports\adapters, of course, but still)? Or is it better to start an additional process via the same process manager on ConnectorAdded\ConnectorResumed events and stop it on ConnectorDeleted\ConnectorPaused?

Most Liked

bottlenecked

bottlenecked

Hmm… I think that @factoryd is on to something here perhaps. Greg Young warns on a few occasions “please dont tell me that you’ve built an event sourced system”, meaning that it only makes sense applying the pattern to a small part of your system where it’s actually really needed, not least because ES comes with a ton of overhead. There are actually quite a few wartime stories about botched ES attempts, and a common theme is attempting to force ES everywhere.

The reason I’m saying this is that it looks like you should perhaps think about separating your app into 2 parts: the first should only concern itself with the polling aspect (configuration screens, external apis etc.) that will produce commands that will then be pushed on to the second part of your system which should only be concerned with accounting (your core business, which may actually benefit from ES), that will simply consume the commands from the first part, convert those into events, apply them and finally publish them for the rest of the system to act upon (you know, the canonical ES flow).

Again, that would be my advice: do not mix a ton of different concerns in an “event sourced application” because this pattern tends to multiply the effects of bad system design decisions

bottlenecked

bottlenecked

OK, I see. Perhaps it’s worth a shot then as a learning experience! But if this solo/personal/internal project ends up being used everywhere (as infrastructure projects are sometimes wont to be) you should make sure that you will be given the required time (by your manager for example) to revisit your designs in case they end up not fitting the problem

Where Next?

Popular in Questions Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
senggen
Erlang/OTP 25 [erts-13.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] 15:22:35.803 [error] gen_event {lager_file_backend...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

Other popular topics Top

dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
273 38985 115
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement