antoniosb

antoniosb

Best approach to use a project with its own database/Repo as a dependency in another project

I have an elixir project that does some crawling and saves the crawled data to a postgres database, called Tenjin.

Tenjin contais all the logic to perform the requests, parse the responses, validate the data and input it to the database using its own entities.

Now I want to create a graphQL api that will serve the data crawled by Tenjin. As I may use Tenjin in another projects, I don’t want to create the API in the same project, thus not coupling the crawling to the API.

Let’s say the api project is called Omoikane. I want that Omoikane can call and use Tenjin modules and functions in order to serve the data that is available on Tenjin.

But the database configuration and crawling settings are all defined at Tenijn, how can I build Omoikane as a complete separate project, perhaps with its own repo and database, but still using Tenjin as a dependency or so?

Thanks in advance <3

Most Liked

peerreynders

peerreynders

That sentence points towards this solution:

  • Clearly Tenjin has been implemented as a single project
  • Omoikane calls some Tenjin modules
  • The functionality currently covered by Tenjin falls into two broad categories: 1. crawling 2. data management
  • Its unlikely that Omoikane will need access to “crawling”. Furthermore it may not even need access to “inserting” or “updating” data.
  • “the touching points at Omoikane” is identifying a third service - common to both Tenjin and Omoikane. That service should be as narrow as possible in order to minimize Omoikane’s coupling to Tenjin.
  • Ideally you should be able to deploy DB + “the service” + Omoikane without the rest of Tenjin to have your functioning API - the fact that it is never going to be deployed this way is irrelevant - the point is to minimize coupling and to make it very clear what functionality is common so that it is clear when Tenjin is maintained which changes will affect Omoikane.
  • From a very high level “the service” looks a lot like a Repository - this is not to be confused with the Ecto repo - the Ecto repo would be part of “the repository”
  • Given that Omoikane may only require read-only access, you may also have Command Query Separation going on
    • I’m not talking about a full blown CQRS system. I’m talking about one module (and possibly multiple support modules) that is only used by Tenjin for inserting/updating data. Then a second module (and possibly multiple separate support modules) that is used by both Tenjin and Omoikane for fetching data.
    • Omoikane may not require the detailed data access that Tenjin needs - so there can be an argument for a two versions of the second module, one for Tenjin, the other (“less detailed”) one for Omoikane - and both versions would likely share a significant number of the support modules. Again the idea being to couple Omoikane only to the significant parts - not to the parts that only Tenjin needs.

You want to enable a deployment where

  • One node runs the DB
  • A second node runs “the service” + Omoikane
  • A third node runs “the service” + Tenjin

while designing it so that initially you can run

  • DB + “the service” + Omoikane + Tenjin on a single node

Bottom line: “touching points at Omoikane” identifies a commonality within Tenjin that should be factored out for the sake of future maintainability; there is value in identifying that commonality and factoring it out now (it identifies a boundary that is worth exposing/segregating at least at the module level of a possible repository application. What you are currently calling Tenjin is actually Tenjin + repository).

al2o3cr

al2o3cr

These two desires are directly in conflict with each other: if you’re sharing code and a database the projects are tightly coupled together - putting them in separate repos doesn’t remove that coupling, it just makes it more complicated.

You have a couple options:

  1. as mentioned by others, put both applications in an umbrella and have Omoikane depend on Tenjin
  2. extract the relevant modules and functions to a third place and have both the Omoikane repo and Tenjin repo use that as a dependency. This approach STILL tightly couples the two together via their expectations about the schema, but their code is separated
  3. create a public API (HTTP? Protocol buffers? grpc?) as part of Tenjin that Omoikane can interact with

Going down the list, each of these options requires writing progressively more code to implement but they also provide progressively more isolation between the applications.

idi527

idi527

:wave:

Maybe you can separate the repo into its own app and make both other apps depend on it.

apps/
- repo
- tenjin (depends on repo)
- omoikane (depends on repo)

You’d still be able to package them separately since the common component is extracted from both of them.

Where Next?

Popular in Questions Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
Werner
Hi, I’m using Ubuntu 18.04 and after updating to OTP-24.0 yesterday i have this warning when I run “mix local.hex”: 14:57:30.512 [warn] ...
New
SoCreat
i’m a new one to elixir which editor can i use vs code? or atom? Thanks! :smiley:
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
Fl4m3Ph03n1x
Background Let’s assume I have a typical GenServer that receives messages as requests, does some operation in a DB and returns responses....
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 42633 214
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
Nvim
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New

We're in Beta

About us Mission Statement