If I got that correctly, your app uses an external API to fetch data, transforms that into your own format, saves that in DB and then exposes that through your own API. The problem is that the external API is really unreliable. As a consequence, you don’t really have an option to implement it via hot calls.

So what I’m suggesting is that you “clone” the external database. You have options here:

1. dump the responses from the API straight into the DB: the advantage here is that you can easily change your own JSON schema (add more attributes, etc.), since the external data sits in your DB
2. store the transformed entities in the DB: the advantage here is speed (negligible)
3. do both

Seems like the problem that you need to solve here is to decide what data and when to pull it from the external API.

As to what:

• can you store everything locally? what’s the estimated data set? can you afford this?
• does your application need all the data?

As to when:

• does the external API has web hooks or any other notification mechanism?
• does the external API has an incremental endpoint (i.e. give me what changed since timestamp)?
• if you need to poll for data, what’s the acceptable timespan? how is this affected by quota?

Depending on the answers, the whole thing might event turn infeasible (I hope not!).

I’d implement this export as a worker that runs continuously in the background under it’s own supervision tree. As to how organize this - it really depends on the structure of the data and what the external API gives you. But my first approach would be a process that has a queue of messages, where each message represents a resource to be fetched. For each message, a task is spawned to fetch the data. If it fails, the message is resent to try again. Some additional process would also periodically request fetching of all resources.

you can do retries with tesla

https://hexdocs.pm/tesla/Tesla.Middleware.Retry.html

the missing data part, sounds domain specific so I think you would have to roll your own, maybe against a changeset, and if it’s invalid you load in the missing data from the api…

for the cron part you can use quantum GitHub - c-rack/quantum-elixir: Cron-like job scheduler for Elixir

I would consider making a mirror of the external system, i.e. dump the data (JSON, XML, whatnot) into your database as is; put that into a worker and handle external API failures there… Then add a separate worker that would transform and upload the data to the destination. This might make for a cleaner design and better performance.

The doesn’t seem to be an issue since the process is going to give you eventual consistency. The exported data is always stale (unless you can update the data in both systems as part of a single transaction or you can block updates in the old system), so it’s best to embrace this fact. You could make the staleness window smaller if you’d have notification from the old API + fast response times + no API quota, but that’s not the case.

I’m happy to be of any help. Also I’m into MCU lately, so I’ll keep an eye on your project