So I have a similarish desire in regard for migrations: startup the application and apply any outstanding migrations as needed in the startup routines. Naturally, there’s a bit more to it than that and I’m not in an embedded scenario, but there are reasons for this approach I believe are valid for wanting my application itself to govern the running of database updates rather than have that as a separate administrative task. So my solution: don’t use Ecto migrations for this purpose or try to bend it to my will.

My viewpoint as a still relative outsider to the world of Elixir is this: Ecto as a tool is designed to meet the needs of a general audience, but a certain kind of general audience. If you’re building a sort of typical web-app and your product is delivered via that web-app, where the database is mostly simpler persistence for a specific application, where you’re doing rigorous continuous integration & (probably) continuous deployment, you don’t have database specialists involved in development (so called “application DBAs”), and you have full control over both the development of the application and the operations of the application: my assessment of Ecto is that it’s most at home in this kind of environment. We can probably extend this a bit if we allow just the operational control by qualified staffing, where the application is brought in from elsewhere, like you might find in libraries or certain utility apps that you might add into your operation. (I’m probably speaking more about Ecto SQL throughout this post moreso than Ecto, but I’ll just keep saying Ecto for now, you know what I mean).

But leave the model that Ecto best supports and it quickly makes less sense to use and trying to make Ecto be all things to all people I think will ultimately diminish the value it does provide to its intended audience.

Watching the forums here and talk around the Web, you’d think that database access and Ecto are necessary partners, but they aren’t. Ecto is a very helpful library that will get you through a lot of use cases. But when it doesn’t hit the mark, it’s probably best to just leave Ecto and its opinions on the shelf and find something else (OK, there really isn’t a lot to find) or make something bespoke to deal with your specific needs… the complexity and size of Ecto is probably because the way it is because it’s trying to be a generalized solution for these kinds of applications.

I looked at the specific problems I’m trying to solve and decided to roll my own migrator (just finishing up that project). I don’t need many of the features that Ecto brings with it and Ecto doesn’t support a number of features I think are important to this specific application, and it’s not really that much code to make things work: it just needs to support my model for database updates well and handle the failure modes that can come out of that model in a sufficiently predicable manner. So in this regard, Postgrex is much more important to my application than Ecto SQL. But the requirements I’ve set for my application are niche compared to the common use cases for Elixir and I wouldn’t suggest anyone follow my model unless they have clear reasons to do something similar. Of course, there’s the query builders and the DSL around building out the schema with Ecto… but I don’t need that either. Plain old SQL is completely sufficient and I’m probably better served since the database does more in my application than in many web-apps.

Anyway… I guess I’m just trying to say is extending the target use cases in a well established something like Ecto isn’t necessarily bad, but it’s worth asking if you should rather than if you can and that’s basically the messaging that I’m seeing in this thread: many voices not sold on extending Ecto in this direction and I think they’re probably right.

Thanks,
Steve

P.S. Perhaps Ironically, I am using Ecto proper for it’s changesets and validations in dealing with internal validations, form processing, etc. I could gin up something for that, too, but for my use case it actually works well so far. I may end up going elsewhere in the end, but for right now those parts of Ecto are superior to anything I could build myself and will likely be more than good enough.

That’s exactly the complexity I was hinting at in my last post. I don’t think I have the one solution here given all approaches come with their individual tradeoffs. The more “dynamic” or unknown (as opposed to static) a system is the more tricky it becomes to handle its state. In the end you’ll need some place to know of all the subcomponents you have. When using the individual Application.start callbacks then this place would be the startup list of applications in your release. Generally I’d suggest having some central component in your system knowing about subcomponents, because then you are in control – instead of delegating to a piece of code you don’t directly control. With that you could have something like a MyCentralApp.run_migrations delegate to a known module in each subcomponents, which handles their individual migration needs. Something like

for sub <- fetch_subcomponent_roots() do
  Module.concat(sub, Migration).run_migration()
end

I’d strongly suggest to let each application deal with it’s own migration needs, ot

That’s imo completely unrelated to migrations, but to anything your dependencies do. I want to start that this is generally not something the beam has a simple solution to. If an application crashes (and it’s a permanent one) then the vm will shut down and is expected to come back online by means outside the vm. Also restarts are the means of recovery on the beam, but (especially without backoff) they can only fix a subset of possible errors in a system. The goal of tools provided by the beam and otp is to keep all the stuff running, which you tell it to keep running and it’ll give up if it cannot do so.

So to build a system, where partial availability is more important than complete availability you’ll need to build in the guard rails. Those could be circuit breakers, starting applications (see e.g. the shoehorn library) or processes with non :permanent restart types, using more elaborate restart mechanisms with backoffs, custom logic (see e.g. the parent library). You’ll essentially want to be able to shut off things, which continue to fail (causing partial downtime) in favor of keeping the ability to connect to the system from the outside. Doesn’t matter if a migration or other code is the cause of the errors.

This becomes even more tricky when you considere NIFs, which when they fail directly bring down the beam VM. Nothing application level can catch errors in NIFs. Also there are outside forces, which might stop your beam VM. E.g. when using linux the oom killer might stop the beam if you’re using to much memory.

While as you can see there are a lot of things to cover, this is not to say you cannot deal with those. But you’ll need to be diligent and careful, know your dependencies, know your failure cases, … The beam has many tools, which help build a system, which has the properties you need. You won’t be getting them automatically though.

I think this didn’t get much traction because that’s a rather unusual approach to migrations. Depending on the environment/build env, the usual thing to do is:

Local development (:dev):
manually run

mix ecto.migrate

Local testing/CI (:test):
run migrations as part of the tests - the test task is aliased to

test: ["ecto.create --quiet", "ecto.migrate", "test"]

in mix.exs

Production (:prod):

• Package the app as a release,
• Add a release task,
• Run the migrations at startup.

If you’re using Docker, this could be:

CMD ["sh", "-c", "bin/app eval MyApp.Release.migrate && bin/app start"]

Usually there are healthchecks in place, so if the migrations fail, the deployment process should pick up on that and abort the deployment.

The case in the original post was about running the new version regardless if the migrations succeed or not, but I think this can be easily achieved with running the migrations as a separate command and just ignoring any errors.

I think this needs more attention. I decided to search the forum for other examples of this procedure but it appears nobody has really taken interest in this post or sharing other methods. I have been considering writing a library for this too but I’m short on time. I’m curious if you’d be interested in taking a look at the example I’ve put up here as it demonstrates the method I’ve been using which was created by Pleroma (https://pleroma.social)

It would be nice if there was a common library that everyone uses for this purpose.

If you follow what @stefanchrobot wrote then running code automatically when starting the application vs. “from the outside” as shown is just a matter of calling MyApp.Release.migrate() within your application startup logic like e.g. MyApp.Application.start instead of via bin/app eval. In the end the created module is just a module with functions like any other. You can call them in a way which fits your project.

The benefit to doing migrations ouf of band is that you can uncouple software deploys from migrations, which is usually done for derisking deployment steps.