mkalygin

mkalygin

Graceful Restart of an Elixir v1.9 Release

Hello!

I’m working on a Phoenix app, but my question is more related to Elixir v1.9 releases. I’ve setup Ansible playbook which builds a release using mix release and uploads it to a production server. So at the time of a deployment, there are actually two releases on the server:

  1. A current one which is up and running.
  2. A new one, which has been deployed with Ansible and is not running yet.

What I do is finding a currently running release and using stop command on it. But I feel like there should be a better way to gracefully “replace” an old release with a new one, so that there is a little downtime during restart. For example, in my case I need to run migrations for the new release, which may take a while.

Do you have any ideas on how to handle it? Thanks!

Marked As Solved

danschultzer

danschultzer

Pow Core Team

Yeah it can be handled on the same server, and nginx will be your load balancer. It’ll be pretty close to zero downtime, though a cluster of individual servers would be more resilient, and probably simpler to work with (but you’ll have three servers instead).

A very simple version can be found here, which should be easy to understand: https://medium.com/@miket969/blue-green-deployments-with-nginx-cbaa9938bcf8

The above example isn’t really zero downtime, since it just cuts off all requests completely from the first server. If a request hasn’t been completed, it’ll fail, but the switch will be instantaneous. A better version is one where nginx does a graceful cut-over and wait for all traffic to have been redirected to the new server.

This is where it can get complex. HAProxy/Nomad or other tools are probably a better choice here.

If you are going down this path, these docs on Nomad and HAProxy may help you:
https://danielparker.me/haproxy/blue-green/deployments/canary/nomad/simple-blue-green-haproxy/
https://www.nomadproject.io/guides/operating-a-job/update-strategies/blue-green-and-canary-deployments.html

Also Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

You’ve got two options: Hot code load, or rolling deploy. In the hot code loading method you use one of Erlang’s unique features whereby you can simply replace the code that was running with new code, and there are some hooks for how to migrate data. The challenge with this approach is that not only do you need to think about migrating database data, any data you have in memory needs to be migrated as well, cause it’s straight up replacing old code with new code without stopping. Notably you need to use distillery releases for this, mix releases don’t support hot code loading.

Rolling deploy is more traditional, and it’s how you solve this in any other language too. In this method need more than 1 server running, and a load balancer that directs traffic to your nodes. You tell the load balancer to stop sending traffic to node A, then you have node A stop the old release, then start the new release. Then you direct traffic back to A, and move on to the next node. Any database migrations need to be compatible with both versions of your code since you’ll have a period of time while both versions are running.

The actual mechanics of how this is accomplished depend on how you’ve got your servers setup and how you have your load balancer(s) setup. There are also variations where you start brand new servers (or docker containers) that run the new version, migrate traffic over, and then delete the old servers (or docker containers) running the old versions.

danschultzer

danschultzer

Pow Core Team

You can’t do out of the box hot code upgrades with 1.9 releases, so your best option is to handle it with a load balancer. One way to do this is to make a blue-green deployment setup. It’s similar to what @benwilson512 says, you start up node b, when it’s running then switch your load balancer (e.g. nginx) to direct traffic to node b instead of a, and then shut down node a when all traffic has been redirected. This all happens on the same server.

You will have to set up you ansible playbook so it’s aware what port/socket each node runs on, and can switch between them during deploy. I’ve had plans for a while to expand on my Ansible guide with blue-green deployment, but have had no time to dig into it.

Where Next?

Popular in Questions Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
logicmason
Hi there, I'm working through my first release with elixir/phoenix. I've built a release with distillery and found that it crashes when I...
New
quazar
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
idi527
I’ve been re-reading swift book again and noticed that multiline strings there don’t have a trailing line break, unlike in elixir iex(2)...
New

Other popular topics Top

yurko
Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for ...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement