You could have a plug that checks for a given application key and returns true if you should return 503.

Then you could have a function in your application, let’s say in the MyApp module that does this:

def slow_shutdown do
  # This is the value that the plug will check to know if it should return 503
  Application.put_env(:my_app, :shutting_down, true)
  # Let's wait a second before finally shutting down
  Process.sleep(60_000)
  # Finally turn everything off
  :init.stop
end

Now you can use the same tips I used above and instead of echo ":init.stop" > to_erl, you can do echo "MyApp.slow_shutdown" > to_erl. If you are using distillery, it is likely distillery ships with an option to run a module+function in the currently running node.

thank you very much for your tip.

we will now see, that/how we can execute the code in the current node. that should do it then.

we have a very similar but slightly different use case.

we want to deliver a 503 on a health endpoint for some seconds on a sigterm, before the server is shutting down. we are using distillery for a phoenix micro service and the service is running in docker.

the reason for the 503 on the health endpoint is, so that the HAProxy in front of the service is recognizing that the service is going down before it is really down. so the service wont get any more request from the haproxy.

has anyone solved a similar situation or has any suggestions, how to solve this?

Shouldn’t you rather tell HAProxy to disable any requests to that proxy directly after the new service is up without having the 503 indirection?

thanks for your answer, but that’s not how it works here with mesos and marathon.

marathon is only reporting the status quo to the HAProxy and not what he intends to do (e.g. shutting down services).

when marathon is shutting down a service, the service must therefore report a 503 via a health endpoint, so that the HA Proxy will remove the service before it is really shut down.

we are looking in the moment into distillery, how we can hook us into the sigterm trap, but we thought that maybe there are already some best practices for this usecase, so we do not have to reinvent the wheel.