I used mix in systemd to launch phx apps for a couple years. But it’s time to replace the simple service with a forked daemon. Here is where I’ll save my notes so when I have to do it again in 3 months and I google “elixir systemd service” I’ll find them again. Oh, and I guess if it helps someone else out, oh well. That’s the price I pay to be lazy.

Server: Ubuntu 20.04.5 LTS
Version: elixir 1.14.3 (compiled with Erlang/OTP 25) - with asdf
username: core (replace as you need)
project name: court (or court_api, replace as you need)

Build the release version in /opt/court_api

MIX_ENV=prod mix compile
MIX_ENV=prod mix assets.deploy
mix phx.gen.release
mix release

I like to load my entire environment from file rather than set each value in systemd’s config. These settings are all basic stuff except the PHX_SERVER=true is needed to tell the elixir daemon to launch the phoenix server. Also since I’m using asdf I include it’s shims and bin folder in the PATH.

vi /etc/environment-court

PATH="/home/core/.asdf/shims:/home/core/.asdf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
SECRET_KEY_BASE=RlfalO4gogetyourownkeyzdNsLE
MIX_ENV=prod
DATABASE_URL=ecto://mydbuser:mypassword@192.168.1.257/court
PORT=5428
PHX_SERVER=true

Now setup the systemd service. Note I’m running it as daemon and not start. This forks it in the background or something like that. The takeaway is it is now running in the background. (If I was in Bash, I would get my prompt back)

vi /lib/systemd/system/court.service

[Unit]
Description=Court
After=network.target
Requires=network.target

[Service]
Type=forking
WorkingDirectory=/opt/court_api
User=core
Group=core
Restart=always
RestartSec=5
EnvironmentFile=/etc/environment-court
ExecStart=/opt/court_api/_build/prod/rel/court_api/bin/court_api daemon
ExecStop=/opt/court_api/_build/prod/rel/court_api/bin/court_api stop

[Install]
WantedBy=multi-user.target

Now restart systemd and start the new court service. I use restart because lets be honest, I’ll typo something or want to add RuntimeMaxSec.

systemctl daemon-reload
systemctl restart court

NOTES - Logs
Bad news is now that phx is running in the background, stdout isn’t caught by systemd. TODO: Look for or add a phx setting that will route stdout to syslog. Until that day, find your logs here:

tail -f /opt/court_api/_build/prod/rel/court_api/tmp/log/erlang.log.1

Not sure if elixir will rotate that or not. If not then something like this (not tested)

vi /etc/logrotate.d/elixir-court

/opt/court_api/_build/prod/rel/court_api/tmp/log/*log* {
  daily
  missingok
  rotate 14
  compress
  delaycompress
  notifempty
  create 640 core core
  sharedscripts  
}

NOTES - console
Losing journalctl logs sucks the big one, but this makes up for it! The ability to jump into the console and break things directly!

/opt/court_api/_build/prod/rel/court_api/bin/court_api remote

I hope I find these notes useful in 3 months when I have to do this again. (I hope someone replies back with the news that elixir/phx now detects if it has been launched in daemon mode and redirects stdout to syslogd)

I’m just saving some more notes. While my notes above work for full production systems, the following is easier and works great for some internal microservices. Rather than build it, just run the mix phx.server. This way the logs are sent to systemd so you can follow them with: journalctl -u example -f

I also figured out how to connect to the iex CLI after the process is started with systemd. Warning: I can do this safely because my services are running behind a firewall. I’d read up on the --name “example@127.0.0.1” part of iex/elixir. You are exposing a management port/connection which is super helpful and a little more dangerous if someone naughty has access to that port. Enough talk, here is how I set it up:

sudo vi /etc/environment-example

PATH="/home/core/.asdf/shims:/home/core/.asdf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
SECRET_KEY_BASE=RlfalO4gogetyourownkeyzdNsLE
MIX_ENV=prod
DATABASE_URL=ecto://mydbuser:mypassword@192.168.1.257/example
PORT=5428

sudo vi /lib/systemd/system/example.service

[Unit]
Description=Example
After=network.target auditd.service

[Service]
EnvironmentFile=/etc/environment-example
User=core
Group=core
Type=simple
WorkingDirectory=/opt/example
ExecStart=/home/core/.asdf/shims/elixir --name "example@127.0.0.1" -S mix  phx.server
KillMode=process
TimeoutSec=0
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Here is how to connect to the iex/CLI of the running service:

mkdir ~/bin
vi ~/bin/example.sh

#!/bin/sh
exec iex --name console-example@127.0.0.1 --remsh example@127.0.0.1

chmod +x example.sh
~/bin/example.sh

@kif that’s an interesting issue, I have checked a production app that runs this way and can confirm that I have stack trace, the error is one that is raised manually with raise and not one that happened “on itself”.

We use exception tracking via AppSignal so I must have missed the issue with consistency between real exceptions and what we get in logs.

I also searched for a more “natural” error and have not found one, though I did find ActionClauseError of Phoenix which was reported properly in the monitoring (stack trace included), but in the log I have only seen the “Sent 400 in 6ms”, so that might be your case: Phoenix handles the error in some fixed way which makes it normal case and not an exception which loses the error info for the resulting output.

I would assume you have the same situation, you can test it by adding a manual raise call and see what you get.

In any case I am pretty sure the issue has nothing to do with OS but rather with application environment and configuration.

Have you tried using GitHub - hauleth/erlang-systemd: systemd utilities for Erlang applications by @hauleth to help with the logs?

I’m using releases, systemd and a service type of ‘forking’. I haven’t tested restarting on crash yet. (What would be a good way of doing that?)

My unit file:

[Unit]
Description=My App
After=network.target
Requires=network.target

[Service]
Type=forking
WorkingDirectory=/home/appuser/app
User=appuser
Group=appuser
Restart=on-failure
RestartSec=5
EnvironmentFile=/home/appuser/env_vars
ExecStart=/bin/bash -c 'PATH=/home/appuser/.asdf/shims:$PATH exec /home/appuser/app/bin/my_app start'
ExecStop=/bin/bash -c 'PATH=/home/appuser/.asdf/shims:$PATH exec /home/appuser/app/bin/my_app stop'

[Install]
WantedBy=multi-user.target

(I’ve installed erlang on the server via asdf so that my releases don’t have to include_erts, and found that I needed to add its shims to PATH.)

Is this “child killing business” why systemd reports a failed state on stop? Is there any way round that?

Relevant commit and discussion at Distillery: