SystemdUnitFiles

Unit Files

Here are the unit files for Apache and Nginx on CentOS 7:

# /usr/lib/systemd/system/httpd.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target
Documentation=man:httpd(8)
Documentation=man:apachectl(8)

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/httpd
ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
ExecStop=/bin/kill -WINCH ${MAINPID}
# We want systemd to give httpd some time to finish gracefully, but still want
# it to kill httpd after TimeoutStopSec if something went wrong during the
# graceful stop. Normally, Systemd sends SIGTERM signal right after the
# ExecStop, which would kill httpd. We are sending useless SIGCONT here to give
# httpd time to finish.
KillSignal=SIGCONT
PrivateTmp=true

[Install]
WantedBy=multi-user.target

# /usr/lib/systemd/system/nginx.service
[Unit]
Description=The nginx HTTP and reverse proxy server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
# Nginx will fail to start if /run/nginx.pid already exists but has the wrong
# SELinux context. This might happen when running `nginx -t` from the cmdline.
# https://bugzilla.redhat.com/show_bug.cgi?id=1268621
ExecStartPre=/usr/bin/rm -f /run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
KillSignal=SIGQUIT
TimeoutStopSec=5
KillMode=process
PrivateTmp=true

[Install]
WantedBy=multi-user.target

We use nginx as a reverse proxy and apache for mod_perl/middleware. I'm uncontainerizing our mod_perl services so I need to switch from docker unit files to directly invoking httpd. It's pretty interesting how different these two files are, right down to the descriptions, even though they are one-for-one technologies, that is, you can use httpd as reverse proxy just as well as nginx. Both can fork and detach. The systemd files are oddly very different.

The biggest question above is type=simple, notify vs forking. Here's a nice article that explains why you want to avoid forking: I htink httpd almost got it right, but it probably should be "simple" unless mod_systemd is loaded, which manages sd_notify. mod_systemd was introduced in httpd 2.5, and CentOS 7 uses httpd 2.4, and hasn't back ported mod_systemd afaict. However, the article fails to mention PIDFile, which nginx correctly uses.

Aside. Why would I have bothered looking at this? We use httpd as middleware on a multi-tenant (app) system so there are multiple instances of httpd running so we have to generate systemd unit files for each app.

This is an area where systemd unit files are not as good as service scripts. We have a single service script that starts all our services through parameterization. You can't do that with unit files, which instead need to be generated. As you can see, both the nginx and httpd unit files are "implicitly coupled" to a lot of behavior about how these services start and stop. For example, nginx -t is a good ExecStartPre value, and httpd should have this, too, and it sort of does in that it uses graceful for Reload which does -t implicitly. It's interesting that nginx doesn't do nginx -s reload, which is the correct, explicitly coupled way to reload nginx, because ExectStartPre is not called on reload.

You may say this doesn't matter, but it turns out it does as this comment mentions (but didn't get as many votes, unfortunately). There's a bug in the sysv init scripts, just like there's a bug in the systemd unit file. You have to use reload, because it runs configtest and then sends the signal. If you just send the signal, nginx will ignore invalid changes, which led to some confusion the post. Turns out httpd has the same behavior (surprise!) so the httpd unit's ExecReload is correct.

Back to the main theme: type=simple. Neither nginx nor httpd are tightly integrated with systemd in CentOS 7 so they should definitely not be notify. Forking is dangerous, because if the port is in use (say you started httpd accidentally), nginx will die after the fork unless you set PIDFile, which nginx correctly does. Don't do this with dockerized services, however, because the PID will be 1.

Note that Systemd requires PIDFile to be publicly readable so put them in /run.

I had mentioned that I don't want to learn systemd unit file syntax, and the careful reader will notice that nginx references $MAINPID and httpd specifies ${MAINPID}. I'd argue $MAINPID is better, because $MAINPID does not contain space. I like the explanation in this post. The example in the man page with echo's is really confusing. Exercise for the reader, what does this output?

ExecStart=/bin/echo "$ONE $TWO $THREE"

Hint: nothing. :)

This is what I've ended up for our app units:

[Unit]
Description={{ app }}
After=network.target remote-fs.target nss-lookup.target

[Service]
# Tested on CentOS 7, and it does have the localtime stat problem
# https://blog.packagecloud.io/eng/2017/02/21/set-environment-variable-save-thousands-of-system-calls/
Environment=TZ=:/etc/localtime
Type=forking
PIDFile=/run/{{ app }}/httpd.pid
ExecStart=/srv/{{ app }}/start
ExecReload=/srv/{{ app }}/reload
ExecStop=/srv/{{ app }}/stop
# Creates directory in /run owned by User for PIDFile
RuntimeDirectory={{ app }}
SyslogIdentifier={{ app }}
KillMode=process
# SIGCONT is ignored by httpd to allow ExecStop to do its job.
KillSignal=SIGCONT
# Second signal is SIGKILL, which kills all processes in the group.
TimeoutStopSec=5

[Install]
WantedBy=multi-user.target

Timers

Timers consist of a service and the timer itself. The timer unit file cannot have many things, e.g. SyslogIdentifier.