Improve systemd configuration documentation

[mauromol]

I must say that Spring Boot documentation regarding application installation under Linux for sysvinit system is well made and I could get it up and running quite fast, so good job there!

Unfortunately I also have to say that the same is not valid for the systemd part. I'll try to describe my today experience to help you improve that.

Just a note: seeking the Internet (mainly StackOverflow and GitHub) the feeling is that you don't want to go too deep with systemd integration because of little knowledge. This is understandable, however IMHO as far as I know most of (not-so) recent versions the main Linux distributions are using systemd now (Debian, Ubuntu, CentOS, Fedore, openSUSE, RHEL...) so if I had to chose where to put more effort in, I would opt for systemd today.

This said...

The documentation at:
https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#deployment.installing.nix-services.script-customization
seems to suggest that the subsequent information applies to both sysvinit and systemd cases. In particular, there's no mention to the fact that the config file is completely ignored in the systemd case, unless the service script references it.

With systemd, Environment= lines should be added to the [Service] section of the service script to set up environment variables. Multiple lines could be given, for instance:

[Unit]
Description=myapp
After=syslog.target

[Service]
User=myappuser
ExecStart=/var/myapp/myapp.jar
SuccessExitStatus=143
Environment=JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
Environment=JAVA_OPTS=-Xmx128m

[Install]
WantedBy=multi-user.target

However, since I like to have myapp.conf along with my application, because it allows me to see and change environment variables right there, without having to mess with the systemd service file, I added an EnvironmentFile directive to my myapp.service:

[Unit]
Description=myapp
After=syslog.target

[Service]
User=myappuser
ExecStart=/var/myapp/myapp.jar
SuccessExitStatus=143
EnvironmentFile=/var/myapp/myapp.conf

[Install]
WantedBy=multi-user.target

It would be useful to add a note, though, that myapp.conf must then be a pure environment file, not a script, so things like the following, which are commonly used by us and do work in the sysvinit case:

JAVA_OPTS=-Xmx64m

# add JMX options
JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999"

do not work here, because $JAVA_OPTS is not expanded in the second assignment. If this is needed, then the above must be changed to:

[Unit]
Description=myapp
After=syslog.target

[Service]
User=myappuser
ExecStart=/bin/bash -ac '. /var/myapp/myapp.conf; exec /var/myapp/myapp.jar'
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

In this case, though, the guidelines for securing the config file mentioned in the docs (with no mention to the fact that we're still talking about sysvinit only) do not apply, because the config file must be readable by the myappuser user, so this should be taken into account. Instead, using EnvironmentFile seems to allow that file to be readable by root only, so it might be better from a security point of view. A much less powerful way to split environment variable declarations in this case could be the one mentioned here, which consists of terminating each line with \ and continuing on the next line, but of course it's not equally flexible and may lead to errors when you want to comment/uncomment some lines.

Please note that the same limitation of EnvironmentFile about variable expansion also applies to variable defined as Environment entries directly in the service script.

Last but not least, logging. There's a timid note in the documentation, just before this link, that says that LOG_FOLDER, and LOG_FILENAME are ignored in the systemd case, even if you get to properly set up the environment like the above. That note points the user to https://www.freedesktop.org/software/systemd/man/systemd.service.html, where however I was not able to locate any mention to how to properly redirect logs.

First of all, the documentation may say that, by default, logs with systemd are redirected to the journal, so issuing journald -u myapp would allow to see them. This said, the page at https://www.freedesktop.org/software/systemd/man/systemd.exec.html is more useful. I then found out that the best way to redirect both the standard output and the standard error to a log file is this:

• with systemd >= 240

[Unit]
Description=myapp
After=syslog.target

[Service]
User=myappuser
ExecStart=/var/myapp/myapp.jar
SuccessExitStatus=143
StandardOutput=append:/var/myapp/logs/myapp.log
StandardError=inherit

[Install]
WantedBy=multi-user.target

• with systemd < 240 (Ubuntu 18.04, for instance), "append:" is not supported, so the best I could find is:

[Unit]
Description=myapp
After=syslog.target

[Service]
User=myappuser
ExecStart=/bin/sh -c 'exec /var/myapp/myapp.jar >>/var/myapp/logs/myapp.log 2>&1'
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

Of course it should be straightforward to change the above to redirect standard out and err to two distinct files.

/bin/sh may be /bin/bash, as well, so the above solutions to include the config file and redirect the log can be combined into:

ExecStart=/bin/bash -ac '. /var/myapp/myapp.conf; exec /var/myapp/myapp.jar >>/var/myapp/logs/myapp.log 2>&1'

I know this is a lot of information, perhaps also a bit out-of-scope. However, unless Spring Boot provides some way to improve the user experience with systemd out-of-the-box, I believe that the documentation should at least give the minimal information required to set up these basic configuration aspects: environment setup and logging.

Hope this is helpful.

[wilkinsona]

Thanks very much for this, @mauromol. There looks to be a lot of valuable information here. We've been aware that the systemd-related documentation needs to be improved for a while (#17052) but the core team lacks the necessary expertise to do so. Hopefully the above will give us a big enough push in the right direction that we're close to being in a position to tackle it.

[wilkinsona]

In this case, though, the guidelines for securing the config file mentioned in the docs (with no mention to the fact that we're still talking about sysvinit only)

This is interesting feedback. Thank you.

The section to which you've linked is nested beneath Installation as an init.d Service (System V). That nesting was intended to make it clear that it's specific to sysvinit. However, that obviously hasn't worked for you. The nesting isn't very visually apparent as there are no heading numbers to indicate the nesting and the font size is very similar (1.4em and 1.3em).

I think things are better in the multi-page documentation as the headings aren't so heavily nested so the distinction between them is more apparent.

[mauromol]

The section to which you've linked is nested beneath Installation as an init.d Service (System V). That nesting was intended to make it clear that it's specific to sysvinit.

Yes, that section is nested, but it's referenced from the section named "Customizing a Script When It Runs” which is not nested under the sysvinit one and talks about the config file and how to secure it, in a way that is not suggesting that is just a "sysvinit" thing. I hope I could explain myself.

With regards to #17052, though, I must admit that I did not need to take the approach described there: using the simple skeleton you provide in the documentation was enough for me and made my Spring Boot application (which consists of a bootJar artifact for an application running Tomcat) running fine as a service, except for the environment and logging issues I described here.

[mhalbritter]

In particular, there's no mention to the fact that the config file is completely ignored in the systemd case, unless the service script references it.

This seems not (no longer?) to be true with Boot 3.x. I just tested it with a systemd service file:

[Unit]
Description=spring-app
After=syslog.target

[Service]
User=moe
ExecStart=/home/moe/Downloads/systemd-demo/demo-0.0.1-SNAPSHOT.jar
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

a file /home/moe/Downloads/systemd-demo/demo-0.0.1-SNAPSHOT.conf is used. If i put

JAVA_OPTS=invalid

in there, the systemd service is no longer able to start.

[vpavic]

You don't really need to build an executable JAR in order to run Spring Boot based application as a systemd service, and actually you're IMO better off if you don't. Systemd itself has all the necessary tools to cover everything needed in service unit file.

Additionally, since systemd is dominant init system these days, the documentation should IMO cover it first (ahead of SysV init). I also think it would be worth considering to deprecate the SysV init support as it's quite archaic (and that in turn would remove the need to support building executable JARs).

Anyway, I had planned to open a PR proposing some changes around the systemd related documentation because lately I've been working with several Spring Boot applications deployed as systemd services and the current state of documentation is not optimal (and I've also contributed to that in the past).

[mhalbritter]

I agree that you're better of not using the executable JAR and let systemd handle aspects like users, state tracking and logging.

I'm not in favor of removing the SysInitV support, as there are distributions which don't want systemd and that's a very hairy debate. As long as the SysInitV support works and doesn't cause too much maintenance, it would not drop it.

[vpavic]

It might seem like a silly question, but can you name a distribution with significant market share that doesn't use systemd by default these days? Ubuntu, Debian, RHEL, CentOS, Fedora, CoreOS and SUSE have all been on systemd since at least 2015 AFAIK.

If you combine that with the fact that systemd support should only cost you a section in the reference manual, while SysV init support does have a noticeable footprint, I'd say that's a compelling reason to consider deprecation.

[mauromol]

In particular, there's no mention to the fact that the config file is completely ignored in the systemd case, unless the service script references it.

This seems not (no longer?) to be true with Boot 3.x. I just tested it with a systemd service file:

Honestly I don't know, I've not tested this with recent Spring Boot. What I described above is what I ended up with Spring Boot 2.4.

Additionally, since systemd is dominant init system these days, the documentation should IMO cover it first (ahead of SysV init).

I perfectly agree with this, as I already said in my original report.

can you name a distribution with significant market share that doesn't use systemd by default these days

Perhaps the most famous ones are Devuan, MX Linux and Slackware, although I don't know how much they're used in server environments in the real world.