skip installation and use Debian packages instead

[anarcat]

So this role can install prometheus from a binary tarball on GitHub (which I feel is a questionable practice, but that's another issue). In our environments, we prefer packages vetted by third parties, namely for auto-upgrades and so on, so we rely on Debian packages.

It's possible to completely skip the install with prometheus_skip_install, but then the role will crash with:

[prometheus.debian.net]: FAILED! => {"changed": false, "checksum": "df2496d755394a2fd11e01e3b4bd566418adc796", "cmd": "/usr/local/bin/promtool check rules /home/anarcat/.ansible/tmp/ansible-tmp-1695839321.1357365-156317-255727393866530/source", "msg": "[Errno 2] No such file or directory: b'/usr/local/bin/promtool'", "rc": 2, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

That's because promtool is in /usr/bin/promtool and not /usr/local/bin/promtool as expected...

There's all sorts of issues like this around the package. For example, the prometheus.service unit file is overridden, poorly IMHO:

--- /usr/lib/systemd/system/prometheus.service	2023-02-04 00:20:19.000000000 +0000
+++ /etc/systemd/system/prometheus.service	2023-09-27 18:46:39.470561824 +0000
@@ -1,39 +1,53 @@
+#
+# Ansible managed
+#
+
 [Unit]
-Description=Monitoring system and time series database
-Documentation=https://prometheus.io/docs/introduction/overview/ man:prometheus(1)
-After=time-sync.target
+Description=Prometheus
+After=network-online.target
+Requires=local-fs.target
+After=local-fs.target
 
 [Service]
-Restart=on-failure
+Type=simple
+Environment="GOMAXPROCS=2"
 User=prometheus
-EnvironmentFile=/etc/default/prometheus
-ExecStart=/usr/bin/prometheus $ARGS
+Group=prometheus
 ExecReload=/bin/kill -HUP $MAINPID
-TimeoutStopSec=20s
-SendSIGKILL=no
+ExecStart=/usr/bin/prometheus \
+  --storage.tsdb.path=/var/lib/prometheus \
+  --storage.tsdb.retention.time=365d \
+  --storage.tsdb.retention.size=0 \
+  --web.config.file=/etc/prometheus/web.yml \
+  --web.console.libraries=/etc/prometheus/console_libraries \
+  --web.console.templates=/etc/prometheus/consoles \
+  --web.listen-address=0.0.0.0:9090 \
+  --web.external-url= \
+  --config.file=/etc/prometheus/prometheus.yml
 
-# systemd hardening-options
-AmbientCapabilities=
-CapabilityBoundingSet=
-DeviceAllow=/dev/null rw
-DevicePolicy=strict
-LimitMEMLOCK=0
-LimitNOFILE=32768
+CapabilityBoundingSet=CAP_SET_UID
+LimitNOFILE=65000
 LockPersonality=true
-MemoryDenyWriteExecute=true
 NoNewPrivileges=true
+MemoryDenyWriteExecute=true
 PrivateDevices=true
 PrivateTmp=true
+ProtectHome=true
+RemoveIPC=true
+RestrictSUIDSGID=true
+#SystemCallFilter=@signal @timer
+
+ReadWritePaths=/var/lib/prometheus
+
 PrivateUsers=true
 ProtectControlGroups=true
-ProtectHome=true
 ProtectKernelModules=true
 ProtectKernelTunables=true
-ProtectSystem=full
-RemoveIPC=true
-RestrictNamespaces=true
-RestrictRealtime=true
-SystemCallArchitectures=native
+ProtectSystem=strict
+
+
+SyslogIdentifier=prometheus
+Restart=always
 
 [Install]
 WantedBy=multi-user.target

now some of those are due to the role's entries not being sorted the same way (if at all, actually), but some are real issues. Restart=always (or, for that matter, even =on-failure) in particular, is dangerous, see https://bugs.debian.org/1022724

There should be a way to leave all of that stuff alone and focus on configuring the package instead...