systemd Integration

Without a unit file, Prometheus is a process in a terminal. The first logout kills it, the first reboot forgets it, and the first out-of-memory kill leaves it dead until someone notices. The system you rely on to notice failures has itself failed, silently, which is the worst shape of failure this course teaches.

The unit file is the fix, and it is more than a start script. It is the contract that records who Prometheus runs as, which flags it runs with, what happens when it dies, and which parts of the host it is allowed to touch. All four target distributions — Ubuntu 24.04, Debian 12, RHEL 9 — run systemd, so this one file is the whole integration.

What the unit buys you

A production unit answers five questions:

1. Identity — the dedicated prometheus user and group (lesson 06 covers the account itself).
2. Invocation — the exact flag set, in one reviewed place.
3. Lifecycle — start at boot, restart on failure, reload without a restart.
4. Resources — file-descriptor limits sized for the target count.
5. Containment — sandboxing directives that bound what a compromised process can do.

A production unit, line by line

# /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus monitoring system
Documentation=https://prometheus.io/docs/
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus

ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.enable-lifecycle

# systemctl reload prometheus sends SIGHUP: config + rules re-read,
# no process restart, no WAL replay.
ExecReload=/bin/kill -HUP $MAINPID

Restart=on-failure
RestartSec=5s

# Scrapes, remote-write connections and open TSDB files all consume
# descriptors; the 1024 default is too small past a few hundred targets.
LimitNOFILE=65536

# Sandboxing: the whole filesystem is read-only to the process...
ProtectSystem=strict
# ...except the one directory it must write.
ReadWritePaths=/var/lib/prometheus
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

The lines that earn their keep:

• After=network-online.target — Prometheus binds a listener and dials scrape targets; starting before the network is configured produces a confusing first crash on some hosts.
• $MAINPID — systemd substitutes the PID it is tracking, so the reload signal provably reaches the right process.
• Restart=on-failure — a crash comes back; a clean systemctl stop stays stopped. RestartSec=5s keeps a crash loop from hammering the CPU while still recovering quickly.
• ProtectSystem=strict mounts the entire filesystem read-only inside the service’s namespace; ReadWritePaths then re-opens exactly the data directory. If you forget it, the symptom is opening storage failed: ... read-only file system.

Reload vs restart

Two different mechanisms, two different costs:

• Reload — systemctl reload prometheus (SIGHUP via ExecReload), or curl -X POST http://localhost:9090/-/reload when --web.enable-lifecycle is set. Prometheus re-reads prometheus.yml and every matched rule file, then rebuilds its scrape pools. The process does not restart: no WAL replay, no readiness gap. Use this for config and rule changes.
• Restart — systemctl restart prometheus. The process exits and starts: WAL replay runs before /-/ready reports ready, and no scraping happens in between — seconds on a small install, minutes on a large one. Flags are fixed at exec time, so anything in ExecStart — retention, storage path, feature flags, listen address — needs this.

A reload cannot change a flag, and a restart is wasted on a rule edit. Most failed “my change did not apply” reports are that distinction, got wrong.

--web.enable-lifecycle also opens POST /-/quit, which stops the server. Anyone who can reach port 9090 can therefore stop your monitoring — bind to a restricted interface or put an authenticating proxy in front before enabling this on a shared network. The security part of the course covers the general problem.

How to validate it

# CONFIGURATION -- pick up a new or edited unit
sudo systemctl daemon-reload

# SERVICE-IMPACT -- enable at boot and start now
sudo systemctl enable --now prometheus

# READ-ONLY -- state, PID, recent log lines
systemctl status prometheus --no-pager
#  prometheus.service - Prometheus monitoring system
#    Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; preset: disabled)
#    Active: active (running) since Thu 2026-08-13 09:14:01 UTC; 2h 11min ago
#  Main PID: 812 (prometheus)

# READ-ONLY -- the properties systemd actually applied
systemctl show prometheus -p User -p LimitNOFILE -p ProtectSystem -p ReadWritePaths
# User=prometheus
# LimitNOFILE=65536
# ProtectSystem=strict
# ReadWritePaths=/var/lib/prometheus

# READ-ONLY -- the server is past WAL replay and serving
curl -s http://localhost:9090/-/ready
# Prometheus Server is Ready.

# READ-ONLY -- the reload path works and is logged
systemctl reload prometheus
journalctl -u prometheus -n 3 --no-pager
# ... level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
# ... level=info msg="Completed loading of configuration file" ...

A bad config on reload is not fatal: Prometheus logs the error and keeps running the old configuration. A bad config on restart is fatal — the process exits at load. That asymmetry is why promtool check config belongs in front of every restart and why reloads are the safer default for config changes.

How it can fail

1. Edited unit, forgotten daemon-reload. systemd keeps the old unit in memory and prints Warning: Unit file changed on disk. Your new flags are not running.
2. Typo in ExecStart. The process exits immediately; Restart=on-failure loops it until the start limit trips: Failed to start ... Unit ... has a start request repeated too quickly. The real error is one bad line, visible in the journal.
3. ReadWritePaths missing. With ProtectSystem=strict the data directory is read-only; startup dies with read-only file system.
4. LimitNOFILE too small. Past a few hundred targets or a busy remote-write queue: accept4: too many open files and flapping scrapes while the process itself looks healthy.
5. systemctl reload with no ExecReload. systemd answers Job type reload is not applicable for unit prometheus.service. The fallback is the lifecycle endpoint — or a restart.
6. POST to /-/reload without the flag. The lifecycle API is off by default; the endpoint returns HTTP 403. The fix is the flag plus a restart, because flags do not change on reload.

How to troubleshoot it

1. Unit state and last log lines together: systemctl status prometheus -l --no-pager. activating (auto-restart) means crash loop — the log excerpt right below it is the real error.
2. Full log, newest first: journalctl -u prometheus -e. Config errors name the file and line; storage errors name the path.
3. What systemd thinks it runs: systemctl cat prometheus (the parsed unit, drop-ins included) and systemctl show for applied properties. If these disagree with the file you edited, daemon-reload.
4. Reproduce by hand: sudo -u prometheus /usr/local/bin/prometheus --config.file=... --storage.tsdb.path=... in the foreground. Works by hand but not under systemd — the problem is the unit (sandboxing, limits, user), not Prometheus.
5. Score the sandbox: systemd-analyze security prometheus. When hardening breaks something, the log line names the blocked path; add a targeted ReadOnlyPaths rather than removing the directive.

Security implications

The unit is a security boundary, not just plumbing. User= keeps the daemon out of root (lesson 06); NoNewPrivileges=true closes the setuid escalation path; ProtectSystem=strict means even remote code execution inside Prometheus cannot replace /usr/local/bin/prometheus or drop a file into /etc; the runtime’s own tools cannot be turned into persistence. The cost is occasional self-inflicted wounds — TLS certificates under /home become unreadable with ProtectHome=true, for instance — and the fix is always a narrower exception, never removing the directive. And re-read the lifecycle note above: --web.enable-lifecycle exposes /‑/quit; treat that endpoint as administrative.

Performance implications

systemd itself adds no measurable overhead — the process runs natively; the cgroup merely accounts for it, which is free telemetry (systemd-cgtop). The knobs that matter are LimitNOFILE (size it to targets plus remote-write connections plus TSDB files, with margin), RestartSec (tight enough to recover, loose enough to avoid a CPU-burning crash loop), and journald’s own rate limits, which can silently drop log lines from a chatty service — keep Storage=persistent in journald.conf so logs survive reboots for post-mortems.

Production guidance

• The unit is code: version it, review it, deploy it with configuration management, daemon-reload in the same pipeline.
• Prefer drop-ins (/etc/systemd/system/prometheus.service.d/) for host-specific flags over forking the whole unit.
• Default to reload for config and rule changes; reserve restarts for flag and binary changes, and schedule them — the scrape gap is real.
• Alert on unit state from outside the host: a second Prometheus or a black-box check watching /-/healthy.
• After any change, run the validation block above. Two minutes, every time.

Verification

You should now be able to answer:

• What does systemctl reload prometheus do with the unit in this lesson, and what does it cost?
• Which changes require a restart rather than a reload, and why?
• What does ProtectSystem=strict block, and what undoes the one write Prometheus legitimately needs?
• Prometheus is in activating (auto-restart). What are your first two commands?
• Why does enabling --web.enable-lifecycle change your network exposure?

Quiz