Permissions and Service User

“It works when I run it as root” is how many monitoring stacks are born — and, quietly, how they stay. Months later the config file is world-readable with a remote-write password in it, the data directory has root-owned files from that one manual test, and nobody dares touch any of it because the current arrangement is the only one anyone has seen work. Permissions debt compounds like any other debt, and the interest is charged during incidents.

This lesson builds the model properly: one unprivileged account, two directory trees with deliberate ownership, no capabilities, and a umask that makes new files private by default. It is an afternoon of work that you do once per fleet.

The account

Prometheus runs as a dedicated system user: no password, no home directory, no login shell.

# CONFIGURATION -- one account per fleet, same flags on every distro
sudo useradd --system --no-create-home \
  --home-dir /var/lib/prometheus \
  --shell /usr/sbin/nologin \
  prometheus

Notes:

• --system allocates the uid from the system range (below 1000 on all three target distros), which keeps the account out of login prompts and user-listing tools.
• --shell /usr/sbin/nologin blocks interactive login. On RHEL 9 the traditional path /sbin/nologin is a symlink to the same binary; either works. The shell only gates interactive sessions — sudo -u prometheus ... still works, which is how you will debug.
• --home-dir /var/lib/prometheus records the data directory as home without creating it; some tooling inspects the home field.
• Use your configuration management to pin the same numeric uid on every host. Restores and shared storage across machines care about the number, not the name.

Ownership and modes

Two trees, two different owners, on purpose:

/etc/prometheus/                 root:prometheus   0750
/etc/prometheus/prometheus.yml   root:prometheus   0640
/etc/prometheus/rules/           root:prometheus   0750
/etc/prometheus/rules/*.yml      root:prometheus   0640

/var/lib/prometheus/             prometheus:prometheus  0750
(everything the TSDB creates     inherits the service
 below it)                       user and its umask)

/usr/local/bin/prometheus        root:root         0755
/usr/local/bin/promtool          root:root         0755

The reasoning, tree by tree:

• Config owned by root, group prometheus, mode 0640. The service reads it (group membership); only root changes it (configuration management runs as root); no other local user reads it — and the file routinely holds basic_auth passwords and remote-write tokens. World-readable config is a credential leak, full stop.
• Data owned by the service, mode 0750. Prometheus must create, mmap and delete files here constantly. Nobody else needs anything.
• Binaries owned by root, writable by no one. The service user must not be able to replace its own binary — that is the difference between a compromised Prometheus and a persistent one.

# CONFIGURATION -- applying the model
sudo install -d -o root -g prometheus -m 0750 /etc/prometheus /etc/prometheus/rules
sudo chown root:prometheus /etc/prometheus/prometheus.yml
sudo chmod 0640 /etc/prometheus/prometheus.yml
sudo install -d -o prometheus -g prometheus -m 0750 /var/lib/prometheus

Why not root

Three concrete wins, not doctrine:

1. Blast radius. A remote-code-execution flaw in Prometheus (or in a dependency) lands the attacker in an account that owns one directory, instead of a root shell on the monitoring host that sees every target.
2. Binary immutability. With /usr/local/bin/prometheus owned by root and the process running as prometheus, the running process cannot modify its own executable. Persistence through binary replacement is off the table; combined with ProtectSystem=strict (lesson 03), even the directories are read-only to it.
3. Auditability. Every file the daemon writes is owned by prometheus; anything else in the data directory is a anomaly worth a question.

The cost is the failure modes below — all of which are self-inflicted and all of which have a log line that names the fix.

Capabilities: none

A capability check is the quickest way to see how little privilege Prometheus wants: it binds port 9090 — an unprivileged port, above 1023 — opens no raw sockets, loads no kernel modules, touches no devices. The server needs zero Linux capabilities. So give it zero:

# in the [Service] section, alongside the lesson-03 sandboxing
NoNewPrivileges=true
CapabilityBoundingSet=
AmbientCapabilities=
UMask=0077

An empty CapabilityBoundingSet means even a future setuid binary or a file-capability attack gains nothing; NoNewPrivileges=true closes the same door from the other side. Contrast with node_exporter, which is a different story: several collectors read root-only paths under /proc and /sys, and some deployments run it as root or with selected capabilities for exactly that reason. The exporter lesson settles that account per collector — do not import its privilege into the server.

UMask=0077 deserves its own sentence: Prometheus creates files mode 0666 and directories 0777, masked down by the umask. With 0077, every WAL segment, block and checkpoint it ever writes lands at 0600/0700 — readable by the service alone — with no per-file work from you. Use 0027 instead if your team deliberately grants the prometheus group read access to the data tree for forensics.

How to validate it

# READ-ONLY -- the account exists, is a system account, cannot log in
id prometheus
# uid=342(prometheus) gid=342(prometheus) groups=342(prometheus)
getent passwd prometheus
# prometheus:x:342:342::/var/lib/prometheus:/usr/sbin/nologin

# READ-ONLY -- the service can read its config and write its data
sudo -u prometheus test -r /etc/prometheus/prometheus.yml && echo config-readable
sudo -u prometheus touch /var/lib/prometheus/.wtest && rm /var/lib/prometheus/.wtest

# READ-ONLY -- every link in the data path is sane
namei -l /var/lib/prometheus/wal

# READ-ONLY -- the identity and sandbox systemd actually applied
systemctl show prometheus -p User -p Group -p UMask -p NoNewPrivileges \
  -p CapabilityBoundingSet -p ProtectSystem -p ReadWritePaths

# READ-ONLY -- files the daemon creates match the umask prediction
ls -l /var/lib/prometheus/wal | head -3
# -rw------- 1 prometheus prometheus 134217728 Aug 13 10:02 00000042

# READ-ONLY -- score the unit
systemd-analyze security prometheus --no-pager

The wal listing is the quiet proof that the umask did its job: files born 0600, owned by the service, without a single chmod from you.

How it can fail

1. Root-owned data directory. Someone created /var/lib/prometheus as root, or a restore recreated it. Startup dies with opening storage failed: ... permission denied on the lock file. Fix: chown -R prometheus:prometheus.
2. Config unreadable by the service. 0600 root:root on prometheus.yml looks tidy and breaks the group-read model: error loading config ... permission denied. Either the file joins group prometheus at 0640, or nothing reads it.
3. Numeric uid drift after a restore. A tar backup restored on a host where prometheus is uid 341 instead of 342 leaves files owned by a stranger; same startup symptom as mode 1. Pin the uid in configuration management; restore with --numeric-owner awareness.
4. World-readable config. 0644 on prometheus.yml leaks the inline basic_auth password to every local account. Silent — nothing fails — which is what makes it the worst mode on this list. Audit with find /etc/prometheus -perm -o=r.
5. “Temporary” root runs. A hand-debugged prometheus run as root leaves root-owned WAL segments; the next non-root start fails on exactly those files. Worked as root, fails as prometheus — the log names the path.
6. Over-tightening. ProtectHome=true plus TLS certificates under /home/..., or ReadWritePaths missing a directory the config references. The fix is a targeted exception (ReadOnlyPaths), never removal of the directive.

How to troubleshoot it

Permission failures are polite: they name the path in the log. Exploit that.

1. Read the line. journalctl -u prometheus -e; the error contains the exact path it could not open.
2. Walk the path. namei -l <path> prints owner and mode of every component — one wrong directory in the chain is the classic find.
3. Compare identities. id prometheus against the ownership from step 2; on RHEL 9 add ls -lZ and ausearch -m avc -ts recent for SELinux denials that modes cannot explain.
4. Check what systemd applied. systemctl show prometheus -p User -p ProtectSystem -p ReadWritePaths — a unit edited without daemon-reload applies the old sandbox.
5. Fix ownership, restart, verify. chown/chmod need no reload, but a service that already exited needs systemctl start; then re-run the validation block above.

Security implications

This lesson is the security control, so the summary is short: the filesystem permission model is the authentication boundary for everything Prometheus holds until the TLS and reverse-proxy lessons land. Keep credentials in password_file-style references or in the config at 0640 root:prometheus; keep the data tree at service-only modes; keep the capability set empty so that “what if Prometheus is compromised” has a boring answer. A world-writable config is worse than a world-readable one: any local user can add a scrape target or repoint remote_write, which is data exfiltration with extra steps.

Performance implications

The permission model costs nothing at runtime — ownership checks are in-kernel and free. The sandboxing directives add a one-time namespace setup at service start. The two measurable items are self-inflicted: SELinux AVC denials on a mislabelled path surface as latency noise and confusing errors, and a journald rate limit can drop the very log lines you need when a permission loop spams — keep journald Storage=persistent and know your rate limits.

Production guidance

• One account, one uid, fleet-wide, from configuration management. Names are for humans; restores and shared volumes honour numbers.
• Encode the ownership table above as a lint check in CI: config 0640 root:prometheus, data 0750 prometheus:prometheus, binaries 0755 root:root.
• Never chmod -R 777; never User=root “temporarily”. Temporary root runs leave root-owned files that break the next non-root start.
• Audit quarterly: getent passwd prometheus (shell still nologin?), find /etc/prometheus -perm -o=r (anything world-readable?), systemd-analyze security prometheus (score drifted?).
• Grant exceptions narrowly: ReadOnlyPaths for the one extra file, not the removal of ProtectSystem.

Verification

You should now be able to answer:

• Why does the Prometheus server need no Linux capabilities, and which two unit directives enforce that?
• Defend 0640 root:prometheus on prometheus.yml in one sentence.
• With UMask=0077, what modes do new files and directories in the data tree get, and why?
• The log shows permission denied on the WAL after a restore from another host. Name the two most likely causes.
• Why does running Prometheus as root “just to test” create a future failure?

Quiz