Lab: Deploy Prometheus

Objective

By the end of this lab you will have a pinned Prometheus 2.55.x, installed from an archive whose checksum you checked, running as an unprivileged account it cannot escape, writing to a TSDB on its own filesystem, under a unit that scores better on systemd-analyze security after your drop-in than before it.

You will then produce, on purpose, the single most common self-inflicted failure of that hardening — a data directory that is read-only to the service — and diagnose it from the journal in two commands. Finally you will prove which kinds of change a reload applies and which it silently does not, by reading the running process rather than the file you edited.

Architecture

One host, one process, four locations. Nothing here is hidden from you, which is the point of the tarball install.

  /usr/local/bin/prometheus         root:root         0755   the server
  /usr/local/bin/promtool           root:root         0755   the validator

  /etc/prometheus/                  root:prometheus   0750
  /etc/prometheus/prometheus.yml    root:prometheus   0640   holds credentials
  /etc/prometheus/rules/*.yml       root:prometheus   0640

  /var/lib/prometheus/              prometheus:prometheus 0750
     |                              its own filesystem, mounted from a loop
     +-- wal/                        write-ahead log, 128 MiB segments
     +-- chunks_head/                memory-mapped head chunks
     +-- 01J.../                     immutable blocks, one per time span
     +-- lock                        flock: one process per data directory

  journald                          all logs; Prometheus writes no log file

  /etc/systemd/system/prometheus.service       the base unit
  /etc/systemd/system/prometheus.service.d/    drop-ins: hardening, retention

Two structural decisions worth naming before you start. The config is owned by root and readable by the service group, not by the service user — configuration management writes it, the daemon reads it, and nobody else on the host can read the remote-write token it will eventually hold. The data directory is a separate filesystem, so that a retention miscalculation fills a volume rather than filling / and taking sshd and journald down with it.

Requirements

A disposable Debian 12 or Ubuntu 24.04 host with sudo and outbound HTTPS to github.com. The lab creates a system account, writes into /etc and /usr/local/bin, and mounts a filesystem. Do not run it on a host you care about.
About 3 GB of free disk (2 GB of it becomes the TSDB volume) and 1 GB of RAM.
curl, jq and sha256sum. Task 1 records which were already present so Cleanup removes only what this lab added.
Port 9090 free. Task 1 checks.
No out-of-band access requirement. This lab does not touch SSH, the firewall, the primary interface, or /etc/fstab. See the note in Task 4 for why the last one is deliberate.

Scenario

The monitoring host for a new environment has to exist by Friday. The previous two were built differently: one is a distribution package two years behind the configuration the runbooks describe, and one is a binary somebody untarred into /opt and started in tmux, which did not survive its first reboot and was not noticed for eleven hours because the thing that would have noticed was the thing that was down.

This one is going to be the one whose install, hardening, validation and teardown are all written down, whose version is pinned in one place, and whose upgrade path is a symlink and a restart rather than an argument.

Tasks

Task 1: Capture the starting state

LAB="$HOME/prometheus-deploy-lab"
mkdir -p "$LAB"
cd "$LAB"

# Port 9090 must be free.
ss -ltnp 2>/dev/null | awk 'NR==1 || /:9090\s/' | tee ports.pre-lab

# Which tools were already here? Cleanup reads this file.
for p in curl jq; do
  printf '%s %s\n' "$p" "$(dpkg-query -W -f='${Status}' "$p" 2>/dev/null || echo notinstalled)"
done | tee packages.pre-lab

# The account must not already exist, and the mount point must be empty.
getent passwd prometheus || echo "no prometheus account - good"
findmnt /var/lib/prometheus || echo "nothing mounted at /var/lib/prometheus - good"

sudo apt-get update
sudo apt-get install -y curl jq ca-certificates

Task 2: Fetch the artefact and verify it

Pin the version in one shell variable and use it everywhere. The download is also the existence check: curl -f fails loudly on a 404, which is what you want if the version you pinned is not a version that exists.

cd "$LAB"
VER=2.55.1
BASE="https://github.com/prometheus/prometheus/releases/download/v${VER}"

curl -fsSLO "${BASE}/prometheus-${VER}.linux-amd64.tar.gz"
curl -fsSLO "${BASE}/sha256sums.txt"

grep "prometheus-${VER}.linux-amd64.tar.gz" sha256sums.txt | sha256sum -c -

That last line must print OK. Treat any other outcome as a stop condition, not a warning — the two things it detects are a truncated download and an archive that is not the one the release pipeline produced, and neither of them gets better if you continue.

What it does not prove is that sha256sums.txt itself is trustworthy. You fetched it over TLS from the project’s own release infrastructure, which is the control that covers that half; the checksum covers the archive.

tar xzf "prometheus-${VER}.linux-amd64.tar.gz"
ls "prometheus-${VER}.linux-amd64"

sudo install -o root -g root -m 0755 \
  "prometheus-${VER}.linux-amd64/prometheus" /usr/local/bin/prometheus
sudo install -o root -g root -m 0755 \
  "prometheus-${VER}.linux-amd64/promtool" /usr/local/bin/promtool

prometheus --version
promtool --version

Both must report the same version. They are cut from the same commit; a mismatch means one of them came from somewhere else, which is a state worth resolving before it validates a config for a server it does not match.

Task 3: Create the account and the config tree

The account is a system account with no password, no home directory it creates, and no login shell:

sudo useradd --system --no-create-home \
  --home-dir /var/lib/prometheus \
  --shell /usr/sbin/nologin \
  prometheus

id prometheus

--shell /usr/sbin/nologin blocks interactive login only. sudo -u prometheus still works, which is how you will debug in Task 8.

sudo install -d -o root -g prometheus -m 0750 /etc/prometheus
sudo install -d -o root -g prometheus -m 0750 /etc/prometheus/rules
sudo cp -a "$LAB/prometheus-${VER}.linux-amd64/consoles" /etc/prometheus/
sudo cp -a "$LAB/prometheus-${VER}.linux-amd64/console_libraries" /etc/prometheus/

# cp -a preserves the ownership the tarball unpacked with, which is your
# account. Put it back on the model: root owns everything under /etc.
sudo chown -R root:prometheus /etc/prometheus/consoles /etc/prometheus/console_libraries

Task 4: Give the TSDB its own filesystem

On a real monitoring host this is an LVM volume. On a lab VM with no spare disk, a file-backed loop device gives you the same boundary — a filesystem that fills independently of / — with nothing to lose if you get it wrong.

sudo install -d -m 0755 /opt/rb-lab
sudo fallocate -l 2G /opt/rb-lab/prometheus-tsdb.img
lsblk
ls -lh /opt/rb-lab/prometheus-tsdb.img

Destructivelab host— Writes an ext4 filesystem into the 2 GB file created above. Not a block device.

$ sudo mkfs.ext4 -q -L rb-prom-tsdb /opt/rb-lab/prometheus-tsdb.img

sudo install -d -o prometheus -g prometheus -m 0750 /var/lib/prometheus
sudo mount -o loop,noatime /opt/rb-lab/prometheus-tsdb.img /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
sudo chmod 0750 /var/lib/prometheus

findmnt /var/lib/prometheus
df -h /var/lib/prometheus

noatime removes a metadata write on every read of a block file, which for a directory the page cache reads constantly is free. The mount is deliberately not in /etc/fstab: production puts it there, but a lab that leaves a broken fstab entry on a host you forget to clean up costs you a boot, and there is nothing here worth that. Cleanup unmounts it.

What lives in the data directory, and why the mount matters

wal/ takes every scraped sample before anything else does; after an unclean stop, startup replays it to rebuild the in-memory head, which is why a large instance takes minutes to report ready. chunks_head/ holds the memory-mapped chunks of the current head block, roughly the last two to three hours. The ULID-named directories are immutable blocks, each with its own meta.json recording the time range and compaction level. lock is a file the process holds an flock on — that file is the entire mechanism preventing two Prometheus processes from opening the same directory, and Task 8 uses it.

Retention deletes whole blocks, and only when a block’s entire range falls outside the limit. It never touches the head or the WAL. So the oldest sample on disk can be older than your configured retention by up to the span of the largest block, and a churn storm can grow the WAL past a size limit that only budgets blocks. Both are reasons the volume needs headroom rather than an exact fit — 20 to 30 per cent is the usual figure, and compaction needs scratch space on the same filesystem to rewrite blocks into larger ones.

Task 5: Write the configuration and validate it before anything runs

sudo tee /etc/prometheus/prometheus.yml > /dev/null <<'YAML'
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    host: rb-lab-monitor

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
YAML

sudo tee /etc/prometheus/rules/self.yml > /dev/null <<'YAML'
groups:
  - name: self
    rules:
      # Prometheus can only tell you about targets it knows about. This rule
      # is about the one target it always has: itself.
      - alert: PrometheusTargetDown
        expr: up == 0
        for: 2m
        labels:
          severity: page
        annotations:
          summary: 'target {{ $labels.instance }} of job {{ $labels.job }} is down'

      - alert: PrometheusTsdbCompactionFailing
        expr: increase(prometheus_tsdb_compactions_failed_total[1h]) > 0
        for: 5m
        labels:
          severity: ticket
        annotations:
          summary: 'TSDB compaction has failed in the last hour'
YAML

sudo chown root:prometheus /etc/prometheus/prometheus.yml /etc/prometheus/rules/self.yml
sudo chmod 0640 /etc/prometheus/prometheus.yml /etc/prometheus/rules/self.yml

promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/self.yml

promtool check config also reports which rule files the glob actually matched. That number is worth reading every time: the rule_files glob matches *.yml and nothing else, so a file renamed to .yaml or .yml.disabled drops out of evaluation with no error anywhere — a glob that matches nothing is not an error condition. The alert simply stops existing, and you find out during the incident it was written for.

Task 6: Write the unit and harden it

The base unit first. Every flag is stated; nothing relies on a default you have not read:

# /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus monitoring system
Documentation=https://prometheus.io/docs/
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus

ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=15d \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=127.0.0.1:9090 \
  --web.enable-lifecycle

# SIGHUP: config and rule files are re-read in place. No restart, no WAL
# replay, no readiness gap. $MAINPID is the PID systemd is tracking, so the
# signal provably reaches the right process.
ExecReload=/bin/kill -HUP $MAINPID

Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Score the unit before the hardening, so the improvement is a measurement rather than a belief:

sudo tee /etc/systemd/system/prometheus.service > /dev/null <<'UNIT'
[Unit]
Description=Prometheus monitoring system
Documentation=https://prometheus.io/docs/
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=15d \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=127.0.0.1:9090 \
  --web.enable-lifecycle
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target
UNIT

sudo systemctl daemon-reload
systemd-analyze security prometheus.service | tail -3

Now the hardening, as a drop-in rather than an edit of the unit, so it survives whatever replaces the unit later:

# /etc/systemd/system/prometheus.service.d/10-hardening.conf
[Service]
# Scrapes, remote-write connections and open TSDB files each hold a
# descriptor. The 1024 default degrades past a few hundred targets, and it
# degrades as flapping scrapes rather than as a crash.
LimitNOFILE=65536

# The entire filesystem is read-only inside this unit's namespace...
ProtectSystem=strict
# ...except the one directory Prometheus legitimately writes.
ReadWritePaths=/var/lib/prometheus

ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
RestrictSUIDSGID=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictNamespaces=true
LockPersonality=true

# The server binds 9090, an unprivileged port, opens no raw sockets and
# touches no devices. It needs zero capabilities, so give it zero.
CapabilityBoundingSet=
AmbientCapabilities=

# Prometheus creates files 0666 and directories 0777 before the umask is
# applied. 0077 makes every WAL segment and block it ever writes 0600/0700.
UMask=0077

sudo install -d -m 0755 /etc/systemd/system/prometheus.service.d
sudo tee /etc/systemd/system/prometheus.service.d/10-hardening.conf > /dev/null <<'DROPIN'
[Service]
LimitNOFILE=65536
ProtectSystem=strict
ReadWritePaths=/var/lib/prometheus
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
RestrictSUIDSGID=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictNamespaces=true
LockPersonality=true
CapabilityBoundingSet=
AmbientCapabilities=
UMask=0077
DROPIN

sudo systemctl daemon-reload
systemd-analyze security prometheus.service | tail -3

Service impact possiblelab host— First start: creates the TSDB layout and begins scraping

$ sudo systemctl enable --now prometheus

Task 7: Validate from the running process

The unit file is what you asked for. /api/v1/status/flags is what you got, and only the second one can be trusted:

# Is it up, and did systemd apply what the drop-in says?
systemctl is-active prometheus
systemctl is-enabled prometheus
systemctl show prometheus -p User -p LimitNOFILE -p ProtectSystem \
  -p ReadWritePaths -p CapabilityBoundingSet -p UMask

# Past WAL replay and serving.
for i in $(seq 1 30); do
  curl -fsS http://127.0.0.1:9090/-/ready && break
  sleep 2
done

# The flags the process is actually running with.
curl -s http://127.0.0.1:9090/api/v1/status/flags \
  | jq -r '.data["config.file"], .data["storage.tsdb.path"],
           .data["storage.tsdb.retention.time"], .data["web.listen-address"]'

# The build the server reports, versus the binary on disk.
curl -s http://127.0.0.1:9090/api/v1/status/buildinfo | jq -r '.data.version'
prometheus --version | head -1

# Ownership of what the daemon created, courtesy of UMask=0077.
sudo ls -ld /var/lib/prometheus /var/lib/prometheus/wal
sudo ls -l /var/lib/prometheus/wal | head -3

# It is scraping itself, and the rules loaded.
curl -sG http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=up' \
  | jq -r '.data.result[] | "\(.value[1]) \(.metric.job) \(.metric.instance)"'
curl -s http://127.0.0.1:9090/api/v1/rules | jq -r '.data.groups[].rules[].name'

If buildinfo and prometheus --version disagree, an upgrade is half-applied: the binary on disk is not the binary that is running, and it will become the running one at the next restart, whenever that happens to be.

Task 8: Break it twice, on purpose

The read-only data directory. This is the failure ProtectSystem=strict causes when ReadWritePaths is forgotten, and it accounts for a large share of “Prometheus will not start after hardening”:

sudo tee /etc/systemd/system/prometheus.service.d/99-break.conf > /dev/null <<'DROPIN'
[Service]
ReadWritePaths=
DROPIN

sudo systemctl daemon-reload
sudo systemctl restart prometheus || true
sleep 3

systemctl is-active prometheus
journalctl -u prometheus -n 15 --no-pager | grep -iE 'read-only|opening storage|error'

The unit is in activating (auto-restart) or failed, and the journal names the path. Two commands identify it: journalctl -u prometheus -e for the error, and systemctl show prometheus -p ReadWritePaths for the cause. Note that the filesystem permissions are untouched and perfectly correct — under ProtectSystem=strict a path missing from ReadWritePaths is read-only to the unit no matter what ls -ld says, which is why checking modes first sends people down a long wrong road.

sudo rm -f /etc/systemd/system/prometheus.service.d/99-break.conf
sudo systemctl daemon-reload
sudo systemctl start prometheus
curl -fsS http://127.0.0.1:9090/-/ready

The data directory lock. One process per data directory, enforced by an flock on a single file:

sudo -u prometheus /usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.listen-address=127.0.0.1:19090 2>&1 | grep -i 'lock' | head -3

The second process exits immediately. There is no registry and no coordination — just a file the running process holds open. That is also why a forgotten container or a hand-started test instance is the whole explanation when the real service refuses to start after a reboot.

Task 9: Reload against restart, proved

Add a scrape job — a config change — and reload. Watch the change take effect while the process does not restart:

BEFORE_START=$(curl -sG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=process_start_time_seconds' | jq -r '.data.result[0].value[1]')

sudo tee -a /etc/prometheus/prometheus.yml > /dev/null <<'YAML'

  # A target that is deliberately not there, so `up == 0` means something.
  - job_name: absent-exporter
    static_configs:
      - targets: ['127.0.0.1:9100']
YAML

promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus
sleep 20

curl -sG http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=up' \
  | jq -r '.data.result[] | "\(.value[1]) \(.metric.job)"'

AFTER_START=$(curl -sG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=process_start_time_seconds' | jq -r '.data.result[0].value[1]')
echo "start time before=$BEFORE_START after=$AFTER_START"

The new job appears with up at 0, and the start time is unchanged: the process never restarted, so there was no WAL replay and no gap in scraping.

Now try the same thing with a flag. Retention lives in ExecStart, and flags are parsed once, at exec:

sudo tee /etc/systemd/system/prometheus.service.d/20-retention.conf > /dev/null <<'DROPIN'
[Service]
ExecStart=
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=1GB \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=127.0.0.1:9090 \
  --web.enable-lifecycle
DROPIN

sudo systemctl daemon-reload
sudo systemctl reload prometheus
sleep 5

# Still the old value. The reload did exactly what a reload does.
curl -s http://127.0.0.1:9090/api/v1/status/flags \
  | jq -r '.data["storage.tsdb.retention.time"], .data["storage.tsdb.retention.size"]'

The empty ExecStart= before the new one is not decoration: without it systemd appends a second command to the list rather than replacing the first.

Service impact possiblelab host— Restarts the server: WAL replay runs before it reports ready, and nothing is scraped in between

$ sudo systemctl restart prometheus

sleep 10
curl -s http://127.0.0.1:9090/api/v1/status/flags \
  | jq -r '.data["storage.tsdb.retention.time"], .data["storage.tsdb.retention.size"]'

Now 30d and 1GB. Both limits are set on purpose: time expresses the intent, size expresses the physics of a 2 GB volume, and whichever is reached first wins. On this filesystem, size will always win — which is the seatbelt working, not a misconfiguration.

Measure the ingestion rather than guessing at it. These are the three numbers a retention decision is actually made from:

for q in 'rate(prometheus_tsdb_head_samples_appended_total[5m])' \
         'prometheus_tsdb_head_series' \
         'prometheus_tsdb_lowest_timestamp'; do
  printf '%s = ' "$q"
  curl -sG http://127.0.0.1:9090/api/v1/query --data-urlencode "query=$q" \
    | jq -r '.data.result[0].value[1] // "no data yet"'
done

df -h /var/lib/prometheus
sudo du -sh /var/lib/prometheus/wal /var/lib/prometheus/chunks_head 2>/dev/null

At roughly two bytes per sample after compression, samples-per-second times seconds-of-retention times two is the planning figure for blocks; add 25 to 30 per cent for the WAL, the head and compaction scratch. Compare prometheus_tsdb_lowest_timestamp against your configured retention on any established server: if the oldest data is much newer than the retention implies, the size limit or the disk is quietly overriding your intent.

Validation

echo "== the unit is running what you wrote"
systemctl is-active prometheus
systemctl is-enabled prometheus
systemctl show prometheus -p NRestarts -p ProtectSystem -p ReadWritePaths \
  -p LimitNOFILE -p CapabilityBoundingSet -p UMask

echo "== the process agrees with the unit"
curl -s http://127.0.0.1:9090/api/v1/status/flags \
  | jq '{config: .data["config.file"],
         path: .data["storage.tsdb.path"],
         time: .data["storage.tsdb.retention.time"],
         size: .data["storage.tsdb.retention.size"],
         listen: .data["web.listen-address"]}'

echo "== the data is on its own filesystem, with headroom"
findmnt -no SOURCE,TARGET,FSTYPE,OPTIONS /var/lib/prometheus
df -h --output=target,size,used,avail,pcent /var/lib/prometheus | tail -1

echo "== ownership and modes are the model, not an accident"
sudo ls -ld /etc/prometheus /var/lib/prometheus
sudo ls -l /etc/prometheus/prometheus.yml
ls -l /usr/local/bin/prometheus

echo "== it is scraping, and the rules are loaded"
curl -sG http://127.0.0.1:9090/api/v1/query --data-urlencode 'query=up' \
  | jq -r '.data.result[] | "\(.value[1]) \(.metric.job)"'
curl -s http://127.0.0.1:9090/api/v1/rules \
  | jq -r '.data.groups[] | "\(.name): \(.rules | length) rule(s)"'

echo "== the API is not on the network"
ss -ltn 2>/dev/null | grep ':9090'

Expected: active, enabled, NRestarts=0; ProtectSystem=strict with ReadWritePaths=/var/lib/prometheus; the flags reporting 30d and 1GB; /var/lib/prometheus shown as its own ext4 mount; prometheus.yml at -rw-r----- root prometheus; one target up and one down; and 9090 bound to 127.0.0.1 only.

Expected Outcome

A pinned 2.55.x whose archive you verified, installed under /usr/local/bin, owned by root and not writable by the account that runs it.
A prometheus system account with no shell, owning the data directory and nothing else.
The TSDB on its own ext4 filesystem, with wal/, chunks_head/ and lock present and mode 0700/0600 courtesy of the umask.
systemd-analyze security scoring lower after the drop-in than before it.
A journal transcript of the read-only-filesystem failure, and a second Prometheus that refused to start because of the lock file.
Proof that a config change applied on reload without restarting the process, and that a flag change did not.

Troubleshooting

sha256sum -c prints FAILED. Stop. Re-download both files. A mismatch is either a truncated transfer or an archive that is not the published one, and neither is something to work around.

The service is in activating (auto-restart). Read the journal before changing anything: journalctl -u prometheus -e. A crash loop means the process exits during startup, and the error line is in there. read-only file system is Task 8’s failure; lock DB directory means a second process holds the data directory; unknown long flag means the binary predates a flag in your ExecStart.

systemctl reload returns “Job type reload is not applicable”. The unit has no ExecReload. Add it, daemon-reload, and note that this is one of the few things that needs a daemon-reload and not a restart.

POST /-/reload returns 403. --web.enable-lifecycle is not set on the running process. It is a flag, so adding it needs a restart, not a reload — which is the same lesson as Task 9 from the other direction.

A change to prometheus.yml did not take effect. Confirm the reload happened (journalctl -u prometheus -n 5 logs “Loading configuration file”), then confirm you edited the file the process is reading: curl -s http://127.0.0.1:9090/api/v1/status/flags | jq -r '.data["config.file"]'. A bad config on reload is not fatal — Prometheus logs the error and keeps serving the previous configuration, which is exactly why the change can appear to have been accepted.

df shows the mount point on / rather than the loop device. The mount failed and Prometheus is writing into the empty mount-point directory on the root filesystem. findmnt /var/lib/prometheus answers it in one line, and it is worth checking early because the symptom otherwise arrives as a full root filesystem weeks later.

Cleanup

LAB="$HOME/prometheus-deploy-lab"

# 1. Stop and forget the service, including both drop-ins.
sudo systemctl disable --now prometheus
sudo rm -rf /etc/systemd/system/prometheus.service.d
sudo rm -f /etc/systemd/system/prometheus.service
sudo systemctl daemon-reload
sudo systemctl reset-failed prometheus 2>/dev/null || true

# 2. Unmount the TSDB filesystem and remove the backing file.
sudo umount /var/lib/prometheus
findmnt /var/lib/prometheus || echo "unmounted"
sudo rm -f /opt/rb-lab/prometheus-tsdb.img
sudo rmdir /opt/rb-lab 2>/dev/null || true

# 3. Remove the trees and the binaries.
sudo rm -rf /etc/prometheus /var/lib/prometheus
sudo rm -f /usr/local/bin/prometheus /usr/local/bin/promtool

# 4. Remove the account.
sudo userdel prometheus
getent passwd prometheus || echo "account removed"

# 5. Remove only packages this lab installed. Read the capture first.
cat "$LAB/packages.pre-lab"
# sudo apt-get purge -y jq    # uncomment only for what was notinstalled

# 6. The lab directory, including the tarball.
rm -rf "$LAB"

Confirm the host is as you found it: ss -ltn should match ports.pre-lab, systemctl status prometheus should report the unit as not found, and findmnt /var/lib/prometheus should return nothing.

Production notes

This install maps onto a change window in four blocks, each with its own rollback:

Artefact and binaries. Rollback is removing two files; nothing is running, so there is no user impact. In a fleet this step is configuration management fetching a pinned version and verifying the checksum in the pipeline, not by hand on the host — a checksum somebody types is a checksum somebody skips.

Account, trees and filesystem. Rollback is userdel and umount. Pin the numeric uid across the fleet from configuration management: restores and shared storage care about the number, not the name.

Unit and drop-ins. Rollback is removing the drop-in and daemon-reload. Prefer drop-ins over forking the shipped unit, so a later change to the base unit does not silently drop your hardening.

First start. The only step with a real cost, because it creates the TSDB layout. On a rebuild rather than a new build, take the host out of whatever watches it first.

Three things this lab simplifies. The TSDB is a 2 GB loop file; production is a sized LVM volume on local SSD-class storage, in /etc/fstab, with an alert on both its utilisation and on the mount existing at all — because the failure where fstab loses a disk and Prometheus writes weeks of data into / is real and is not visible from inside Prometheus. The listener is on loopback with no TLS; production terminates TLS at a proxy or configures --web.config.file, and treats the lifecycle endpoints as administrative. And 15 to 30 days of local retention is the normal shape, with everything beyond that streamed to an object-storage-backed system by remote_write — which is a stream, not a backup, and does not help you restore local history.

What You Learned

The install method is the upgrade and rollback path you are choosing. The verified tarball plus a unit file gives a pinned artefact you can roll back to by pointing ExecStart at the previous binary. That property is the whole argument, and it is why this course defaults to it for production VMs.
The unit is a security boundary, not a start script. User=, NoNewPrivileges, an empty CapabilityBoundingSet and ProtectSystem=strict mean a compromised Prometheus cannot replace its own binary or write anywhere except one directory. The cost is one self-inflicted failure mode, and you produced and diagnosed it deliberately rather than at 03:00.
Filesystem permissions and namespace permissions are different things. ls -ld said the data directory was writable while the process could not write to it. The command that answers that question is systemctl show -p ReadWritePaths, and knowing to reach for it is most of the diagnosis.
Reload and restart are not two names for the same thing. A config or rule change reloads with no WAL replay and no scrape gap. A flag change — retention, storage path, listen address — needs a restart, and a reload will report success while applying nothing.
Trust the running process, not the file you edited. /api/v1/status/flags reads the server; systemctl cat reads the disk. Every half-applied upgrade and every forgotten daemon-reload shows up as a disagreement between the two.

Objectives

Prerequisites