medium riskinformational~90 min

Runbook: Install Prometheus

1 · Prerequisites

Confirm every item is in place before any state change.

Prometheus Installation Methods
Filesystem Layout
systemd Integration
Retention and Storage
Permissions and Service User
The exact Prometheus version this estate runs, written down as a full version string - never "latest"
A host running systemd, with sudo, and either outbound TLS to the project release page or a local mirror of the release tarball and its checksum file
A sized filesystem for the TSDB, or a recorded decision that this host is a lab and the root filesystem is acceptable
The prometheus.yml and rule files this server will run, in version control - not composed on the host during the change
A decision recorded on whether this server is the first, replaces an existing one, or runs alongside one as an HA pair

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The target version is an exact string, and it matches what the rest of the estate reports from its own process (curl -s "http://$PEER:9090/api/v1/status/buildinfo") rather than what the wiki says is deployed. A new host built on a version nobody else runs is a host whose behaviour nobody can predict from experience.
· The release archive and the published sha256sums.txt have both been fetched over TLS from the project release page, and the archive verifies. A mismatch is a stop condition, not a warning: it means either a truncated download or a mirror you should not be using.
· Nothing already listens on the port this server will bind: ss -lntp | grep :9090 returns nothing. A forgotten container on 9090 is the commonest reason a correct unit file starts and immediately dies.
· No process already holds the intended data directory. A second Prometheus on the same path exits at start with opening storage failed: lock DB directory; pgrep -a prometheus and docker ps between them find it. The lock is a file in the data directory, not a registry, so nothing else will warn you.
· The filesystem for --storage.tsdb.path is mounted, is local ext4 or XFS, and is the size the retention plan assumed. findmnt /var/lib/prometheus proves it is a mount and not an empty mount-point directory on /; df -h proves the size. NFS and SMB do not give the TSDB the locking and durability semantics it needs.
· The retention figure has been computed rather than copied: samples per second, times seconds of retention, times about 2 bytes per sample, plus 25-30% headroom for compaction scratch, the WAL and the head. The built-in default is 15d, which is almost never what an estate actually wants.
· The configuration that will be installed parses under the promtool from this release: promtool check config and promtool check rules both exit 0. The promtool in the tarball is the only one guaranteed to agree with the binary next to it.
· A decision is recorded on whether this server replaces an existing one. Two Prometheus servers scraping the same targets is a deliberate HA pattern that needs distinct external_labels; without that decision it is an accident that shows up later as duplicated series.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Verify the artefact before anything is installed. Fetch the tarball and sha256sums.txt from the project release page, then check the archive against the published digest. Do this on the host that will run it, so that what you verified and what you unpack are the same bytes.
2Create the service account. useradd --system --no-create-home --home-dir /var/lib/prometheus --shell /usr/sbin/nologin prometheus. Pin the numeric uid from configuration management: restores and shared storage care about the number, not the name.
3Install the binaries as root, owned by root, mode 0755. prometheus and promtool go to /usr/local/bin. The service user must not be able to replace its own executable - that is the line between a compromised Prometheus and a persistent one.
4Create the configuration tree at /etc/prometheus and /etc/prometheus/rules, owned root:prometheus mode 0750, and copy consoles/ and console_libraries/ out of the tarball into it. The service reads through group membership; only root writes.
5Create the data directory at /var/lib/prometheus/data - a subdirectory of the dedicated mount, not the mount root - owned prometheus:prometheus mode 0750. Keeping the TSDB one level below the mount point is what lets a recovery move the whole database aside without disturbing the mount, and it costs nothing to do now.
6Deploy prometheus.yml and the rule files from version control, mode 0640 root:prometheus, then validate them with the promtool you just installed. Config routinely holds basic_auth passwords and remote-write tokens; world-readable is a credential leak.
7Write the unit file with every flag explicit - config path, storage path, retention, console paths - plus ExecReload, Restart=on-failure, a raised LimitNOFILE, and the sandbox. ProtectSystem=strict must be paired with ReadWritePaths for the data directory or the process cannot write at all.
8**daemon-reload, then enable --now, then read the journal.** systemctl start returning 0 means systemd forked the process, not that Prometheus is healthy. The startup banner names the storage path it actually opened; read it.
9Run the read-only proof sequence against the running process, not against the files on disk: version, effective flags, resolved config, readiness, targets, rules. The API answers what the process loaded; the unit file answers what you intended. They are different claims.
10Decide the exposure of port 9090 before this host is reachable by anything else. The HTTP API is unauthenticated by default, and --web.enable-lifecycle additionally exposes POST /-/quit, which stops the server. Bind to a restricted interface or front it with an authenticating proxy.
11Record the install: version string and archive digest, the config repository commit, the retention settings, the data path and its volume size, and who is permitted to reach 9090. An unrecorded monitoring server is a host holding scrape credentials that nobody is tracking.

4 · Verification

Confirm the procedure actually fixed the problem.

✓prometheus --version on the host and curl -s http://localhost:9090/api/v1/status/buildinfo from the API report the same version string. A difference means an upgrade is half-applied: new binary on disk, old process still running.
✓curl -s http://localhost:9090/api/v1/status/flags reports the config.file and storage.tsdb.path you intended, and the retention values you computed. This reads the running process and cannot be fooled by a stale unit file or a forgotten drop-in.
✓findmnt /var/lib/prometheus shows the dedicated filesystem, and /var/lib/prometheus/data now contains wal/ and chunks_head/ owned by prometheus. Files owned by anyone else are the fingerprint of an earlier root-run test and need cleaning up now, not later.
✓systemctl show prometheus -p User -p LimitNOFILE -p ProtectSystem -p ReadWritePaths returns the identity, limit and sandbox you wrote. systemctl is-enabled prometheus prints enabled, which is the only thing that makes this install survive a reboot.
✓curl -s http://localhost:9090/-/healthy and /-/ready both return 200, and the journal shows the configuration loaded without error. healthy means the process is alive; ready means the TSDB is open and the config is valid. Only the second one is a claim about correctness.
✓curl -s http://localhost:9090/api/v1/targets lists every job in the configuration and every target reports up, or each exception is written down with a reason. A job that is absent entirely means the config Prometheus loaded is not the file you edited.
✓curl -s http://localhost:9090/api/v1/rules lists every rule group, each with health: ok. An empty result when rule files exist almost always means the rule_files glob matched nothing - a glob that matches nothing is not an error anywhere in the system.
✓A restart proves itself: systemctl restart prometheus, then /-/ready returns 200 again and the target list is unchanged. An install that has never been restarted is an install whose first reboot is an experiment.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Nothing on any scrape target changed, so rollback here is removal rather than repair. Stop and disable first: systemctl disable --now prometheus, then confirm with systemctl is-active prometheus and pgrep -x prometheus.
↶If the install is being abandoned: remove the unit and daemon-reload, remove /usr/local/bin/prometheus and /usr/local/bin/promtool, then remove /etc/prometheus and the data directory. Take the data directory last, and only after confirming nothing else was mounted underneath it.
↶Treat prometheus.yml as a credential file on the way out. If it carried basic_auth passwords or remote-write tokens, deleting the host does not rotate them - those credentials are now in whatever backup ran while the file existed, and the rotation is a separate change.
↶If this host was intended to replace an existing Prometheus, leave the old one running and scraping until this one has passed the full verification list. A cutover is a separate step with its own decision, not the tail end of an install.
↶If the service starts but is wrong rather than broken, prefer a targeted fix over a rebuild: flags need daemon-reload plus a restart, while config and rule changes need only systemctl reload. Restarting for a rule edit costs a WAL replay and a scrape gap for nothing.
↶The one thing removal does not undo: if targets were reconfigured to expose or allow this scraper - a firewall rule, an exporter binding, a token issued for remote write - those changes live on the targets and survive the host. Reverse them explicitly.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the platform owner if the checksum does not match. Do not retry the download until it happens to succeed, and do not skip the check to make progress: a failing digest is either a broken mirror or a tampered artefact, and both are somebody else name to investigate.
· Escalate to the storage owner if no dedicated filesystem is available and the estate expects production retention. Running a production TSDB on the root filesystem is a capacity decision with a host outage at the end of it, and it is not an operator convenience call.
· Escalate to the security owner before exposing port 9090 beyond the host. The API is unauthenticated by default, --web.enable-admin-api adds endpoints that delete series, and --web.enable-lifecycle adds one that stops the server.
· Escalate to the network or platform team if scrape targets are unreachable from this host after the install verifies locally. Prometheus being healthy and Prometheus being able to see anything are separate problems with separate owners.
· Escalate before pointing alert routing at this server while another Prometheus already covers the same targets. Which server is authoritative for paging is a change-control question, not an install step.

Prometheus is a single statically linked binary, and that is exactly why this runbook exists. Getting it running takes about ninety seconds. Getting it running in a shape you can upgrade, roll back, back up and hand to somebody else takes the rest of this page.

The difference is not the binary. It is four decisions the binary does not make for you: which version you are on and how you can prove it, who the process runs as, where its data goes and what happens when that fills, and how the machine brings it back after a reboot. An install that skips those produces a Prometheus that works today and becomes an archaeology exercise in six months.

When to use this runbook

Building the first Prometheus for an estate.
Adding a second server - a separate environment, a separate region, or the second half of an HA pair.
Rebuilding a host after a loss. Use this runbook for the build, and the recovery runbook for the question of what to do about the data that only existed on the lost machine.

It does not apply when:

Prometheus is already installed and needs a newer version. That is the upgrade runbook. The mechanics overlap; the risk profile does not, because an upgrade has existing data underneath it and this does not.
The estate runs Prometheus in containers. The decisions are the same - pinned tag instead of pinned tarball, named volume instead of a mount - but the commands are not, and a half-container half-tarball estate is worse than either done consistently.
A distribution package is the standing policy. That is a legitimate choice with a different upgrade story. Follow it consistently rather than mixing methods on one host.

Blast radius

Steps 1 to 7 touch nothing that exists. No scrape target is contacted, no alert is routed, no other service is reconfigured. On a greenfield host this whole runbook is informational right up to step 8.

Step 8 starts a process that opens a listener and begins connecting outward to every target the configuration names. Two things change at that moment: the host is now reachable on 9090 by anything that can route to it, and the targets are now receiving requests from a new source. On a shared network, that is the point at which the change becomes visible to other people.

There is one way this runbook becomes service-affecting earlier: if the host is not greenfield. Something already bound to 9090, or an existing data directory at the path you are about to use, means you are not installing - you are colliding. The pre-checks exist to find that before step 2.

Step 1: Verify the artefact

Read-only / Safefetch and verify the release tarball

VER=2.55.1
BASE=https://github.com/prometheus/prometheus/releases/download

curl -fsSLO "$BASE/v$VER/prometheus-$VER.linux-amd64.tar.gz"
curl -fsSLO "$BASE/v$VER/sha256sums.txt"

grep "prometheus-$VER.linux-amd64.tar.gz" sha256sums.txt | sha256sum -c -

prometheus-2.55.1.linux-amd64.tar.gz: OK

Illustrative output

The digest proves the archive on this disk is byte-identical to the one the release pipeline produced. It catches truncated downloads and tampered mirrors. It does not, on its own, prove the checksum file is trustworthy - that comes from fetching it over TLS from the project’s own release page, which is why both URLs above point at the same place.

Treat a mismatch as a full stop. The temptation at 22:00 is to re-run the download until it passes, and occasionally that is genuinely a flaky link; but a second failure is information, not noise.

Step 2: The service account

Configuration changeuseradd

sudo useradd --system --no-create-home \
--home-dir /var/lib/prometheus \
--shell /usr/sbin/nologin \
prometheus

id prometheus

--system keeps the uid below 1000 and out of login prompts. The nologin shell blocks interactive login without blocking sudo -u prometheus, which is how you will reproduce a failure by hand in step 8.

Pin the numeric uid from configuration management. The name is what you read; the number is what the filesystem stores, and a restore onto a host where prometheus resolved to a different uid produces a data directory the service cannot open.

Steps 3 to 5: Binaries, config tree, data directory

Configuration changeinstall binaries and directories

VER=2.55.1
tar xzf "prometheus-$VER.linux-amd64.tar.gz"
cd "prometheus-$VER.linux-amd64"

# Binaries: root-owned, so the service user cannot replace its own executable.
sudo install -o root -g root -m 0755 prometheus /usr/local/bin/prometheus
sudo install -o root -g root -m 0755 promtool   /usr/local/bin/promtool

# Config tree: root writes, the service reads through the group.
sudo install -d -o root -g prometheus -m 0750 /etc/prometheus /etc/prometheus/rules
sudo cp -r consoles console_libraries /etc/prometheus/

# Data directory: one level below the mount point, owned by the service.
sudo install -d -o prometheus -g prometheus -m 0750 /var/lib/prometheus
sudo install -d -o prometheus -g prometheus -m 0750 /var/lib/prometheus/data

One layout decision hides in that last line. The dedicated filesystem is mounted at /var/lib/prometheus, and the TSDB lives one level below it at /var/lib/prometheus/data. That single extra directory is what makes the recovery runbook possible: a damaged database can be renamed out of the way, on the same filesystem, without touching the mount point itself. Point --storage.tsdb.path at the mount root and that move stops being available on the night you need it.

The ownership split is the whole security model of the install, and it is worth one sentence each. Binaries root-owned means a compromised Prometheus cannot rewrite itself and survive a restart. Config root-owned with group read means configuration management is the only thing that changes it, while the service can still read the basic_auth passwords and remote-write tokens that live in it. Data service-owned because the TSDB creates, memory-maps and deletes files there constantly, and nobody else has any business in there at all.

Step 6: Configuration from version control

Configuration changedeploy and validate configuration

sudo install -o root -g prometheus -m 0640 \
/srv/monitoring-config/prometheus.yml /etc/prometheus/prometheus.yml
sudo install -o root -g prometheus -m 0640 \
/srv/monitoring-config/rules/node.yml /etc/prometheus/rules/node.yml

# Validate with the promtool from this release, not one already on the host.
promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/*.yml

Checking /etc/prometheus/prometheus.yml
SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Illustrative output

promtool check config parses the file exactly as the server would, and it follows rule_files to check that the globs resolve. That second part is the one that earns its keep: a rule file renamed to .yaml, or parked as alerts.yml.disabled during a test and never restored, drops silently out of evaluation. A glob that matches nothing is not an error to anyone.

Compose the configuration in the repository and copy it here. A file written by hand on the host is a file that differs from every other host in the estate, and the difference will be discovered during an incident rather than during a review.

Step 7: The unit file

Configuration change/etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus monitoring system
Documentation=https://prometheus.io/docs/
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/var/lib/prometheus

[Service]
User=prometheus
Group=prometheus

ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus/data \
--storage.tsdb.retention.time=30d \
--storage.tsdb.retention.size=450GB \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--web.listen-address=127.0.0.1:9090 \
--web.enable-lifecycle

# SIGHUP: re-read config and rules, no restart, no WAL replay.
ExecReload=/bin/kill -HUP $MAINPID

Restart=on-failure
RestartSec=5s
LimitNOFILE=65536

ProtectSystem=strict
ReadWritePaths=/var/lib/prometheus
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
CapabilityBoundingSet=
AmbientCapabilities=
UMask=0077

[Install]
WantedBy=multi-user.target

Every flag is written out rather than left to a default, because a default you have not read is a decision somebody else made for you and did not tell you about. Retention is the clearest case: the built-in default is 15 days, and an estate that assumed 30 discovers the difference the first time an investigation needs three-week-old data.

Setting both retention limits is deliberate. Time expresses intent, size expresses physics, and whichever is reached first wins. The size budget covers compacted blocks only - the WAL and the head sit outside it - so leave real headroom rather than budgeting to the edge of the volume.

ProtectSystem=strict mounts the entire filesystem read-only inside the service’s namespace, and ReadWritePaths re-opens exactly one directory. Forget the second line and startup dies with read-only file system naming the data path. UMask=0077 makes every WAL segment and block the process ever writes private to the service without any per-file work from you.

Step 8: Start it, and read the journal

Service impact possibleenable and start

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus

# systemctl returning 0 means systemd forked it. That is not the same claim.
journalctl -u prometheus -n 30 --no-pager

The startup lines are free validation. Prometheus announces the storage path it opened, then reports the configuration file it loaded, then says it is ready to receive web requests. If the path in that banner is not the path you created in step 5, stop: the unit systemd loaded is not the file you edited, and daemon-reload is the usual reason.

Three failures show up here and nowhere else, each with an unambiguous line in the journal:

opening storage failed: lock DB directory - something else already holds the data directory. Find it with pgrep -a prometheus and docker ps.
read-only file system naming the data path - ProtectSystem=strict without the matching ReadWritePaths.
unknown long flag - the binary predates a flag in your ExecStart. On a fresh tarball install that means you are not running the binary you think you are; check prometheus --version.

If it works when you run it by hand and fails under systemd, the problem is the unit - sandboxing, limits or identity - and not Prometheus:

Read-only / Safereproduce by hand as the service user

sudo -u prometheus /usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus/data

Step 9: The read-only proof sequence

Run these in order. Each answers a question the next one assumes.

Read-only / Safeproof sequence

PROM=http://localhost:9090

# 1. Which binary is the process actually running?
curl -fsS "$PROM/api/v1/status/buildinfo" | jq -r '.data.version'

# 2. Which flags are in effect? Reads the process, not the unit file.
curl -fsS "$PROM/api/v1/status/flags" \
| jq -r '.data["config.file"], .data["storage.tsdb.path"],
         .data["storage.tsdb.retention.time"]'

# 3. Is the TSDB open and the config valid?
curl -fsS "$PROM/-/ready"

# 4. Did it find every job, and can it reach them?
curl -fsS "$PROM/api/v1/targets" \
| jq -r '.data.activeTargets[] | "\(.labels.job) \(.health)"'

# 5. Did the rule files load, and are they evaluating?
curl -fsS "$PROM/api/v1/rules" \
| jq -r '.data.groups[] | "\(.name) \(.rules | length)"'

Check 2 is the one people skip and the one that catches the most. It reports what the running process loaded, so a stale unit file, an unapplied daemon-reload or a forgotten drop-in under /etc/systemd/system/prometheus.service.d/ cannot hide behind it.

Check 4 distinguishes two very different failures that look identical on a dashboard. A job present with targets down means the configuration is right and the network or the exporter is wrong. A job absent entirely means Prometheus never loaded the configuration you think it did.

Step 10: Prove the reboot

Service impact possiblerestart and re-verify

sudo systemctl restart prometheus
sleep 10
curl -fsS http://localhost:9090/-/ready
systemctl is-enabled prometheus

Prometheus Server is Ready.
enabled

Illustrative output

An install that has never been restarted is an install whose first reboot is an experiment, run unattended, at whatever hour the kernel update lands. The restart also teaches you the shape of your own recovery time: on a small server readiness returns in seconds, and on a large one WAL replay can take minutes during which nothing is scraped. Knowing which you have is worth the ten seconds now.

Where you stop and think

The checksum does not match. Not a retry loop. Either the mirror is broken or the artefact is not what the project published, and both belong to somebody else to answer.

There is no dedicated filesystem. Continuing puts a growing database on the root filesystem, where filling it takes down sshd, journald and the host’s ability to tell anyone about it. For a lab, record the decision and set a small retention. For production, this is a capacity conversation before it is an install.

The host is not greenfield. Something on 9090, or an existing data directory at your target path, means the pre-checks found a collision. Resolve what is already there before creating a second thing that wants the same lock.

Alert routing is about to point here while another server already covers these targets. Which Prometheus is authoritative for paging is a change-control decision. Two servers both routing to Alertmanager is a valid HA design with distinct external_labels; it is also what an accident looks like.

Rollback

Destructiveabandon the install

sudo systemctl disable --now prometheus
systemctl is-active prometheus
pgrep -x prometheus

sudo rm -f /etc/systemd/system/prometheus.service
sudo systemctl daemon-reload
sudo rm -f /usr/local/bin/prometheus /usr/local/bin/promtool

# Config may hold credentials. Shred rather than unlink.
sudo shred -u /etc/prometheus/prometheus.yml
sudo rm -rf /etc/prometheus

# Data last, and only after confirming nothing else is mounted here.
findmnt /var/lib/prometheus
sudo rm -rf /var/lib/prometheus

Nothing on any scrape target changed, so this is removal rather than repair - which is the easiest rollback in this course, and the reason the install is worth doing carefully before anything depends on it.

Two things removal does not undo. First, if prometheus.yml carried credentials and any backup ran while it existed, those credentials are now in the backup system and deleting the host does not rotate them. Second, if targets were changed to accept this scraper - a firewall rule, an exporter binding, a token issued for remote write - those changes live on the targets and survive this host entirely. Reverse them explicitly or write them down.

Common patterns

Symptom	Likely cause	Resolution
Service starts, then dies within a second	Another process holds the data directory, or a flag the binary does not know	`journalctl -u prometheus -e`; the line names which
`read-only file system` on the data path	`ProtectSystem=strict` without `ReadWritePaths`	Add the directive, `daemon-reload`, restart
Flags in the API do not match the unit file	Unit edited without `daemon-reload`, or a drop-in is overriding it	`systemctl cat prometheus` shows the parsed unit and every drop-in
Targets page is empty; no errors anywhere	Prometheus loaded a different config than you edited	`/api/v1/status/flags` names the file it actually read
Rules exist on disk but `/api/v1/rules` is empty	`rule_files` glob matches nothing after a rename	`promtool check config` follows the glob and reports it
`too many open files` once targets grow	`LimitNOFILE` left at the 1024 default	Raise it in the unit; verify with `systemctl show -p LimitNOFILE`
Data directory grows onto `/` despite a dedicated volume	The volume did not mount and the mount point was writable	`findmnt`; add `RequiresMountsFor=` to the unit

Why this matters

The install decides your next two years with this server. Not because the binary is hard to replace - it is one file - but because everything expensive about running Prometheus is downstream of choices made in the first hour.

A pinned version with a recorded digest is what makes an upgrade a comparison rather than a guess. A dedicated filesystem is what turns “the TSDB filled up” into a monitoring outage instead of a host outage. A unit file with explicit flags is what lets the next engineer read one file and know what is running, instead of reverse-engineering it from the process table. And a service account that cannot rewrite its own binary is what keeps a bad afternoon from becoming a persistent compromise.

The failure this procedure exists to prevent is the quiet one. Not a Prometheus that does not start - that announces itself in the journal within a second. It is the Prometheus that starts, looks entirely healthy, and is silently not what anyone believes it is: an older version than the fleet, a config file nobody has in git, a rule file that dropped out of a glob, data on the wrong disk. Every check in step 9 exists because it reads the running process rather than the intention.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

When to use this runbook

Blast radius

Step 1: Verify the artefact

Step 2: The service account

Steps 3 to 5: Binaries, config tree, data directory

Step 6: Configuration from version control

Step 7: The unit file

Step 8: Start it, and read the journal

Step 9: The read-only proof sequence

Step 10: Prove the reboot

Where you stop and think

Rollback

Common patterns

References