Prometheus Installation Methods · Observability

Ask three engineers how Prometheus got onto their servers and you will get three answers: the distribution package manager put it there, Ansible untarred it there, or it runs in a container that nobody remembers starting. All three answers produce a running Prometheus. Only some of them produce a Prometheus you can upgrade, roll back, and patch at 03:00 with confidence.

This lesson compares the four realistic ways to install Prometheus 2.55.x on the platforms this course targets — Ubuntu 24.04, Debian 12, RHEL 9, and Docker 28.x — and finishes with what a production install actually needs beyond a binary that starts.

What it is

Prometheus ships as a single, statically linked Go binary. There is no runtime, no interpreter, no library dependency to satisfy. Every installation method is, at bottom, a different answer to the same three questions: where does the binary come from, who rebuilds it when a security fix lands, and how does it get onto the host repeatably.

The four methods:

Prebuilt binaries from prometheus.io — official release tarballs published by the Prometheus project for linux/amd64, linux/arm64 and more. The course default for VMs and bare metal.
Distribution packages — apt install prometheus on Debian and Ubuntu; EPEL on RHEL 9. Convenient, and almost always behind upstream.
Container images — prom/prometheus on Docker Hub (mirrored at quay.io/prometheus/prometheus). The course default for labs and container estates.
Building from source — cloning prometheus/prometheus and building with the project’s own tooling. Almost never the right answer in production.

Why a sysadmin cares

The install method is not a one-time choice; it is the upgrade path you will live with for years. It decides:

How fast you get security fixes. With the official tarball you can be on the fixed release the day it ships. With a distro package you wait for the maintainer — or you get the distro’s backport, which is a different promise (see below).
Whether rollback is boring or terrifying. A tarball install rolls back by pointing a symlink or unit file at the previous binary. A latest container tag has no previous anything.
Whether the install is reproducible. Fifty hosts installed by hand in tmux sessions diverge. Fifty hosts installed by one verified artefact plus one unit file do not.

Method 1: prebuilt binaries

The release tarball contains exactly what the name implies — no installer, no post-install scripts, nothing written outside the directory you unpack:

prometheus-2.55.1.linux-amd64/
├── prometheus            # the server, statically linked
├── promtool              # config/rule validator, same version
├── prometheus.yml        # minimal example config
├── consoles/             # console templates
├── console_libraries/    # helper libraries for consoles
├── LICENSE
└── NOTICE

The install procedure is short, and every step is auditable:

# READ-ONLY -- fetch the archive and the published checksum list
VER=2.55.1
curl -fsSLO "https://github.com/prometheus/prometheus/releases/download/v${VER}/prometheus-${VER}.linux-amd64.tar.gz"
curl -fsSLO "https://github.com/prometheus/prometheus/releases/download/v${VER}/sha256sums.txt"

# READ-ONLY -- verify the archive against the published checksum
grep "prometheus-${VER}.linux-amd64.tar.gz" sha256sums.txt | sha256sum -c -
# prometheus-2.55.1.linux-amd64.tar.gz: OK

# CONFIGURATION -- install binaries and skeleton directories
tar xzf "prometheus-${VER}.linux-amd64.tar.gz"
sudo install -o root -g root -m 0755 \
  "prometheus-${VER}.linux-amd64/prometheus" /usr/local/bin/prometheus
sudo install -o root -g root -m 0755 \
  "prometheus-${VER}.linux-amd64/promtool" /usr/local/bin/promtool

The checksum step is not ceremony. It proves the archive on your disk is byte-identical to the one the release pipeline produced — corrupted downloads and tampered mirrors both fail it. What it does not prove on its own is authenticity of the checksum file itself; fetch it over TLS from the project’s own release page, and treat any mismatch as a stop condition, not a warning.

The tarball deliberately does not create a user, a config in /etc, a data directory, or a unit file. Those are lessons 02, 03 and 06 of this module, and their absence here is a feature: nothing is hidden from you.

Method 2: distribution packages

apt install prometheus works, and for a throwaway VM it is the fastest path to a running server. The cost is version lag that follows from how stable distributions are made: the version is frozen when the release branches, and afterwards only selected fixes are backported.

Debian 12 (bookworm) ships Prometheus 2.42.x.
Ubuntu 24.04 (noble) ships 2.45.x.
RHEL 9 ships no Prometheus in BaseOS or AppStream; EPEL carries a package that trails upstream and follows Fedora file layout conventions, which differ from the layout this course uses.

Against 2.55.x that is a gap of one to two years. Concretely, that means config fields and flags your runbooks reference may not exist, and the service fails at startup with unknown long flag the day someone copies a modern config onto an old binary.

The honest trade-off: a distro package gives you the distribution’s security machinery — backported fixes delivered through the same apt upgrade cadence as everything else — at the price of feature currency. For a homelab or a host where Prometheus is incidental, that trade can be correct. For the platform your incident response depends on, running the version your documentation describes wins.

Method 3: the container image

The official image is prom/prometheus. Two properties matter before you run it:

The entrypoint is the prometheus binary itself, so every container argument is a Prometheus flag: docker run prom/prometheus:v2.55.1 --config.file=/etc/prometheus/prometheus.yml.
The process inside runs as nobody (uid 65534), not root — good for security, and the source of the classic bind-mount permission failure covered in lesson 04.

Pin an exact tag. prom/prometheus:v2.55.1 is a reproducible artefact you can roll back to; latest is a pointer that can jump a major version the next time the container is recreated, and major versions change default behaviour. Lesson 04 builds the full Compose service, including the volume the TSDB needs to survive container replacement.

Method 4: building from source

# READ-ONLY -- what a source build involves (do not do this on prod hosts)
git clone https://github.com/prometheus/prometheus.git
cd prometheus
make build   # uses the project's promu build tool

You need a Go toolchain and Node for the web UI, and what you get at the end is a binary functionally identical to the one in the tarball — minus the release testing, minus the published checksums, plus a permanent obligation: every upstream security fix now requires you to rebuild and redeploy. Teams that build from source to “stay current” routinely end up six months behind, because rebuilding is work and downloading is not.

Build from source when you are developing a patch for upstream, or when you target a platform with no prebuilt binary. Otherwise take the artefact the project already built and verified.

How to validate it

Whatever the method, the acceptance checks are the same:

# READ-ONLY -- the binary reports the version you think you installed
prometheus --version
# prometheus, version 2.55.1 (branch: HEAD, revision: 4f3c2a1...)
#   build user:       root@buildkitsandbox
#   build date:       20250108-11:22:33
#   go version:       go1.23.4
#   platform:         linux/amd64
#   tags:             netgo,builtinassets

# READ-ONLY -- the config parses before the service ever sees it
promtool check config /etc/prometheus/prometheus.yml
# Checking /etc/prometheus/prometheus.yml
#  SUCCESS: prometheus.yml is valid prometheus config file syntax

# READ-ONLY -- the service is up under systemd
systemctl status prometheus --no-pager

# READ-ONLY -- the server finished WAL replay and serves traffic
curl -s http://localhost:9090/-/ready
# Prometheus Server is Ready.

# READ-ONLY -- the flags actually in effect, from the horse's mouth
curl -s http://localhost:9090/api/v1/status/flags | jq -r '.data["storage.tsdb.path"]'
# /var/lib/prometheus

The last check matters more than it looks: it reads the running process, not the unit file. If the two disagree, the unit file is not what is running — a failure mode in itself.

How it can fail

Distro version too old for the config. A flag or field that postdates the distro build makes the service exit at startup with unknown long flag. The version gap is discovered during the change, not before it.
Unplanned major upgrade via latest. A routine container recreate pulls a new major version with different defaults; behaviour changes or the config no longer loads. Symptom: the outage coincides with a routine docker compose up -d.
Skipped checksum. A truncated download unpacks into a binary that crashes unpredictably, or worse. sha256sum -c costs two seconds.
Hand-started process. Prometheus launched from a shell or tmux “just for now” dies with the session and does not return after reboot. Monitoring is silently absent until the next incident needs it.
Source build drift. The locally built binary is never rebuilt; CVEs accumulate; the version string no longer matches any upstream release, so nobody can tell what it contains.
Container without a volume. The TSDB lives in the container’s writable layer; the next docker compose down or recreate takes all history with it. Covered in depth in lesson 04.

How to troubleshoot it

When “Prometheus is weird” after an install or upgrade, in order:

Which binary is actually running? curl -s localhost:9090/api/v1/status/buildinfo | jq .data.version for the server; prometheus --version for the binary on disk. If they differ, an upgrade is half-applied.
Who owns the process? systemctl status prometheus or docker ps. A process owned by neither is the tmux failure mode.
Which flags are in effect? /api/v1/status/flags, not the unit file on disk.
What did it say at startup? journalctl -u prometheus -e or docker logs. unknown long flag means version gap; checksum and extraction problems surface earlier, at install time.
Reproduce by hand. Run the binary in the foreground with the same flags as the service user. If it works by hand but not under the service manager, the problem is the unit or the packaging, not Prometheus.

Security implications

The tarball is a supply-chain artefact: fetch over TLS from the project’s own release infrastructure and verify the SHA256. A monitoring binary is a privileged position — it holds credentials for remote write and basic auth, and it talks to everything.
Distro packages shift patching to the distribution’s security team, which backports fixes to its frozen version. That is a real control, not an inferiority; weigh it against the feature lag.
Container images: trust only the prom/ organisation (or the quay.io mirror), pin tags, and rebuild your deployment on a schedule so base-image fixes reach you.
Regardless of method: the Prometheus HTTP API is unauthenticated by default. Installation is the right time to bind it to a restricted interface; the hardening lessons of this course cover the rest.

Performance implications

The running code is identical across methods, so steady-state performance does not depend on how you installed. The exceptions are all about where the bytes land: a container TSDB without a volume writes through the overlay filesystem (slower, and lost on recreate), and a source build forgotten on an old Go runtime misses years of allocator improvements. Installation method is a performance topic only when it changes the storage path — which is lesson 02’s subject.

Production guidance

Standardise on one method per estate. For VM fleets: verified tarball plus systemd unit, driven by configuration management. For container estates: pinned prom/prometheus tag with a named volume.
Pin the version everywhere — in the download URL, in the image tag, in the configuration management variable.
Verify the checksum in the pipeline, not by hand on the host.
Never build from source for production; never run latest.
Record the method in the runbook: the next engineer should not have to reverse-engineer the install from the process table.

Verification

You should now be able to answer:

Which installation method does this course default to for a VM, and why?
Why do Debian 12 and Ubuntu 24.04 ship older Prometheus versions, and what do you get in exchange for the lag?
What does sha256sum -c against the release checksums prove, and what does it not prove?
Why is prom/prometheus:latest unacceptable in production?
Name four things a production install needs besides the binary.