Filesystem Layout

An engineer inherits a host that “runs Prometheus”. The service is up and dashboards render. Now the questions start: which file is the live config? Where is the data, and how much room is left before the TSDB eats the operating system disk? Where did last night’s crash log go? If any of those take more than ten seconds to answer, the install was never finished — it was only started.

This lesson fixes the mental map: one config tree, one data directory, one log stream. Everything else is detail.

The contract

Prometheus makes almost no demands on the filesystem. It reads one configuration file, it writes one data directory, and it logs to standard output. Everything else — where those live, who owns them, which disk they sit on — is your decision. The layout this course uses:

/usr/local/bin/prometheus        server binary        (root:root 0755)
/usr/local/bin/promtool          config/rule checker  (root:root 0755)

/etc/prometheus/prometheus.yml   main configuration   (root:prometheus 0640)
/etc/prometheus/rules/*.yml      recording + alert rules
/etc/prometheus/filesd/*.yml     file_sd target lists (if used)
/etc/prometheus/consoles/        console templates
/etc/prometheus/console_libraries/

/var/lib/prometheus/             TSDB data directory, own filesystem
                                 (--storage.tsdb.path, prometheus:prometheus 0750)

journald                         all daemon logs (journalctl -u prometheus)

Distribution packages rearrange this — Debian 12’s package, for example, points the data directory at a metrics subdirectory of /var/lib/prometheus and drops config fragments in its own places. Neither layout is wrong; mixing them on one host is. systemctl cat prometheus tells you which one you actually have.

The data directory, entry by entry

--storage.tsdb.path is the only filesystem flag that matters for data. Set it explicitly — the default is a relative data directory under the process working directory, which is a trap under a service manager (see the failure modes below). A healthy data directory looks like this:

/var/lib/prometheus/
├── lock                      # flock guard: one process per data dir
├── queries.active            # bookkeeping for running queries
├── wal/                      # write-ahead log, 128 MiB segments
│   ├── 00000042
│   ├── 00000043
│   └── checkpoint.00000041/  # compacted older WAL data
│       └── 00000000
├── chunks_head/              # memory-mapped chunks of the head block
│   ├── 00000007
│   └── 00000008
├── 01J4B2YVEQ9Z8K0M7N3P1R6T5W/   # a block: one span of history
│   ├── chunks/000001
│   ├── index
│   ├── meta.json             # time range, compaction level, stats
│   └── tombstones            # deletion markers for this block
└── 01J4B93AF2QW7E1R8T6Y5U4I3O/
    └── ...

• lock — a file the process holds an flock on. A second Prometheus pointed at the same directory exits immediately with opening storage failed: lock DB directory. This file is the whole mechanism; there is no central registry.
• wal/ — every scraped sample is appended here before anything else. After an unclean stop, startup replays the WAL to rebuild the in-memory head; that replay is why a large instance can take minutes to become ready. Segments fill to 128 MiB and roll; checkpoint.* directories are the compacted leftovers of truncation.
• chunks_head/ — compressed sample chunks for the head block (roughly the most recent two to three hours), memory-mapped into the process. This is where recent queries are served from disk-wise.
• Block directories — named by ULID, each holding one immutable span of time. meta.json records the range and the compaction level; tombstones records series deleted via the admin API without rewriting the data. Blocks are written atomically: built in a tmp sibling, then renamed into place.

The configuration tree

/etc/prometheus/prometheus.yml is passed explicitly via --config.file; there is no compiled-in default path. It pulls in the rest by reference:

# /etc/prometheus/prometheus.yml -- skeleton showing the layout hooks
global:
  scrape_interval: 15s          # default for all jobs
  evaluation_interval: 15s      # how often rules are evaluated

rule_files:
  - /etc/prometheus/rules/*.yml # glob: only *.yml matches, nothing else

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  - job_name: node
    file_sd_configs:
      - files:
          - /etc/prometheus/filesd/node.yml   # re-read on change, no reload

Three layout consequences worth memorising:

1. The rule_files glob matches exactly what it says. A rule file renamed to .yaml or .yml.bak silently drops out of evaluation — no error, because a glob that matches nothing is not an error.
2. file_sd_configs directories are the supported way to feed targets from configuration management: drop a file, Prometheus re-reads it without a reload.
3. Consoles only work if the unit passes --web.console.templates=/etc/prometheus/consoles and --web.console.libraries=/etc/prometheus/console_libraries pointing at the real directories.

Logs live in the journal

Prometheus writes no log file. Everything goes to standard output, which systemd captures into the journal:

# READ-ONLY -- recent daemon logs
journalctl -u prometheus -n 30 --no-pager
# Aug 13 09:14:01 mon-01 prometheus[812]: ts=2026-08-13T09:14:01.902Z caller=main.go:577 level=info msg="Starting Prometheus Server" mode=server version="(version=2.55.1, ...)"
# Aug 13 09:14:01 mon-01 prometheus[812]: ts=2026-08-13T09:14:01.915Z caller=main.go:628 level=info host=localhost storage.tsdb.path=/var/lib/prometheus
# Aug 13 09:14:02 mon-01 prometheus[812]: ts=2026-08-13T09:14:02.114Z caller=main.go:1042 level=info msg="Server is ready to receive web requests."

# READ-ONLY -- follow logs since the last restart
journalctl -u prometheus -f

The second line of that output is free validation: the daemon announces its storage.tsdb.path at startup. Verbosity is a flag (--log.level=debug); it changes on restart, not reload.

Giving the TSDB its own disk

The data directory is the only part of the layout that grows without bound until retention stops it. Put it on its own filesystem:

# CONFIGURATION -- one-time: dedicated LV, filesystem, mount
sudo lvcreate -L 500G -n prometheus vg0
sudo mkfs.xfs /dev/vg0/prometheus
sudo install -d -m 0755 /var/lib/prometheus
echo '/dev/mapper/vg0-prometheus /var/lib/prometheus xfs defaults,noatime 0 0' \
  | sudo tee -a /etc/fstab
sudo mount /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
sudo chmod 0750 /var/lib/prometheus

The reasons are operational, not aesthetic:

• Blast radius. A TSDB that outgrows its quota fills its own filesystem. sshd, journald and every other service on the host keep working. A TSDB on the root filesystem takes the whole host down with it when it fills /.
• Independent growth. lvextend plus xfs_growfs (or resize2fs for ext4) grows the volume online; Prometheus sees the new space immediately, no restart.
• Backup policy. Snapshots and rsync target one mount point with a known size, separate from the OS.

Use local ext4 or XFS. Do not put the TSDB on NFS or SMB: the upstream storage documentation is explicit that network filesystems do not provide the durability and locking semantics the TSDB expects, and corruption there is a when, not an if.

How to validate it

# READ-ONLY -- flags the running process actually uses
curl -s http://localhost:9090/api/v1/status/flags \
  | jq -r '.data["config.file"], .data["storage.tsdb.path"]'
# /etc/prometheus/prometheus.yml
# /var/lib/prometheus

# READ-ONLY -- the data directory is its own mount, with headroom
findmnt /var/lib/prometheus
df -h /var/lib/prometheus

# READ-ONLY -- block inventory as Prometheus sees it on disk
sudo promtool tsdb list /var/lib/prometheus | head -5

# READ-ONLY -- rule files the glob actually loads
promtool check config /etc/prometheus/prometheus.yml

The first command is the authoritative one: it reports the running process, so it cannot be fooled by a stale unit file or a forgotten symlink.

How it can fail

1. Relative data path under systemd. Without --storage.tsdb.path, data lands in ./data under the unit’s working directory — /data for a default system unit. It works until somebody “cleans up” the strange directory at the root of the filesystem.
2. Two processes, one data dir. A forgotten container or a hand-started test instance already holds the lock; the real service exits with opening storage failed: lock DB directory.
3. TSDB on the root filesystem. Retention misjudged, / fills, and the monitoring outage becomes a host outage: logs stop, package installs fail, shells behave oddly. The node_filesystem alert fires into a platform that can no longer tell anyone.
4. Network filesystem for the TSDB. NFS locking and mmap semantics do not match the TSDB’s expectations; symptoms are stalled starts, invalid magic number errors after failover, and corruption that only promtool tsdb forensics will see.
5. Rule file renamed out of the glob. alerts.yml becomes alerts.yml.disabled for a test and never comes back. No error anywhere; the alert simply never fires again. The discovery happens during the incident it was meant to catch.
6. Live copy as backup. Tar of a running data directory captures blocks mid-compaction; the restore fails with inconsistent chunks. Use the snapshot API or stop the service before copying.

How to troubleshoot it

Path problems announce themselves at startup, so the order is:

1. Read the log first. journalctl -u prometheus -e. The error names the path: lock DB directory, permission denied, no space left on device each point at a different fix.
2. Confirm the flags in effect. /api/v1/status/flags if the server is up; the unit file via systemctl cat prometheus if it is not.
3. Walk the path. namei -l /var/lib/prometheus/wal shows ownership and mode of every component — the classic cause of “permission denied” is one wrong directory in the chain.
4. Check the filesystem. findmnt and df -h: is the dedicated volume actually mounted, or has the TSDB been writing into the empty mount-point directory on / since the last boot? That last one is embarrassingly common and worth checking early.
5. Inspect the TSDB. sudo promtool tsdb list /var/lib/prometheus for block inventory; against a live, locked data directory, work on a snapshot instead.

Security implications

• prometheus.yml routinely contains credentials — basic_auth passwords, bearer tokens for remote write. 0640 root:prometheus keeps them out of every local user’s reach; lesson 06 covers the model in full.
• The data directory is a record of your infrastructure’s behaviour — 0750 prometheus:prometheus and nothing wider.
• nodev and nosuid are reasonable mount options for the TSDB filesystem; nothing legitimate is executed from it, so noexec is defensible too.
• On RHEL 9 with SELinux enforcing, a data directory at a non-standard path needs a file-context rule; check ls -lZ and ausearch -m avc -ts recent before blaming modes.

Performance implications

The data directory is where Prometheus lives or dies. Local SSD class storage for anything beyond a lab; the WAL is append-mostly, blocks are read via mmap and the page cache, and compaction periodically rewrites gigabytes. noatime removes pointless metadata writes. RAM matters because the kernel caches the blocks your dashboards read — a host with plenty of free memory answers range queries from cache instead of disk.

Production guidance

• Set --storage.tsdb.path and --config.file explicitly in the unit, always. Never rely on defaults you have not read.
• Dedicate a filesystem to the TSDB; alert on its usage and on the mount existing at all.
• Keep the whole config tree in version control; deploy /etc/prometheus from configuration management, not by hand.
• Snapshot before backup: POST /api/v1/admin/tsdb/snapshot (with --web.enable-admin-api) produces a consistent copy via hard links, which is cheap on the same filesystem.
• Verify the layout from the running process (/api/v1/status/flags) in your post-change checklist.

Verification

You should now be able to answer:

• What does each of wal/, chunks_head/, a ULID block directory, lock, and queries.active do?
• Why is the default value of --storage.tsdb.path dangerous under systemd?
• Why does this course give the TSDB its own filesystem, and which filesystems are off-limits?
• A rule file stops evaluating without any error in the log. What is the first thing you check?

Quiz