The architecture decision record and its as-built capture

Intermediate⏱ ~17 minceph

What you'll learn

Structure a storage architecture decision record
Capture the as-built cluster state as evidence
Define the triggers that reopen the decision
Review the record against the running cluster

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The decision is made once by people with the context and lived with for years by people without it.

What the record contains

Section	Content	Example
Requirement	capacity now and in three years, IOPS, latency, RPO, RTO	400 TiB now, 900 TiB by 2029
Failure tolerance	what must survive, stated as an event	one rack, one site
Alternatives	each option with why it was rejected	managed object service, rejected on egress cost
Chosen architecture	pools, rules, profiles, device classes	4+2 on hdd, size 3 on nvme
Cost	capital and recurring, both sites	with the arithmetic, not the total
Accepted risks	what this design does not cover	correlated firmware fault across a batch
Invalidation triggers	what would make this wrong	see below

Capturing the as-built state

OUT=/secure/arch/$(date +%Y-%m-%d)
mkdir -p "$OUT"
ceph osd pool ls detail            > "$OUT/pools.txt"
ceph osd crush rule dump           > "$OUT/crush-rules.json"
ceph osd crush tree                > "$OUT/crush-tree.txt"
ceph osd crush class ls            > "$OUT/device-classes.txt"
ceph osd erasure-code-profile ls   > "$OUT/ec-profiles.txt"
ceph df detail                     > "$OUT/df.txt"
ceph versions                      > "$OUT/versions.txt"
ceph orch ls                       > "$OUT/services.txt"

for p in $(ceph osd erasure-code-profile ls); do
  echo "== $p"; ceph osd erasure-code-profile get "$p"
done > "$OUT/ec-profiles-detail.txt"

The prose says what was intended. The capture says what is running. The
diff between two captures is the only reliable record of what changed.

Invalidation triggers

Trigger	Why it reopens the decision
Capacity passes the three-year projection early	the sizing assumption failed
Host count crosses k plus m plus one	erasure coding becomes self-healing
A second site is funded	replication topology becomes available
Measured latency stops meeting the stated target	the workload changed or the cluster did
The rejected alternative changes price or capability	the rejection reason may no longer hold
The team that operates it changes size	the operational capacity assumption failed

Reviewing

diff -u /secure/arch/2026-02-01/pools.txt /secure/arch/$(date +%Y-%m-%d)/pools.txt
diff -u /secure/arch/2026-02-01/crush-rules.json \
        /secure/arch/$(date +%Y-%m-%d)/crush-rules.json

cat >> /secure/arch/decisions.md <<'EOF'
2026-08-18  reviewed. Capacity 61% of projection, on track.
            Host count now 8; ec42 is self-healing as designed.
            No trigger fired. Next review 2027-02.
EOF

Quiz

Knowledge check · 4 questions

Q1. Why does an architecture decision record need the rejected alternatives and their specific reasons?

Auditors require a documented comparison of options before capital expenditure, and the record is what satisfies that requirement at review time The reasons age differently from the decision, so recording them turns a yearly re-argument into a check of whether each reason still holds It documents which vendors were contacted, so the next procurement round can reuse those quotes rather than starting the tendering process again The rejected alternatives may need to be deployed later, so keeping their designs written down shortens the work if the chosen one has to be replaced
Q2. A scheduled machine capture of pools, rules, and profiles is more useful during an incident than the architecture prose.

True False
Q3. Review a two-year-old storage architecture decision.

A cluster was designed in 2024 for 400 TiB on 6 hosts with a 4+2 erasure-coded pool. It now holds 380 TiB across 9 hosts. Nobody has looked at the document since.
Q4. Name three events that should reopen a storage architecture decision.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Capture pools, CRUSH rules, erasure code profiles, device classes, and versions on a schedule and diff successive captures — the drift that diff exposes is what makes an incident confusing years after the design. Write the invalidation triggers into the record so the review has something specific to check rather than an opinion to re-form.

Cross-course references

Kubernetes: a cluster manifest in git and a cluster in production diverge without a reconciliation report
Linux: configuration drift is made of changes that each seemed too small to record