RPO and RTO

It is 03:14. A page fires for a backend you cannot see in Grafana. You open the URL and get a 502 from the Grafana frontend. The on-call Slack channel is already arguing about which dashboard to open first, and there are no dashboards to open. You need a number in your head for how long this will take to fix and how much historical data you will lose.

That number is what this lesson is about. RPO and RTO are the two budgets you set before the incident so the recovery procedure can be measured against them in real time. Without them, restore the stack is a vibe. With them, it is a measurable operation.

What RPO and RTO are

RPO — Recovery Point Objective — is the maximum acceptable gap between the last good state and the incident. For metrics it is usually expressed in seconds or samples. For Loki it is usually expressed in lines lost or in time. For Tempo it is usually expressed in traces lost, because traces are unbounded in cardinality.

RTO — Recovery Time Objective — is the wall-clock time from incident detection to the platform being usable again for investigations. It includes detection lag, on-call decision time, restore execution, and validation. It does not include the time to refill the lost data.

The two are independent. A platform can have a 30-second RPO and a 4-hour RTO — tight data, slow rebuild — or a 4-hour RPO and a 30-second RTO — loose data, fast hot spare. They trade against cost and complexity in opposite directions.

Why a sysadmin cares

The first time the targets are needed is the worst possible time to discover they are missing or wrong. A runbook that says restore from the nightly backup without a target invites the on-call engineer to declare the restore done the moment the service answers HTTP, before any data has been verified. A target is what lets the on-call engineer tell the incident commander whether the restore is on track or already failing.

It is also what makes the conversation with finance about storage cost defensible. A 30-day Loki retention with cross-region replication costs a known amount per terabyte; a 6-month retention with no replication costs less and admits a larger RPO. The target is the budget. The spend is the price of meeting it.

How it works: the per-tier approach

The observability stack is not one product. It is four storage shapes bolted together. Each has its own durability model and its own restore primitive.

+----------------------+--------------------+-----------------------+
| Signal               | Source of truth    | Restore primitive     |
+----------------------+--------------------+-----------------------+
| Prometheus metrics   | Local TSDB plus    | Snapshot restore or   |
|                      | remote_write       | replay from           |
|                      | receiver           | remote_write          |
+----------------------+--------------------+-----------------------+
| Loki logs            | Object store plus  | Recreate ingesters;   |
|                      | ingester WAL       | replay WAL; restore   |
|                      |                    | chunks from object    |
|                      |                    | store                 |
+----------------------+--------------------+-----------------------+
| Tempo traces         | Object store       | Empty ingesters       |
|                      | (block backend)    | flush; backend        |
|                      |                    | already has history   |
+----------------------+--------------------+-----------------------+
| Grafana              | SQLite or          | Restore grafana.db    |
|                      | Postgres plus      | plus provisioning     |
|                      | provisioning dir   | directory             |
+----------------------+--------------------+-----------------------+

A single-tier outage — one Prometheus host, one Loki ingester — is short. A platform-tier outage — the whole stack, or the object store — is long. The targets must distinguish.

Under the hood

The budgets come from how those layers combine, not from any single layer in isolation.

How to set the targets

Start from the product question: when a customer-facing incident fires at 03:00, how much history do you need to investigate, and how long can you wait for it?

1. Prometheus metrics. Alert rules and SLI panels need the shortest RPO. If paging depends on a 5-minute burn-rate window, the platform cannot afford to be blind for 30 minutes. A reasonable Prometheus target is 5 minutes RPO — driven by the remote_write queue — 15 minutes RTO for a single-instance rebuild, and 4 hours RTO for a multi-zone cluster rebuild.
2. Loki logs. Operational debugging needs the recent window. A reasonable Loki target is 5 minutes RPO, 30 minutes RTO for an ingester rebuild, 8 hours RTO for a full-stack rebuild after object-store loss.
3. Tempo traces. Trace data is sampled. A reasonable Tempo target is 10 minutes RPO for the head — un-sampled tail — with loss of older traces during a backend outage being a known acceptable cost. RTO matches Loki because the block backend is shared.
4. Grafana. Dashboards and alert rules can be rebuilt from provisioning. A reasonable Grafana target is 1 hour RPO — backup frequency — and 30 minutes RTO from a database restore.

How to validate it

The targets are only useful if they can be measured. Two drills, run quarterly:

# 1. Prove the restore time on a single Prometheus host.
#    Snapshot, delete the data directory, restore, time the gap.
date +%s && promtool tsdb snapshot /var/lib/prometheus/data && \
  rm -rf /var/lib/prometheus/data && \
  systemctl start prometheus && \
  sleep 90 && curl -sf http://prom:9090/-/ready && date +%s

# 2. Prove the Loki ingester rebuild.
#    Stop one ingester, confirm distributors reroute, time the gap.
kubectl -n loki delete pod loki-ingester-0 --grace-period=0
date +%s
kubectl -n loki wait --for=condition=ready pod -l app=loki-ingester
date +%s

The two timestamps are your measured RTO for that drill. Anything that is more than 50 percent over the target is a finding that needs work, not a finding that needs optimism.

How it can fail

• Targets written once and never re-validated. The platform changes; the runbook does not. The 30-minute RTO quietly becomes 90 minutes because a new rule evaluation pass added 30 minutes to startup.
• Per-tier targets hidden by a single DR objective line in a slide deck. When everyone believes the platform has a 1-hour RTO, the alert-rule metric layer that actually needs 5 minutes of RPO is exposed.
• A drill that succeeds in staging but fails in production because production uses a different backup transport. The targets were measured against the wrong system.
• The remote_write receiver treated as canonical without a RPO and RTO of its own. If the receiver has a 24-hour recovery point and 4-hour recovery time, that is the real RPO and RTO of the metrics tier — not the local Prometheus instance.
• Targets defined for the happy path only. A single-instance outage has a different RTO than a full-cluster outage; if the target is set for the easy case, the hard case overruns without anyone noticing.

How to troubleshoot it

The diagnostic order:

1. Was the platform healthy before the change? If no, the incident was not a DR test; it was an unmitigated outage.
2. What does the runbook say the targets are? If the runbook is silent, the targets are not real.
3. What does the drill report from last quarter say? If the last drill overran, the platform is already past the target.
4. Form a hypothesis: the platform grew, the targets did not.
5. Test the hypothesis by running the drill against the current platform.
6. Update both the targets and the runbook with the measured number.

Security implications

Cross-region replication adds attack surface — a second bucket, a second set of credentials, a second KMS key. Both copies must be audited under the same IAM policy. A bucket replicated to a less-restricted destination is a compliance failure, not a DR improvement.

Restoring a database backup must use the same secret material as production. Restoring Grafana with a stale admin password forces a manual rotation step that the runbook tends to forget.

Performance implications

Per-tier targets are a workload allocation. A 5-minute Prometheus RPO implies the remote_write receiver is sized to absorb at least 10 minutes of write backlog; otherwise a 10-minute outage overruns the target. A 30-minute Loki RTO implies the ingesters can rebuild that fast on the available compute; if the compute is sized for steady state, the cold rebuild stalls.

Production guidance

• Write the targets into the same git repository as the runbooks. Drift between the slide and the runbook is the failure shape.
• Schedule the quarterly drill on the same calendar entry as the post-mortem review. Both fall through the same cracks when separated.
• Treat the RPO and RTO document as code. It must be re-validated every time the platform changes shape.

Verification

You should now be able to answer:

• What does RPO measure for an observability platform, in a way that is distinct from RTO?
• Why is a single blanket target worse than per-tier targets for metrics, logs, traces, and dashboards?
• How do you prove a target is still honest after the platform has changed?
• What is the cheapest, most repeatable validation step?

Quiz