Registry monitoring and alerting — the metrics that matter

A registry that is not monitored is a registry that fails silently. The deploys that pull from the registry fail loudly; the registry that serves them fails only when the deploys time out. The monitoring discipline is to expose the four metrics that predict the deploy failure before the deploy fails, and to alert on the thresholds that distinguish a regional outage from a partial degradation.

flowchart LR
    A["registry metrics"] --> B["pull latency p99"]
    A --> C["pull error rate"]
    A --> D["storage used"]
    A --> E["replication lag"]
    B --> F["SLO: p99 under 2s"]
    C --> G["SLO: error rate under 0.1 percent"]
    D --> H["alert: 80 percent full"]
    E --> I["SLO: lag under 60s"]
    F --> J["dashboard"]
    G --> J
    H --> J
    I --> J
    J --> K["on-call alert"]

The four production metrics

The four metrics every registry is monitored on:

• Pull latency. The end-to-end time from the kubelet issuing the pull request to the registry returning the last blob byte. The metric is a histogram; the alert is on the p99 (or p95, depending on the team’s tolerance). A p99 that climbs from two seconds to ten seconds is a sign of a registry under stress; a p99 that climbs to sixty seconds is a sign of a partial degradation.
• Pull error rate. The fraction of pull requests that return 5xx (server errors) over a window. A baseline rate of 0.01% (one in ten thousand) is normal; a rate of 1% is a sign of a regional degradation; a rate of 50% is a sign of a regional outage.
• Storage used. The fraction of the registry’s storage capacity that is in use. The metric is a gauge; the alert is on the percentage full. A registry that is 80% full is a registry whose garbage collection policy is overdue; a registry that is 95% full is a registry that may start rejecting pushes.
• Replication lag. The elapsed time between a push to the primary and the replica’s confirmation that the artifact is current. The metric is a gauge; the alert is on the seconds of lag. A lag that climbs above the RPO design constraint is a sign that the replication policy is failing.

Alert thresholds and SLOs

The alert thresholds are derived from the SLOs. The SLOs are written before the incident, not after:

• SLO: pull latency p99 under two seconds. Measured over a five-minute window. The alert threshold is p99 over four seconds for three consecutive five-minute windows (a sustained degradation, not a transient spike).
• SLO: pull error rate under 0.1%. Measured over a five-minute window. The alert threshold is over 1% for one five-minute window (a sustained outage threshold) or over 5% for one minute (an emergency threshold).
• SLO: replication lag under sixty seconds. Measured as a gauge. The alert threshold is over five minutes for one window (the replica is no longer meeting the RPO).
• Alert: storage used over 80%. A capacity alert, not an SLO. The threshold is 80% for warning, 95% for critical.

# Prometheus alert rules (illustrative)
groups:
  - name: registry-slos
    rules:
      - alert: RegistryPullLatencyHigh
        expr: histogram_quantile(0.99, sum(rate(registry_pull_duration_seconds_bucket[5m])) by (le)) > 4
        for: 15m
      - alert: RegistryPullErrorRateHigh
        expr: sum(rate(registry_pull_requests_total{status=~"5.."}[5m])) / sum(rate(registry_pull_requests_total[5m])) > 0.01
        for: 5m
      - alert: RegistryReplicationLagHigh
        expr: registry_replication_lag_seconds > 300
        for: 5m
      - alert: RegistryStorageNearFull
        expr: registry_storage_used_bytes / registry_storage_capacity_bytes > 0.8
        for: 30m

The dashboard

The dashboard exposes the four metrics, the SLO burn rate, and the per-cluster breakdown. The composition:

• Top row: the four metrics as single-stat panels with the SLO threshold as a red line.
• Middle row: the per-cluster pull latency p99 over the last six hours, to spot the cluster that is degrading first.
• Bottom row: the SLO burn rate (the fraction of the error budget consumed in the last 24 hours), to spot a slow degradation that has not yet tripped an alert.

The dashboard is the on-call engineer’s first stop. The alert fires into the runbook; the dashboard tells the engineer which scope (Part XCIX-01) the failure belongs to and whether the replica (Part XCIX-03) is current.

Mapping alerts to the runbook

Every alert fires into a runbook entry. The mapping:

• RegistryPullLatencyHigh → Part XCIX-01 (the registry failure scenario) — check the scope of the failure.
• RegistryPullErrorRateHigh → Part XCIX-02 (the impact on deployment) — confirm the tier of degradation.
• RegistryReplicationLagHigh → Part XCIX-03 (the replication strategy) — promote the replica if the lag exceeds the RTO.
• RegistryStorageNearFull → Part XCVII-05 (the registry recovery) — schedule garbage collection before the registry starts rejecting pushes.

The mapping is written in the alert’s runbook_url annotation; the alert fires with a link to the runbook entry. The on-call engineer opens the runbook, follows the steps, and confirms the recovery in the dashboard.

Production discipline

1. The four production metrics are pull latency, pull error rate, storage used, replication lag. A registry that is not monitored on all four is a registry with blind spots.
2. Alert thresholds are derived from SLOs. The SLOs are written before the incident; the alerts are configured from the SLOs.
3. The dashboard is the on-call engineer’s first stop. The alert fires the runbook; the dashboard tells the engineer which scope the failure is.
4. Every alert has a runbook URL. The on-call engineer follows the link, not the alert text.
5. Alerts are windowed, not lifetime. A five-minute window catches the degradation that a lifetime average hides.

Cross-course references

• Git, CI/CD & GitOps — Part XCIX-01 (The Registry Failure Scenario) is the runbook the latency and error-rate alerts fire into.
• Git, CI/CD & GitOps — Part XCIX-03 (Registry Replication) is the runbook the replication-lag alert fires into.
• Git, CI/CD & GitOps — Part XCVII-05 (Recovering the Artifact Registry) is the runbook the storage alert fires into.

Quiz