The CD loop closing — observability and feedback

A Kubernetes delivery pipeline that ends at “merge to GitOps branch” ships changes but cannot learn from production. The loop only closes when runtime signals - sync status, application health, rollout events, alert noise - drive the next commit, issue, or page. Without that arrow back to the source, the pipeline gets slower the longer it runs because every bad rollout lingers undetected until a user notices.

The runtime signals

Four signals tell the team whether the last change was right:

flowchart LR
    A["OCI manifest bundle"] --> B["Controller sync"]
    B --> C["Live cluster state"]
    C --> D["Sync status and OutOfSync"]
    C --> E["Application health: Progressing, Healthy, Degraded"]
    C --> F["Rollout events: replicas, ready, updated"]
    C --> G["SLO telemetry: latency, errors, saturation"]
    D --> H["Notification controller"]
    E --> H
    F --> H
    G --> H
    H -->|"on violation"| I["Open issue or page on-call"]
    H -->|"on SLO breach"| J["Trigger revert PR or auto-rollback"]
    I --> K["Next commit to Git"]
    J --> K
    K --> A

Each signal answers a different question:

• Sync status answers: is the cluster’s live state equal to the declared state? OutOfSync means drift; the controller’s reconcile tick will resolve it unless self-heal is off.
• Application health answers: are the workloads progressing, healthy, or degraded? Progressing is the normal rollout state; Degraded means the rollout failed.
• Rollout events answers: how many replicas are ready, how many are updated, how many are unavailable? A stalled rollout blocks all of them.
• SLO telemetry answers: are users experiencing this rollout as intended? Error rate, latency, and saturation metrics tell the team whether the change is right.

Notifications and the issue path

The notification controller subscribes to controller events and posts to Slack, opens issues in GitHub, pages on-call in PagerDuty, or fires webhooks. The events are typed:

triggers:
  - on-degraded: true
  - on-health-degraded: true
  - on-sync-failed: true
templates:
  - template: slack-rollout-failed
    slack:
      channel: "#deploys"
      message: "App {.app.metadata.name} degraded after sync. Rollout stalled."
subscribers:
  - slack:deploys
  - github-issues:deploys

The wiring is the difference between an event that a human notices and an event that drives an action. A Slack message nobody reads is noise. A Slack message that opens an issue, assigns an owner, and triggers a follow-up is feedback.

SLOs for rollout health

An SLO is a measurable target the team commits to. The SLOs that close the loop on Kubernetes delivery are:

• Time-to-detect bad rollouts. From rollout complete to SLO breach detected. Target: minutes, not hours.
• Rollout success rate. Percentage of rollouts that complete without an SLO breach. Target: high enough that the team trusts auto-promotion.
• Mean time to recovery. From SLO breach to revert merged and synced. Target: shorter than the time to detect, so the team always recovers before the breach is observable to users.
• Drift detection latency. From manual cluster change to OutOfSync alert. Target: shorter than the next sync window.

Each SLO has an error budget. When the budget is exhausted, the team pauses auto-promotion and audits the pipeline.

Closing the loop on SLO breach

The right pattern is not “alert and hope”. The right pattern is “alert and trigger”:

flowchart LR
    A["SLO breach detected"] --> B["Notification fires"]
    B --> C["PagerDuty pages on-call"]
    B --> D["GitHub issue opened"]
    D --> E["Workflow runs revert PR"]
    E --> F["Revert merged"]
    F --> G["Controller rolls back cluster"]
    G --> H["SLO recovers"]

The revert PR is automated: the workflow opens a PR that reverts the offending commit and assigns it to the on-call. The on-call merges the revert; the controller rolls the cluster back through the reconcile loop. The whole sequence runs without manual intervention unless the on-call decides the revert is wrong.

This is the loop closing: the runtime signal drove a commit, the commit was merged, the cluster rolled back, the SLO recovered. The next change inherits the lessons of this one.

Production discipline

1. Notifications drive actions, not channels. A Slack message that nobody reads is noise; an issue that opens a PR is feedback.
2. SLOs are measurable and budgeted. Without an error budget, an SLO is a slogan. With one, it is a contract.
3. Revert is automated, not manual. The on-call’s job is to confirm the revert, not to write it.
4. Drift alerts are routed to the team that owns the change. A drift alert to a generic channel is noise; one routed to the team that owns the offending resource is feedback.
5. The loop’s arrow is reviewed. The team reads the feedback path monthly and asks: did the signal drive the commit? If not, the path is broken.

Cross-course references

• Kubernetes for Production Sysadmins - Parts XI-XIV cover Argo CD notifications, Flux’s notification controller, and rollout health.
• Observability for Production Sysadmins - Parts XXIII-XXVI cover Prometheus, SLOs, and the alerting path the loop depends on.
• This course, Part LVII (Approvals) - covers the human gate that the loop’s revert PR must respect.
• This course, Part XLVII (PipelineGraph) - covers how the loop’s feedback path is a DAG edge, not a side channel.

Quiz