Communication and customer notifications — the social contract of change

Communication is the contract between the change and the people who depend on the cluster. The discipline is: pre-announce, in-flight, post-change, customer-facing, internal, escalation. A change that arrives without communication is a change that has decided the audience does not need to know.

The three communication phases

A change’s communication runs in three phases:

1. Pre-announce. Sent before the window opens. The audience knows the change is coming, the expected impact, the rollback path, and the close-out.
2. In-flight. Sent during the window. The audience knows the change has started, any intermediate state, and the escalation path if it fails.
3. Post-change. Sent after the window closes. The audience knows the change succeeded, the actual impact, and the follow-up actions.

sequenceDiagram
    participant Operator
    participant Internal
    participant Customer
    participant OnCall
    Operator->>Internal: Pre-announce
    Operator->>Customer: Pre-announce
    Operator->>OnCall: Window opens
    OnCall->>Operator: Validation passes
    Operator->>Internal: Post-change
    Operator->>Customer: Post-change
    Operator->>OnCall: Window closes

The audience is multiple: the on-call, the customer success team, the downstream consumers, the security team, and the customer-facing users. Each audience gets a different channel, a different timing, and a different level of detail.

The audience matrix

The matrix is the discipline. A change that is communicated to the on-call but not the customer is a change that has decided the customer does not need to know.

The pre-announce

The pre-announce is the contract. It is sent before the window opens. It contains:

• The change’s summary.
• The expected impact.
• The expected duration.
• The rollback path.
• The escalation contact.

The pre-announce is sent on the customer’s channel: the status page, the email, the customer-facing interface. The internal audience may receive a more detailed version; the customer-facing audience receives the version that is appropriate to their relationship with the cluster.

The in-flight notification

The in-flight notification is sent during the window. It contains:

• The change has started.
• Any intermediate state.
• The next checkpoint.
• The escalation contact.

The in-flight notification is the moment when the customer trusts the team. The customer that has been notified knows the change is in progress; the customer that has not been notified discovers the change from the synthetic traffic.

The post-change notification

The post-change notification is sent after the window closes. It contains:

• The change succeeded.
• The actual impact (vs the expected impact).
• The follow-up actions.
• The post-incident review schedule.

The post-change notification is the moment when the customer trusts the team. The customer that has been notified knows the change is complete; the customer that has not been notified discovers the change from the alert.

Failure modes

The worst production failure is missing communication. Common modes:

• Late notification. The customer is told after the change has started. The customer cannot prepare.
• Wrong audience. The on-call is told but the customer is not. The customer discovers the change from the alert.
• No rollback notification. The customer is told the change succeeded but not that the rollback was triggered. The customer does not know the cluster is in a different state.
• Lying in the notification. The notification says the change succeeded when it failed. The customer trusts the notification; the trust is broken.

The discipline is to communicate. The cost of a redundant notification is a small annoyance; the cost of a missing notification is a customer incident.

Production discipline

A communication plan is the cluster’s contract with the people who depend on it. The discipline is the same scale-free: every change gets a pre-announce, an in-flight notification, and a post-change notification. The audience matrix is the rule. The deviation is the failure.

• Pre-announce before the window opens. The customer can prepare.
• In-flight during the window. The customer knows the change is in progress.
• Post-change after the window closes. The customer knows the change succeeded.
• Use the audience matrix. Each audience gets the channel that matches their relationship with the cluster.

Quiz