Containment and isolation — the security incident's first hour

When a workload is confirmed compromised, the first hour is spent stopping the spread rather than understanding it. A default-deny NetworkPolicy around the affected namespace, a cordon on the node it ran on, and a rotated ServiceAccount token cut the attacker’s reach in minutes, and each of them is reversible if the alert turns out to be wrong. Move too slowly and the credential is used elsewhere; move without care and you destroy the audit trail the investigation needs.

The containment

The containment is the impact limitation. The containment is the cluster’s first response to a security incident: stop the bleeding, limit the impact, prevent the spread.

flowchart TD
    A[Security incident] --> B{Containment}
    B --> C[Network policy]
    B --> D[Cordon]
    B --> E[Eviction]
    B --> F[Credential rotation]

The containment is the cluster’s impact limitation.

The network policy

The network policy is the cluster’s network isolation. The network policy denies traffic to and from the compromised Pod.

# Deny all traffic to the compromised Pod
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-compromised
  namespace: prod
spec:
  podSelector:
    matchLabels:
      compromised: "true"
  policyTypes:
  - Ingress
  - Egress

The network policy is the cluster’s network isolation.

The cordon

The cordon is the cluster’s node isolation. The cordon prevents new Pods from being scheduled on the node; the eviction removes the running Pods.

# Substitute your own values before running:
NODE=worker-03

# Cordon the node
kubectl cordon "$NODE"

# Drain the node
kubectl drain "$NODE" --ignore-daemonsets --force

The cordon is the cluster’s node isolation.

The credential rotation

The credential rotation is the cluster’s secret hygiene. The credential rotation invalidates the leaked credential.

# Substitute your own values before running:
NS=prod
SA=billing-sa
SA_TOKEN_SECRET=billing-sa-token-4x7qz
CERT=apiserver

# Rotate the ServiceAccount token
kubectl delete secret "$SA_TOKEN_SECRET" -n "$NS"

# Rotate the ServiceAccount
kubectl delete serviceaccount "$SA" -n "$NS"
kubectl apply -f serviceaccount.yaml

# Rotate the certificate (run on a control-plane node)
kubeadm certs renew "$CERT"

The credential rotation is the cluster’s secret hygiene.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
NS=prod

# 1. Check the network policies
kubectl get networkpolicy -A

# 2. Check the cordoned nodes
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.unschedulable}{"\n"}{end}'

# 3. Check the rotated credentials
kubectl get secrets -n "$NS"

# 4. Check the audit logs
kubectl logs -n kube-system -l component=kube-apiserver --tail=200 | grep -i audit

# 5. Check the events
kubectl get events -A --field-selector type=Warning

The diagnostic is the network policies, the cordoned nodes, the rotated credentials, the audit logs, and the events.

The remediation

The remediation depends on the type:

# Substitute your own values before running:
NODE=worker-03
NS=prod
SA_TOKEN_SECRET=billing-sa-token-4x7qz

# Option 1: Apply a network policy
kubectl apply -f deny-compromised.yaml

# Option 2: Cordon the node
kubectl cordon "$NODE"
kubectl drain "$NODE" --ignore-daemonsets --force

# Option 3: Rotate the credential
kubectl delete secret "$SA_TOKEN_SECRET" -n "$NS"
kubectl apply -f serviceaccount.yaml

# Option 4: Notify the security team
# (escalation-specific)

The remediation is the containment.

Production discipline

A security incident containment is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the security layer, identify the cause, apply the remediation. The security is the cluster’s protection; the remediation is the containment.

• Contain the impact. The containment is the cluster’s first response.
• Apply a network policy. The network policy is the cluster’s network isolation.
• Cordon the node. The cordon is the cluster’s node isolation.
• Rotate the credential. The credential rotation is the cluster’s secret hygiene.

Quiz