Detection and triage — the security incident's first hour

A security incident arrives as scattered signals: a Falco alert on an unexpected exec, an audit entry from a ServiceAccount that should never have written anything, an egress connection to an address nobody recognises. None of them states the scope, and containing the wrong thing burns the window in which the blast radius can still be bounded. This lesson covers turning those signals into a classified incident — what kind of violation, how urgent, and how far it reaches — before any containment action is taken.

The security incident

A security incident is a violation of the cluster’s security policy. The incident may be:

• Compromised credential. A token or certificate is leaked.
• Malicious deployment. A deployment with malicious intent is shipped.
• Privilege escalation. A workload has gained more privileges than it should.
• Data exfiltration. Data is being sent to an external destination.

flowchart TD
    A[Security incident] --> B{Type?}
    B -->|Credential| C[Rotate the credential]
    B -->|Deployment| D[Contain the deployment]
    B -->|Privilege| E[Revoke the privilege]
    B -->|Exfiltration| F[Block the destination]

The security incident is the cluster’s violation.

The detection

The detection is the cluster’s signal. The detection is performed by:

• Falco. Runtime detection of anomalous behaviour.
• Trivy. Static analysis of vulnerabilities.
• Audit logs. The API server’s audit logs.
• Network policies. The NetworkPolicy’s enforcement.

# Check the Falco alerts
kubectl logs -n falco -l app=falco --tail=200

# Check the audit logs
kubectl logs -n kube-system -l component=kube-apiserver --tail=200 | grep -i audit

# Check the network policies
kubectl get networkpolicy -A

The detection is the cluster’s signal.

The triage

The triage is the incident’s classification. The triage identifies the type, the severity, and the scope.

# Step 1: identify the type
kubectl get events -A --field-selector type=Warning

# Step 2: identify the severity
# (severity: critical, high, medium, low)

# Step 3: identify the scope
# (scope: cluster-wide, namespace-wide, workload-wide)

The triage is the incident’s classification.

The severity

The severity is the incident’s classification. The severity is the urgency:

• Critical. The cluster is compromised; the data is exposed; the impact is immediate.
• High. A vulnerability is exploited; the cluster is at risk; the impact is imminent.
• Medium. A vulnerability is discovered; the cluster is at risk; the impact is possible.
• Low. A vulnerability is reported; the cluster is at risk; the impact is unlikely.

The severity is the incident’s urgency.

The diagnostic

The canonical diagnostic:

# 1. Check the Falco alerts
kubectl logs -n falco -l app=falco --tail=200

# 2. Check the audit logs
kubectl logs -n kube-system -l component=kube-apiserver --tail=200 | grep -i audit

# 3. Check the network policies
kubectl get networkpolicy -A

# 4. Check the workloads
kubectl get pods -A -o wide

# 5. Check the events
kubectl get events -A --sort-by=.lastTimestamp

The diagnostic is the Falco alerts, the audit logs, the network policies, the workloads, and the events.

The remediation

The remediation depends on the type:

# Substitute your own values before running:
NS=production               # namespace the incident is in
DEPLOYMENT=checkout-api     # the compromised workload
SECRET=checkout-db-creds    # the credential it used

# Option 1: Contain the deployment
kubectl delete deployment "$DEPLOYMENT" -n "$NS"

# Option 2: Rotate the credential
kubectl delete secret "$SECRET" -n "$NS"

# Option 3: Block the destination
kubectl apply -f deny-egress.yaml

# Option 4: Notify the security team
# (escalation-specific)

The remediation is the containment.

Production discipline

A security incident is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the security layer, identify the cause, apply the remediation. The security is the cluster’s protection; the remediation is the containment.

• Check the Falco alerts. Falco is the runtime detection — kubectl logs -n falco -l app=falco is the first read.
• Check the audit logs. The audit logs are the API server’s history — an entry from a ServiceAccount that should never have written anything is the signal.
• Check the NetworkPolicy. The NetworkPolicy is the network’s enforcement; kubectl get networkpolicy -A shows whether the egress an exfiltration would use was ever constrained.

Quiz