Container forensics — the security incident's investigation

A compromised Pod holds its evidence in memory and on a writable layer that vanishes the moment the container restarts — and its Deployment will restart it within seconds of any clumsy interference. Order of work matters more than tooling here: capture the running state first, reconstruct what the attacker’s credential did through the API server’s audit record, and only then decide what to shut down. This lesson covers what to collect from a live container, how to read the audit trail of a compromised ServiceAccount, and how to work back from what was touched to how the attacker got in.

The container forensics

The container forensics is the cluster’s investigation. The forensics identifies the entry point, the scope, and the impact.

flowchart TD
    A[Compromised Pod] --> B[Take a snapshot]
    B --> C[Audit logs]
    B --> D[File system]
    B --> E[Network]
    B --> F[Process list]

The forensics is the cluster’s investigation.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
POD=frontend-7d8f-abcde
NS=prod
CONTAINER=frontend

# 1. Take a snapshot of the compromised Pod
kubectl debug "$POD" -n "$NS" --image=busybox --target="$CONTAINER" -- sleep 3600

# 2. Check the audit logs
kubectl logs -n kube-system -l component=kube-apiserver --tail=200 | grep -i audit

# 3. Check the file system
kubectl exec -it "$POD" -n "$NS" -- ls -la /

# 4. Check the network
kubectl exec -it "$POD" -n "$NS" -- netstat -anp

# 5. Check the process list
kubectl exec -it "$POD" -n "$NS" -- ps aux

The diagnostic is the snapshot, the audit logs, the file system, the network, and the process list.

The audit logs

The audit logs are the API server’s history. The audit logs record every API call: who, what, when, where.

# Check the audit logs
kubectl logs -n kube-system -l component=kube-apiserver --tail=200 | grep -i audit

A real audit log:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "auditID": "abc-123",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/prod/pods",
  "verb": "create",
  "user": {
    "username": "system:serviceaccount:prod:compromised",
    "groups": ["system:authenticated"]
  },
  "sourceIPs": ["10.0.0.1"],
  "userAgent": "kubectl/v1.34.0",
  "objectRef": {
    "resource": "pods",
    "namespace": "prod",
    "name": "frontend-7d8f-abcde"
  },
  "responseStatus": {"metadata": {}, "code": 201},
  "requestReceivedTimestamp": "2026-08-16T04:23:01.123456Z",
  "stageTimestamp": "2026-08-16T04:23:01.234567Z"
}

The audit log is the cluster’s history.

The file system

The file system is the container’s state. The file system may contain the malicious binary.

# Substitute your own values before running:
POD=frontend-7d8f-abcde
NS=prod

# Check the file system
kubectl exec -it "$POD" -n "$NS" -- ls -la /

# Find the malicious binary
kubectl exec -it "$POD" -n "$NS" -- find / -name "*.sh" -mtime -1

The file system is the container’s state.

The network

The network is the container’s connections. The network may contain the exfiltration destination.

# Substitute your own values before running:
POD=frontend-7d8f-abcde
NS=prod

# Check the network
kubectl exec -it "$POD" -n "$NS" -- netstat -anp

# Check the DNS queries
kubectl exec -it "$POD" -n "$NS" -- cat /etc/resolv.conf
kubectl logs -n kube-system -l k8s-app=kube-dns | grep -i "$POD"

The network is the container’s connections.

The entry point

The entry point is the initial compromise. The entry point is the way the attacker gained access.

flowchart TD
    A[Entry point] --> B{Vulnerability?}
    B -->|CVE| C[Patch the CVE]
    B -->|Misconfig| D[Fix the misconfig]
    B -->|Credential| E[Rotate the credential]
    B -->|Supply chain| F[Audit the supply chain]

The entry point is the initial compromise.

The remediation

The remediation depends on the entry point:

# Substitute your own values before running:
NS=prod
DEPLOY=frontend
CONTAINER=frontend
PATCHED_IMAGE=registry.example.com/frontend:1.4.2

# Option 1: Patch the CVE
kubectl set image deployment "$DEPLOY" -n "$NS" "$CONTAINER"="$PATCHED_IMAGE"

# Option 2: Fix the misconfig
kubectl apply -f fixed-config.yaml

# Option 3: Rotate the credential
TOKEN_SECRET=compromised-token-x4k2p
kubectl delete secret "$TOKEN_SECRET" -n "$NS"

# Option 4: Audit the supply chain
# (supply chain-specific)

The remediation is the entry point identification.

Production discipline

A security incident investigation is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the security layer, identify the entry point, apply the remediation. The security is the cluster’s protection; the remediation is the entry point identification.

• Take a snapshot. The snapshot is the cluster’s evidence.
• Check the audit logs. The audit logs are the cluster’s history.
• Check the file system. The file system is the container’s state.
• Check the network. The network is the container’s connections.

Quiz