Systematic network troubleshooting — the methodology

The systematic network troubleshooting methodology starts from the application and moves down the stack: application, Pod, CNI, Service, EndpointSlice, DNS, NetworkPolicy, kube-proxy. This lesson walks the methodology, the diagnostic tools, and the operational discipline.

The systematic methodology

The systematic methodology:

flowchart TD
    A[Application] --> B[Pod]
    B --> C[CNI]
    C --> D[Node network]
    D --> E[Cluster network]
    E --> F[External network]
    A1[Application failure?] -->|Yes| G[Check logs]
    B1[Pod has IP?] -->|No| H[Check CNI]
    C1[CNI configured?] -->|No| I[Check CNI logs]
    D1[Node routes correct?] -->|No| J[Check routes]
    E1[Cluster routes correct?] -->|No| K[Check BGP/overlay]
    F1[External access OK?] -->|No| L[Check LB/Ingress]

The methodology starts from the application and moves down the stack. The fix is to investigate the first layer that fails.

The seven layers

The seven layers of the network path:

1. Application: the application itself.
2. Pod: the Pod’s network namespace.
3. CNI: the CNI plugin’s configuration.
4. Service: the Service’s configuration.
5. EndpointSlice: the EndpointSlice’s backends.
6. DNS: the cluster’s DNS service.
7. NetworkPolicy: the NetworkPolicy’s rules.
8. kube-proxy: the kube-proxy’s data plane.

Each layer has a failure mode. The methodology isolates the layer.

The diagnostic tools

The diagnostic tools:

The tools are the operator’s primary tools for network troubleshooting.

The layer-by-layer checklist

The layer-by-layer checklist:

flowchart LR
    A[Application] --> B{Pod Ready?}
    B -->|No| C[Check probes]
    B -->|Yes| D{Logs OK?}
    D -->|No| E[Check app]
    D -->|Yes| F{Pod has IP?}
    F -->|No| G[Check CNI]
    F -->|Yes| H{NetworkPolicy?}
    H -->|Yes| I[Check rules]
    H -->|No| J{Service?}
    J -->|No| K[Check DNS]
    J -->|Yes| L[Check endpoints]
    L --> M[Check kube-proxy]
    M --> N[Check routes]

The checklist is the operator’s guide. The cluster operator must follow the checklist.

The failure modes

The seven layers’ failure modes:

• Application: the application is buggy, the configuration is wrong, the resource is exhausted.
• Pod: the CNI is not configured, the namespace is wrong, the readiness probe is failing.
• CNI: the CNI plugin is not running, the conflist is wrong, the IPAM is exhausted.
• Service: the selector is wrong, the port is mismatched, the EndpointSlice is empty.
• EndpointSlice: the Pods are not Ready, the labels are wrong, the controller is not running.
• DNS: the CoreDNS is not running, the records are stale, the upstream is unreachable.
• NetworkPolicy: the policy is wrong, the selector is wrong, the CNI does not enforce.
• kube-proxy: the kube-proxy is not running, the dataplane is wrong, the rules are corrupted.

The operational discipline

The network troubleshooting’s operational discipline:

• Document the methodology. The methodology is the cluster’s operational reference.
• Use the diagnostic tools. The tools are the operator’s primary tools.
• Follow the checklist. The checklist is the operator’s guide.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the methodology. The methodology is the team’s tools.
• Plan the methodology’s evolution. The methodology can be updated over time.

Quiz

Production discipline

• The methodology is systematic, not random. The cluster operator must follow the methodology.
• Document the methodology. The methodology is the cluster’s operational reference.
• Use the diagnostic tools. The tools are the operator’s primary tools.
• Follow the checklist. The checklist is the operator’s guide.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the methodology. The methodology is the team’s tools.
• Plan the methodology’s evolution. The methodology can be updated over time.
• Set up alerts on the network metrics. The alerts are the leading indicator of outages.