Network troubleshooting — the diagnostic workflow

Network troubleshooting in Kubernetes is a systematic approach from the application to the network. The diagnostic tools are kubectl exec, tcpdump, and nslookup. The failure modes are: application, Pod, CNI, Service, EndpointSlice, DNS, NetworkPolicy, kube-proxy. This lesson walks the troubleshooting approach, the tools, and the operational patterns.

The systematic troubleshooting approach

The systematic troubleshooting approach:

flowchart TD
    A[Application] --> B[Pod]
    B --> C[CNI]
    C --> D[Node network]
    D --> E[Cluster network]
    E --> F[External network]
    A1[Application failure?] -->|Yes| G[Check logs]
    B1[Pod has IP?] -->|No| H[Check CNI]
    C1[CNI configured?] -->|No| I[Check CNI logs]
    D1[Node routes correct?] -->|No| J[Check routes]
    E1[Cluster routes correct?] -->|No| K[Check BGP/overlay]
    F1[External access OK?] -->|No| L[Check LB/Ingress]

The systematic approach starts from the application and moves down the stack. The fix is to investigate the first layer that fails.

Layer 1: application

The application is the first layer. The application may be failing for non-network reasons.

# Substitute your own value before running:
POD=billing-7d9f6b58c4-mn2wq

kubectl logs "$POD"

The application’s logs show the application’s failure. The fix is to investigate the application’s logs.

The application may also be failing because of:

• Misconfiguration: the application is configured with the wrong host or port.
• Resource exhaustion: the application is out of memory or CPU.
• Bug: the application has a bug.

The application layer is the first layer to check. The fix is to investigate the application’s logs and configuration.

Layer 2: Pod

The Pod’s network is the second layer. The Pod’s network may be failing because of the CNI or the Pod’s configuration.

# Substitute your own value before running:
POD=billing-7d9f6b58c4-mn2wq

kubectl exec "$POD" -- ip addr show
kubectl exec "$POD" -- ip route

The Pod’s network interfaces and routes show the Pod’s network configuration. The fix is to investigate the Pod’s network.

The Pod’s network may also be failing because of:

• CNI failure: the CNI plugin is failing.
• NetworkPolicy: the NetworkPolicy is blocking the traffic.
• DNS: the DNS is failing.

The Pod layer is the second layer to check. A Pod with no address on eth0 never received one from the CNI’s IPAM; a Pod with an address but no default route cannot leave the node.

Layer 3: CNI

The CNI plugin is the third layer. The CNI plugin may be failing because of the configuration or the runtime.

# Substitute the DaemonSet label your CNI uses:
CNI_LABEL=calico-node

kubectl logs -n kube-system -l "k8s-app=$CNI_LABEL"

The CNI plugin’s logs show the CNI’s failure. The fix is to investigate the CNI’s logs.

The CNI plugin may also be failing because of:

• Configuration: the CNI plugin is configured incorrectly.
• Network: the CNI plugin’s network is failing.
• Runtime: the CNI plugin’s runtime is failing.

The CNI layer is the third layer to check. A CNI that fails at Pod creation surfaces as a FailedCreatePodSandBox event on the Pod before the logs are read.

Layer 4: Service

The Service is the fourth layer. The Service may be failing because of the selector or the EndpointSlice.

# Substitute your own value before running:
SVC=billing

kubectl describe svc "$SVC"
kubectl get endpointslices -l "kubernetes.io/service-name=$SVC"

The Service’s spec and the EndpointSlice show the Service’s configuration. The fix is to investigate the Service’s configuration.

The Service may also be failing because of:

• Selector mismatch: the Service’s selector does not match the Pods.
• Empty EndpointSlice: the EndpointSlice is empty.
• Port mismatch: the Service’s port does not match the Pod’s port.

The Service layer is the fourth layer to check. An empty EndpointSlice sends traffic nowhere; kube-proxy rejects the connection rather than letting it time out.

Layer 5: DNS

The DNS is the fifth layer. The DNS may be failing because of the CoreDNS or the Pod’s configuration.

# Substitute your own value before running:
POD=checkout-6b4c9d7f85-t4xn2

kubectl exec "$POD" -- nslookup billing.prod-app.svc.cluster.local

The DNS resolution shows the DNS’s status. The fix is to investigate the DNS’s configuration.

The DNS may also be failing because of:

• CoreDNS failure: the CoreDNS is failing.
• Wrong namespace: the DNS record is in the wrong namespace.
• Search path: the search path is incorrect.

The DNS layer is the fifth layer to check. A name that resolves as an FQDN but not as a short name is a search path or ndots problem, not a CoreDNS problem.

Layer 6: kube-proxy

The kube-proxy is the sixth layer. The kube-proxy may be failing because of the implementation or the configuration.

kubectl logs -n kube-system -l k8s-app=kube-proxy

The kube-proxy’s logs show the kube-proxy’s failure. The fix is to investigate the kube-proxy’s logs.

The kube-proxy may also be failing because of:

• Implementation: the kube-proxy implementation is failing.
• iptables: the iptables rules are failing.
• IPVS: the IPVS rules are failing.

The kube-proxy layer is the sixth layer to check. A stale rule set shows as traffic reaching a Pod that the EndpointSlice no longer lists.

Layer 7: NetworkPolicy

The NetworkPolicy is the seventh layer. The NetworkPolicy may be blocking the traffic.

kubectl get networkpolicy -A

The NetworkPolicies show the cluster’s network rules. The fix is to investigate the NetworkPolicy.

The NetworkPolicy may also be failing because of:

• Selector mismatch: the NetworkPolicy’s selector does not match the Pods.
• Blocking traffic: the NetworkPolicy is blocking the traffic.
• Missing NetworkPolicy: the NetworkPolicy is missing.

The NetworkPolicy layer is the seventh layer to check. The fix is to investigate the NetworkPolicy.

The diagnostic tools

The diagnostic tools:

The diagnostic tools are the operator’s primary tools for network troubleshooting. The fix is to use the right tool for the right layer.

The packet capture

The packet capture:

# Substitute your own value before running:
POD=billing-7d9f6b58c4-mn2wq

kubectl exec "$POD" -- tcpdump -i eth0 -w /tmp/capture.pcap

The packet capture is the operator’s primary tool for diagnosing the network. The fix is to capture the packets and analyze them.

The packet capture’s failure modes:

• tcpdump not installed: the Pod’s image does not include tcpdump. The fix is to use a debug sidecar.
• Permission denied: the Pod’s security context denies the packet capture. The fix is to grant the permission.

The packet capture is the operator’s last resort. The fix is to investigate the network systematically.

The operation patterns

The operation patterns:

• Use the systematic approach. The approach is: application, Pod, CNI, Service, DNS, kube-proxy, NetworkPolicy.
• Use the diagnostic tools. The tools are: kubectl exec, kubectl logs, kubectl describe, kubectl get, nslookup, tcpdump, curl, nc, mtr.
• Document the troubleshooting. The documentation is the operator’s reference.
• Audit the troubleshooting at every release. The audit catches the failures.
• Monitor the troubleshooting’s metrics. The metrics expose the cluster’s networking health.
• Test the troubleshooting in non-production. A staging cluster that mirrors production is the right place to test the troubleshooting.
• Document the troubleshooting’s design. The troubleshooting is the cluster’s operational reference.

Quiz

Production discipline

• Use the systematic approach. The approach is: application, Pod, CNI, Service, DNS, kube-proxy, NetworkPolicy.
• Use the diagnostic tools. The tools are: kubectl exec, kubectl logs, kubectl describe, kubectl get, nslookup, tcpdump, curl, nc, mtr.
• Document the troubleshooting. The documentation is the operator’s reference.
• Audit the troubleshooting at every release. The audit catches the failures.
• Monitor the troubleshooting’s metrics. The metrics expose the cluster’s networking health.
• Test the troubleshooting in non-production. A staging cluster that mirrors production is the right place to test the troubleshooting.
• Document the troubleshooting’s design. The troubleshooting is the cluster’s operational reference.