Ingress troubleshooting — diagnosing the cluster HTTP gateway

Ingress troubleshooting requires checking the controller, the Ingress resource, the backend Service, the TLS, and the metrics. This lesson walks the diagnostic flow and the operational discipline.

The diagnostic flow

The diagnostic flow for Ingress:

flowchart TD
    A[Ingress degraded] --> B{Controller running?}
    B -->|No| C[Restart controller]
    B -->|Yes| D{Ingress valid?}
    D -->|No| E[Fix Ingress]
    D -->|Yes| F{Backend ready?}
    F -->|No| G[Check endpoints]
    F -->|Yes| H{TLS valid?}
    H -->|No| I[Fix TLS]
    H -->|Yes| J[Check metrics]

The diagnostic flow is the operator’s guide. The cluster operator must understand the flow.

Step 1: Check the controller

The first step is to verify the controller is running:

kubectl get pods -n ingress-nginx

NAME                            READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-abc   1/1     Running   0          12d
ingress-nginx-controller-def   1/1     Running   0          12d

The Pods must be Running and Ready. If the Pods are not Running, the fix is to investigate the Pod’s status.

Step 2: Check the Ingress

The second step is to verify the Ingress:

kubectl describe ingress billing

The Ingress must have the right rules, the right TLS configuration, and the right backend.

Step 3: Check the backend

The third step is to verify the backend Service:

kubectl get endpointslices -l kubernetes.io/service-name=billing

The backend Service must have endpoints. The endpoints must be Ready.

Step 4: Check the TLS

The fourth step is to verify the TLS configuration:

kubectl get secret billing-tls -o yaml

The TLS secret must exist. The certificate must be valid.

Step 5: Check the metrics

The fifth step is to verify the metrics:

# Substitute your own value before running - the address of an
# ingress-nginx controller Pod:
CONTROLLER_IP=192.0.2.51

curl "http://$CONTROLLER_IP:10254/metrics"

The metrics show the request rate, the response rate, the latency, and the error rate. The cluster operator monitors these metrics.

The common failure modes

The common failure modes of Ingress:

• Controller down: the Ingress controller is unavailable. The fix is to restart the controller.
• Backend not ready: the backend Service has no Pods. The fix is to verify the backend.
• TLS error: the TLS certificate is invalid. The fix is to update the certificate.
• Routing error: the Ingress routes to the wrong backend. The fix is to verify the rules.
• Default backend missing: the Ingress has no default backend. The fix is to set the default.

The operational discipline

The Ingress troubleshooting’s operational discipline:

• Document the diagnostic flow. The flow is the cluster’s operational reference.
• Run the diagnostics regularly. The diagnostics are the leading indicator.
• Monitor the metrics. The metrics are the leading indicator of degradation.
• Test the recovery in staging. The recovery must be tested before production.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.

Quiz

Production discipline

• Document the diagnostic flow. The flow is the cluster’s operational reference.
• Run the diagnostics regularly. The diagnostics are the leading indicator.
• Monitor the metrics. The metrics are the leading indicator of degradation.
• Test the recovery in staging. The recovery must be tested before production.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.
• Set up alerts on the metrics. The alerts are the leading indicator of outages.
• Document the troubleshooting in the runbook. The runbook is the cluster’s operational reference.
• Plan the troubleshooting’s evolution. The troubleshooting is the cluster’s operational reference; the operator must keep it current.