Service mesh capture — Istio, Linkerd, Cilium, and the mTLS rewrites

A service mesh adds a sidecar (Envoy, linkerd-proxy, or Cilium’s envoy) to every Pod. The sidecar intercepts all inbound and outbound traffic, applies L7 routing rules, and encrypts the byte stream with mTLS. The application sees a clean TCP connection to the upstream; the wire sees encrypted traffic between sidecars. This lesson walks through what capture sees at each layer of the mesh.

How a service mesh rewrites traffic

The standard mesh sidecar pattern:

flowchart LR
    A[app container] -->|localhost:15001| B[sidecar]
    B -->|mTLS to upstream| C[network]
    C --> D[upstream sidecar]
    D -->|localhost:8080| E[upstream app]

The application sends plaintext to localhost:15001 (Istio) or localhost:4191 (Linkerd). The sidecar intercepts via iptables (or eBPF for Cilium), establishes an mTLS connection to the upstream sidecar, and forwards the plaintext to the upstream application.

flowchart TB
    subgraph "Pod A"
      App1[app] -->|plaintext| S1[sidecar]
    end
    subgraph "Pod B"
      S2[sidecar] -->|plaintext| App2[app]
    end
    S1 -->|mTLS ciphertext| Net[underlay]
    Net --> S2

From the application’s perspective, the upstream is upstream-svc.namespace.svc.cluster.local:8080. From the wire’s perspective, the connection is between two Pod IPs, encrypted.

What packet capture sees at each layer

The production rule: capture at L7 (the sidecar’s access log) for protocol-level debugging, not at L4 (the wire) for mesh traffic.

Istio capture patterns

Istio exposes the sidecar’s state through istioctl and Envoy’s admin interface. The production patterns:

# Substitute your own values before running:
POD=checkout-7d9f6b58c4-mn2wq
NS=production

# Inspect the sidecar's configuration for a Pod
istioctl proxy-config routes "$POD.$NS"
istioctl proxy-config clusters "$POD.$NS"
istioctl proxy-config endpoints "$POD.$NS"
istioctl proxy-config listeners "$POD.$NS"

These commands show what the sidecar knows: the routes, the upstream clusters, the endpoints, the listeners. A sidecar that does not have the right route is a configuration bug, not a network bug.

# Substitute your own values before running:
POD=checkout-7d9f6b58c4-mn2wq
NS=production

# Capture mesh traffic via the sidecar's tap
istioctl dashboard controlz "$POD.$NS"

Envoy’s controlz is the runtime admin UI. The dashboard shows the active connections, the cluster stats, and the listener configuration.

For packet-level capture, Istio’s tcpdump works but the output is ciphertext. The mesh’s value is in the L7 access logs, not the wire capture.

# Enable Istio access logs
kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
  - providers:
    - name: envoy
    disabled: false
EOF

The access log shows the request method, path, response code, source workload, destination workload, and the mTLS peer identity. This is what most “the mesh is dropping traffic” investigations need.

Linkerd capture patterns

Linkerd exposes a similar set of tools:

# Substitute your own values before running:
DEPLOY=checkout
PEER_DEPLOY=payments
NS=production

# Tap a workload's traffic
linkerd viz tap "deploy/$DEPLOY" -n "$NS" --to "deploy/$PEER_DEPLOY"

# Top routes by request rate
linkerd viz routes "deploy/$DEPLOY" -n "$NS"

# Live requests
linkerd viz stat "deploy/$DEPLOY" -n "$NS" --to "deploy/$PEER_DEPLOY"

Linkerd’s tap is closer to “capture what the application is sending” than Istio’s proxy-config, because Linkerd’s proxy emits a structured event per request.

# Substitute your own values before running:
DEPLOY=checkout
NS=production

# Live TCP-level capture (still mTLS ciphertext)
linkerd viz tap "deploy/$DEPLOY" -n "$NS" --method tcp

Linkerd’s mTLS is opportunistic. By default, Linkerd upgrades plaintext to mTLS when both ends have a linkerd-proxy. The tap command can capture both plaintext (before encryption) and the post-encryption metadata.

Cilium service mesh capture

Cilium uses eBPF, which means the sidecar is partly in the kernel. The capture patterns are different:

# Substitute your own values before running:
NS=production
POD=checkout-7d9f6b58c4-mn2wq

# Hubble observes all traffic (eBPF-based)
hubble observe --namespace "$NS" --pod "$POD"

# Hubble's flow logs (L7 visibility)
hubble observe --namespace "$NS" --follow --protocol http

Hubble is the kernel-level observability for Cilium. It sees the L3/L4 traffic as eBPF events and the L7 traffic (by Envoy integration) as structured HTTP/gRPC events. This is more comprehensive than sidecar capture because it covers Pod-to-Pod without sidecars.

# Substitute your own value before running:
NS=production

# Capture at L7 via Cilium's policy verdicts
hubble observe --verdict DROPPED --namespace "$NS"

hubble observe --verdict DROPPED is the production default for “the mesh is dropping traffic” — it shows exactly which flows the policy engine rejected.

mTLS and decryption

Production meshes use mTLS by default (Istio STRICT mode, Linkerd always). The wire is encrypted; the sidecar decrypts. To capture plaintext at the application, capture on the loopback interface inside the Pod:

# Inside the application Pod, on the loopback interface
tcpdump -ni lo port 15001 -w /tmp/app-to-sidecar.pcap

# On the Pod's eth0 (between sidecar and sidecar)
tcpdump -ni eth0 -w /tmp/sidecar-to-sidecar.pcap

The first capture is plaintext (app-to-sidecar). The second is mTLS ciphertext. The first is what the application sent; the second is what the wire carried.

The Istio and Linkerd access logs are the production-grade replacement for both: they show the L7 information without capturing bytes.

Quiz

Production discipline

• Default to the mesh’s L7 tooling. istioctl proxy-config, linkerd viz tap, hubble observe. These are designed for the mesh’s debugging questions.
• Wire-level capture is a last resort. mTLS means the wire is ciphertext. Use the access log; use the policy verdict log; use the trace ID.
• Capture plaintext on the loopback inside the Pod. tcpdump -ni lo port 15001 shows what the application sent to the sidecar.
• Capture policy verdicts for “the mesh is dropping”. hubble observe --verdict DROPPED (Cilium) or Envoy access log with response_flags (Istio) shows the decision.
• Treat the mesh’s metrics as the first signal. Request rate, error rate, latency (the RED method) tells you that there is a problem before you start asking what the problem is.