Migrating from annotation-based sidecars to native sidecars

Migrating from annotation-based sidecars to native sidecars is a cluster-wide operation. This lesson walks through the migration steps, the verification at each stage, and the rollback plan if the migration causes issues.

Migration prerequisites

Before migrating, verify:

1. Kubernetes version: cluster is 1.28 or later (native sidecars GA in 1.28).
2. Service mesh version: the mesh supports native sidecars (Istio 1.20+, Linkerd 2.14+, etc.).
3. Mutating webhook configuration: the mesh’s sidecar injector is configured to emit native sidecars.
4. Test environment: the migration is tested in staging first.

kubectl version
# Server Version: v1.30.x

istioctl version
# 1.20.x or later

Step 1: Configure the mesh

For Istio:

istioctl install --set values.global.proxy.image=... \
  --set values.global.proxy.nativeSidecar=true \
  --set values.global.proxy.lifecycle.imagePullPolicy=IfNotPresent

Or via the IstioOperator:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    defaultConfig:
      proxyMetadata:
        ISTIO_META_ENABLE_HBONE: "true"
  components:
    proxy:
      nativeSidecar: true

For Linkerd, the sidecar injection is already native-sidecar compatible in recent versions; verify with linkerd check.

For other service meshes, consult the mesh’s documentation.

Step 2: Restart workloads

After enabling native sidecars in the mesh, restart all workloads so they receive the new sidecar format:

kubectl rollout restart deployment -A
# or per-namespace
kubectl rollout restart deployment -n team-a-prod

This triggers a rolling update; new Pods are created with the new sidecar format. Old Pods are terminated; new Pods have native sidecars.

Step 3: Verify the migration

For each namespace, verify that the sidecar is now a native sidecar:

kubectl get pod web-7c8 -o jsonpath='{.spec.initContainers[0].restartPolicy}'
# Always

If the output is empty, the sidecar is still a sleep-infinity workaround. Check the mesh’s configuration.

Other checks:

# Sidecar has a startup probe
kubectl get pod web-7c8 -o jsonpath='{.spec.initContainers[0].startupProbe}'
# should not be empty

# Sidecar resources are set
kubectl get pod web-7c8 -o jsonpath='{.spec.initContainers[0].resources}'
# should not be empty

The native sidecar has explicit restartPolicy: Always, startupProbe, and resources. The sleep-infinity workaround has none of these.

Step 4: Test termination ordering

After the migration, test that the new ordering works:

kubectl delete pod web-7c8 --grace-period=30
# Watch the termination sequence
kubectl get pod web-7c8 -w

The expected sequence:

1. Pod’s deletionTimestamp is set.
2. Main container receives SIGTERM.
3. Main container drains, exits.
4. Sidecar receives SIGTERM (after main exits).
5. Sidecar drains, exits.
6. Pod is removed.

For the sleep-infinity workaround, SIGTERM is sent to main and sidecar in parallel; the main may exit while the sidecar is being killed.

sequenceDiagram
    participant K as Kubelet
    participant M as Main
    participant S as Sidecar

    rect rgb(200, 240, 200)
    Note over M: Native sidecar (after migration)
    K->>M: SIGTERM
    M-->>M: drain, exit
    K->>S: SIGTERM (after main)
    S-->>S: drain, exit
    end

    rect rgb(240, 200, 200)
    Note over M: Sleep-infinity workaround (before)
    K->>M: SIGTERM (parallel)
    K->>S: SIGTERM (parallel)
    M-->>M: drain (sidecar may already be dead)
    S-->>S: dies mid-drain
    end

Step 5: Monitor for issues

After the migration, watch for:

• Pods stuck in Pending: native sidecars may take longer to start if their startupProbe is more strict than the sleep-infinity workaround.
• Sidecar restart loops: the sidecar may crash if its configuration is incompatible with the new format.
• Memory pressure: native sidecars with explicit resources may use more memory than the workaround.

# Pods stuck in Pending with sidecar-related events
kube_pod_container_status_waiting_reason{reason="ContainerCreating"}

# Sidecar restart count
kube_pod_init_container_status_restarts_total

Rollback procedure

If the migration causes issues:

1. Revert the mesh configuration: disable native sidecar injection, restore the sleep-infinity workaround.
2. Restart workloads: kubectl rollout restart deployment -A to trigger re-injection with the workaround.
3. Verify the rollback: check that the new Pods have the workaround (no restartPolicy on the init container).

istioctl install --set values.global.proxy.nativeSidecar=false
kubectl rollout restart deployment -A

The rollback is the inverse of the migration. The old workaround Pods are not graceful (parallel SIGTERM), but they work.

Migration in stages

For a large cluster, do the migration in stages:

1. One namespace first: pick a low-risk namespace; run the migration; verify; document issues.
2. Cluster-wide rollout: enable native sidecars cluster- wide; rollout restart all workloads.
3. Verification: monitor for 24-48 hours; verify no regressions.

# Stage 1: one namespace
kubectl label namespace team-a-staging istio-injection=enabled
istioctl install --set ... --set values.global.proxy.nativeSidecar=true
kubectl rollout restart deployment -n team-a-staging

# Stage 2: cluster-wide
kubectl label namespace --all istio-injection=enabled
kubectl rollout restart deployment -A

Production patterns

Pre-migration checklist:

• Cluster is 1.28+
• Service mesh version supports native sidecars
• Mesh is configured for native sidecar injection
• Test environment: migration verified in staging
• Rollback procedure documented and tested
• Monitoring in place (Pending Pods, sidecar restarts)
• Communication: users notified of the change

Post-migration checklist:

• All workloads have native sidecars (restartPolicy: Always on init container)
• No Pods stuck in Pending
• Sidecars have startup probes
• Sidecars have explicit resources
• Termination ordering verified under load
• No increase in 5xx errors during the migration window

Cross-course references

• The Linux course part IX-Linux-Boot covers service restart patterns; the migration is the cluster-level equivalent.
• The Docker course part XXIX-Docker-Build covers image upgrades; the mesh upgrade is the cluster-level equivalent.
• The Ansible course part XXXV-Ansible-Scripting covers rolling restart discipline; the staged rollout is the same idea.

Quiz

Production discipline

• Test the migration in staging first. Verify termination ordering, sidecar restarts, and resource usage before going cluster-wide.
• Stage the rollout by namespace. Reduce blast radius; catch issues early.
• Pre-stage the rollback. Document and test the rollback procedure before the migration.
• Verify each workload’s sidecar after migration. restartPolicy: Always on the init container is the marker.
• Monitor for 24-48 hours after migration. Pods stuck in Pending, sidecar restart loops, memory regressions.