Native sidecar containers (KEP-753, 1.28+)

Before Kubernetes 1.28, sidecars were implemented as init containers with a workaround: a sidecar injects itself as an init container, but never exits, so it runs alongside the main container. The problem: lifecycle ordering is broken — the sidecar can be killed before the main container has finished its work. Native sidecars (KEP-753, GA in 1.28) fix this by allowing init containers to declare restartPolicy: Always and giving them proper sidecar semantics.

The sidecar problem

A sidecar is a container that runs alongside the main container to provide auxiliary functionality: log shipping, metrics collection, service mesh proxy, etc. The requirements:

• Start before the main container: the sidecar must be running before the main container starts (e.g., a service mesh proxy must be configured before the app accepts traffic).
• Stop after the main container: the main container finishes its work, then the sidecar drains and exits.
• Restart on failure: if the sidecar crashes, the kubelet restarts it (and only it, not the main container).

Before 1.28, the workaround was:

initContainers:
- name: sidecar
  image: sidecar:1.0
  command: ["sleep", "infinity"]   # never exits; acts as a long-running container
containers:
- name: app
  image: app:1.0.0

The sidecar is an init container that runs sleep infinity and never exits. The kubelet starts it before the main container; the main container starts once the init “succeeds” (by virtue of the sleep continuing). The sidecar runs in the background.

The problem: shutdown ordering. When the Pod is terminated:

• The kubelet sends SIGTERM to all containers in parallel.
• The main container may still be processing requests when the sidecar is killed.
• The sidecar may need to drain connections but is killed before it can.

sequenceDiagram
    participant K as Kubelet
    participant S as Sidecar (init)
    participant M as Main

    K->>S: SIGTERM (parallel)
    K->>M: SIGTERM (parallel)
    S-->>S: dies mid-drain
    M-->>M: continues, sees sidecar dead, fails

Native sidecars

In Kubernetes 1.28 (KEP-753, GA), init containers can declare restartPolicy: Always. The kubelet treats them as sidecars:

initContainers:
- name: sidecar
  image: sidecar:1.0
  restartPolicy: Always        # native sidecar
  command: ["sidecar", "--port=9090"]
containers:
- name: app
  image: app:1.0.0

The behaviour:

• The init container starts as a normal init container (sequentially before main containers).
• When the init container’s main process is running, the kubelet starts the next init container (or the main containers).
• The init container is restarted on failure (like a normal container with restartPolicy: Always).
• On Pod termination, the init container is terminated after the main container. The kubelet sends SIGTERM to the main container first, waits, then sends SIGTERM to the sidecar.

sequenceDiagram
    participant K as Kubelet
    participant S as Native sidecar
    participant M as Main

    K->>S: start sidecar
    S-->>S: running
    K->>M: start main (after sidecar)
    M-->>M: running
    Note over K,M: Pod termination
    K->>M: SIGTERM
    M-->>M: drain, exit
    K->>S: SIGTERM (after main)
    S-->>S: drain, exit

The ordering: main container terminates before sidecar. This is the proper semantics for sidecars that drain connections (service mesh proxies, log shippers).

Native sidecar ordering rules

The kubelet enforces these ordering rules for init containers with restartPolicy: Always:

• Start order: native sidecars start in the order they appear in spec.initContainers. Each sidecar must reach a “ready” state (or the kubelet considers it started) before the next init container or the main containers start.
• Readiness for ordering: the kubelet considers a sidecar “started” when its process is running. The sidecar can declare a startup probe to make the readiness explicit; the kubelet waits for the startup probe to succeed before starting the next container.
• Stop order: on Pod termination, the main containers receive SIGTERM first. After the main containers exit (or the grace period expires), the sidecars receive SIGTERM.
• Sidecar-to-sidecar ordering: sidecars are started in the order they appear in the list. Shutdown order is the reverse: the last sidecar is stopped first, the first sidecar is stopped last.

stateDiagram-v2
    [*] --> Sidecar1
    Sidecar1 --> Sidecar2: started
    Sidecar2 --> Main: started
    Main --> MainTerm: SIGTERM
    MainTerm --> Sidecar2Term: main exited
    Sidecar2Term --> Sidecar1Term: sidecar2 exited
    Sidecar1Term --> [*]

Migrating from annotation-based sidecars

Service meshes (Istio, Linkerd) historically injected sidecars as init containers with the sleep infinity workaround. With native sidecars GA, the meshes can declare their sidecars as proper init containers with restartPolicy: Always.

For a workload with an Istio sidecar:

# Before 1.28 (annotation-based workaround)
initContainers:
- name: istio-proxy
  image: istio/proxyv2:1.20
  command: ["sh", "-c", "sleep infinity"]   # workaround

# After 1.28 (native sidecar)
initContainers:
- name: istio-proxy
  image: istio/proxyv2:1.20
  restartPolicy: Always
  startupProbe:
    httpGet:
      path: /healthz/ready
      port: 15021
    failureThreshold: 30
    periodSeconds: 2

The behaviour change:

• Before: sidecar is a regular init container (sleep infinity); on termination, sidecar and main are SIGTERM’d in parallel; sidecar may die first.
• After: sidecar is a native sidecar; on termination, main is SIGTERM’d first, sidecar is SIGTERM’d after main exits.

For service mesh traffic draining: the sidecar continues running while the main container drains; only when the main exits does the sidecar begin its drain.

Production patterns

Native sidecar with startup probe:

initContainers:
- name: log-shipper
  image: fluent-bit:3.0
  restartPolicy: Always
  startupProbe:
    httpGet:
      path: /
      port: 2020
    failureThreshold: 30
    periodSeconds: 2

The log shipper must be ready (HTTP responding on its API port) before the main container starts. The startup probe makes the readiness explicit.

Native sidecar with shared volume:

initContainers:
- name: log-shipper
  image: fluent-bit:3.0
  restartPolicy: Always
  volumeMounts:
  - name: logs
    mountPath: /var/log/app
containers:
- name: app
  image: app:1.0.0
  volumeMounts:
  - name: logs
    mountPath: /var/log/app
volumes:
- name: logs
  emptyDir: {}

The sidecar reads logs from the shared volume.

Native sidecar with resource limits:

initContainers:
- name: sidecar
  image: sidecar:1.0
  restartPolicy: Always
  resources:
    requests:
      cpu: 50m
      memory: 64Mi
    limits:
      cpu: 200m
      memory: 128Mi

Native sidecars have separate resource requests/limits, like main containers. The scheduler accounts for them in the Pod’s total resource needs.

Diagnosing native sidecar issues

Sidecar not starting:

kubectl describe pod web-7c8
# Events:
#   ... reason: Failed   message: container "sidecar" failed to start

The sidecar’s image pull failed or the container cannot start. Check kubectl logs web-7c8 -c sidecar.

Sidecar restarting repeatedly:

kubectl get pod web-7c8 -o jsonpath='{.status.initContainerStatuses[*].restartCount}'

If the restartCount is high, the sidecar is crashing. Check the logs.

Main container not starting:

kubectl get pod web-7c8 -o jsonpath='{.status.initContainerStatuses[*].state}' | jq

If a sidecar’s state is waiting (not running), the main container is waiting for the sidecar to start. Check the sidecar’s status.

Termination ordering broken:

If the sidecar dies before the main container, you may have a non-native sidecar (sleep infinity). Verify the sidecar has restartPolicy: Always and the kubelet version is 1.28+.

Cross-course references

• The Linux course part VI-Linux-Processes covers process lifecycle and signal handling; native sidecars are the cluster-level equivalent of ordered process shutdown.
• The Docker course part XXX-Docker-Lifecycle covers container shutdown; native sidecars are the cluster-level extension.
• The Ansible course part XXXV-Ansible-Scripting covers service shutdown; native sidecars are the cluster-level equivalent of dependency-aware shutdown.

Quiz

Production discipline

• Migrate from sleep-infinity to native sidecars. Native sidecars (1.28+) provide proper lifecycle ordering.
• Configure your service mesh to inject native sidecars. Verify the version supports it; opt in via configuration.
• Use startup probes on native sidecars. The kubelet waits for the sidecar to be ready before starting main containers.
• Test termination ordering. Trigger a Pod deletion, verify the main container drains before the sidecar is killed.
• Audit sidecar resource limits. Native sidecars have separate resource fields; size them appropriately.