Production sidecar patterns — logging, mesh, init migrations

The previous lessons covered init containers and native sidecars in the abstract. This lesson applies them to real production patterns: log shipping, service mesh, init migrations, and secret bootstrapping. For each pattern: the architecture, the sizing, and the pitfalls.

Log shipping sidecar

The classic sidecar pattern. The application writes logs to stdout (or to a file in a shared volume); the sidecar tails the log and ships it to a log aggregator (Loki, Elasticsearch, CloudWatch).

initContainers:
- name: log-shipper
  image: fluent/fluent-bit:3.0
  restartPolicy: Always
  resources:
    requests: {cpu: 50m, memory: 64Mi}
    limits:   {cpu: 200m, memory: 128Mi}
  volumeMounts:
  - name: logs
    mountPath: /var/log/app
containers:
- name: app
  image: app:1.0.0
  resources:
    requests: {cpu: 500m, memory: 512Mi}
    limits:   {cpu: 1, memory: 1Gi}
  volumeMounts:
  - name: logs
    mountPath: /var/log/app
volumes:
- name: logs
  emptyDir: {sizeLimit: 1Gi}

The flow:

1. The application writes logs to /var/log/app/app.log (or stdout, picked up by the runtime).
2. Fluent Bit tails the file and ships lines to the log aggregator.
3. On termination, the main container drains; the sidecar flushes its buffer and exits.

flowchart LR
    App[App] -->|"writes logs"| Vol["emptyDir: logs"]
    Vol -->|"tail -f"| FB[Fluent Bit sidecar]
    FB -->|"forwards to"| Loki[Loki / Elasticsearch]

Sizing

A Fluent Bit sidecar for a typical workload:

• CPU: 50-200m depending on log volume. A high-traffic app may need more.
• Memory: 64-256Mi. Fluent Bit buffers logs in memory; higher throughput needs more buffer.

Production discipline: size for peak log volume, not average. The sidecar can OOMKill if it cannot keep up with the application’s log output.

Pitfalls

• emptyDir size limit too small: the application writes faster than the sidecar reads; the volume fills; the application may block on writes.
• Sidecar’s buffer not flushed on shutdown: the sidecar may exit before flushing its in-memory buffer; logs are lost. Configure Fluent Bit with a graceful shutdown timeout.
• Sidecar restart loops: a misconfigured Fluent Bit config causes the sidecar to crash repeatedly. The Pod becomes Pending because of the startupProbe.

Service mesh sidecar

A service mesh injects a sidecar proxy (Envoy, Linkerd-proxy) that intercepts all network traffic to/from the main container. The proxy enforces policy, collects metrics, and provides mutual TLS.

initContainers:
- name: istio-proxy
  image: istio/proxyv2:1.20
  restartPolicy: Always
  resources:
    requests: {cpu: 100m, memory: 128Mi}
    limits:   {cpu: 500m, memory: 256Mi}
  startupProbe:
    httpGet: {path: /healthz/ready, port: 15021}
    failureThreshold: 30
    periodSeconds: 2
  args:
  - proxy
  - sidecar
  - --domain
  - $(POD_NAMESPACE).svc.cluster.local
  - --serviceCluster
  - web
containers:
- name: app
  image: app:1.0.0
  resources:
    requests: {cpu: 500m, memory: 512Mi}
    limits:   {cpu: 1, memory: 1Gi}

The flow:

1. The Istio sidecar starts before the main container.
2. The sidecar connects to the Istio control plane (istiod) and receives its configuration (xDS).
3. The main container starts; its traffic is redirected to the sidecar via iptables (or eBPF).
4. The sidecar enforces policy, encrypts traffic, reports metrics.

flowchart LR
    App[App] -->|"traffic redirected via iptables"| Proxy[Istio proxy]
    Proxy -->|"mTLS"| Peer[Other Pod's proxy]
    Proxy -->|"metrics"| Control[istiod]

Sizing

Istio sidecar sizing depends on traffic volume:

• Low traffic: 100m CPU, 128Mi memory.
• Medium traffic: 500m CPU, 256Mi memory.
• High traffic: 1 CPU, 512Mi memory.

Production discipline: monitor the sidecar’s CPU and memory in production. Sizing too low causes OOMKill; sizing too high wastes node resources.

Pitfalls

• xDS connection failures: the sidecar cannot reach the control plane. Common causes: network policy, DNS resolution, cert issues.
• Configuration errors: invalid Envoy config causes the sidecar to crash. Validate config with istioctl analyze.
• Memory leaks: older Istio versions had known leaks; upgrade to a current version.
• Sidecar injection not applied: a Pod without the istio-injection=enabled label does not get a sidecar. Check the namespace’s labels.

Init migration sidecar

A pattern where the init container runs a database migration, and the main container starts after the migration succeeds.

initContainers:
- name: db-migration
  image: migrate/migrate:4
  command:
  - migrate
  - -path=/migrations
  - -database=postgres://user:pass@db:5432/app?sslmode=disable
  - up
  resources:
    requests: {cpu: 100m, memory: 256Mi}
    limits:   {cpu: 500m, memory: 512Mi}
  volumeMounts:
  - name: migrations
    mountPath: /migrations
containers:
- name: app
  image: app:1.0.0
volumes:
- name: migrations
  configMap:
    name: db-migrations

The flow:

1. The init container runs the migration tool against the database.
2. If the migration succeeds, the main container starts.
3. If the migration fails, the Pod is stuck in Pending; the kubelet retries.

This is technically an init container (not a native sidecar — restartPolicy is not Always). The migration runs once and exits.

Pitfalls

• Non-idempotent migrations: the migration tool may fail on subsequent runs if it doesn’t track applied migrations. Use a tool that does (golang-migrate, Flyway, Liquibase).
• Long migrations blocking rollouts: a migration that takes minutes delays the main container’s start. The rolling update stalls until all Pods complete the migration.
• Migration requires the database to be writable: the Pod must have credentials with DDL permission. Don’t use the application’s read-only user.

Secret bootstrapping sidecar

A pattern where the init container fetches secrets from an external system (Vault, AWS Secrets Manager) and writes them to a shared volume. The main container reads the secrets from the volume.

initContainers:
- name: vault-fetcher
  image: vault:1.15
  restartPolicy: Always   # native sidecar to support refresh
  resources:
    requests: {cpu: 50m, memory: 64Mi}
    limits:   {cpu: 100m, memory: 128Mi}
  volumeMounts:
  - name: secrets
    mountPath: /secrets
containers:
- name: app
  image: app:1.0.0
  volumeMounts:
  - name: secrets
    mountPath: /etc/app/secrets
volumes:
- name: secrets
  emptyDir: {}

The flow:

1. The Vault sidecar authenticates to Vault and fetches the secrets.
2. The sidecar writes the secrets to /secrets.
3. The main container starts only once the native sidecar has started — gated by its startupProbe when one is set — and then reads the same emptyDir at /etc/app/secrets.

For dynamic secrets (Vault leases), the sidecar periodically refreshes the secrets; the main container reads them.

Pitfalls

• Sidecar authentication: the Vault sidecar needs credentials to authenticate. Use the Pod’s ServiceAccount token (with Vault’s Kubernetes auth method) or a static token (less secure).
• Sidecar vs CSI: the Vault CSI driver mounts Vault secrets as a CSI volume; this is cleaner than a sidecar for static secrets. Use a sidecar for dynamic refresh.
• Secret leakage: secrets in emptyDir are visible to any process in the Pod. Ensure the main container restricts access.

Cross-course references

• The Linux course part XXXV-Linux-Scripting covers shell scripting patterns; init container scripts are the cluster-level equivalent.
• The Docker course part XXXI-Docker-Networking covers container networking; service mesh sidecars are the cluster-level extension.
• The Ansible course part XLIX-Ansible-Compliance covers secret management; the sidecar pattern is the cluster-level equivalent.

Quiz

Production discipline

• Set sizeLimit on every shared emptyDir in sidecar patterns. Unbounded shared volumes are a node-stability risk.
• Size sidecar resources for peak load. A log shipper or service mesh proxy that OOMKills under load is worse than over-provisioning.
• Use the Vault CSI driver for static secrets. Reserve sidecar patterns for dynamic secrets that need refresh.
• Separate long migrations from Pod startup. Use a Job; let the main container start with the schema already migrated.
• Verify sidecar injection after mesh upgrades. A mesh upgrade can break injection if the webhook configuration is wrong.