Filesystem remount and read-only — the storage integrity

When the kernel hits an I/O error it remounts the filesystem read-only to stop the damage spreading. The Pod does not crash: it keeps running, its liveness probe keeps passing, and every write fails with EROFS until someone reads the application log closely. Restarting the Pod remounts the volume read-write and makes the symptom disappear, which is precisely why the underlying error has to be identified first — this lesson covers doing that before the evidence is discarded.

The read-only filesystem

A read-only filesystem is the kernel’s response to a filesystem error. The kernel remounts the filesystem as read-only to prevent further corruption.

# Substitute your own values before running:
POD=postgres-0
NS=production

# Check the filesystem
kubectl exec -it "$POD" -n "$NS" -- mount | grep /data

A real mount:

/dev/sda on /data type ext4 (ro,relatime)

The ro flag is the read-only filesystem.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
POD=postgres-0
NS=production
PVC=data-postgres-0       # the claim the Pod mounts at /data

# 1. Check the filesystem
kubectl exec -it "$POD" -n "$NS" -- mount | grep /data

# 2. Check the kernel logs
kubectl exec -it "$POD" -n "$NS" -- dmesg | tail -50

# 3. Check the storage backend
# (backend-specific)

# 4. Check the PVC
kubectl get pvc -n "$NS"
kubectl describe pvc "$PVC" -n "$NS"

# 5. Check the events
kubectl get events -n "$NS" --field-selector involvedObject.name="$POD"

The diagnostic is the filesystem, the kernel logs, the storage backend, and the PVC.

Common failures

• Filesystem error. The filesystem has an error. The remediation is to run fsck and remount.
• Storage backend error. The storage backend is reporting errors. The remediation is to fix the backend.
• Network partition. The network partition caused the filesystem to remount as read-only. The remediation is to fix the network.
• Volume detached. The volume was detached while the Pod was running. The remediation is to re-attach the volume.

flowchart TD
    A[Filesystem read-only] --> B{Filesystem error?}
    B -->|Yes| C[Fix the filesystem]
    B -->|No| D{Backend error?}
    D -->|Yes| E[Fix the backend]
    D---|No| F{Network partition?}
    F -->|Yes| G[Fix the network]
    F -->|No| H{Volume detached?}
    H -->|Yes| I[Re-attach the volume]
    H -->|No| J[Unknown]

The remediation

The remediation depends on the cause:

# Option 1: Reboot the Pod
# (the kubelet will remount the filesystem)

# Option 2: Run fsck
POD=postgres-0
NS=production
kubectl exec -it "$POD" -n "$NS" -- fsck /dev/sda

# Option 3: Fix the backend
# (backend-specific)

# Option 4: Re-attach the volume
# (force-detach, then re-attach)

The remediation is the storage integrity recovery.

The remount

The remount is the kernel’s recovery. The kubelet remounts the filesystem when the Pod is restarted.

# Substitute your own values before running:
POD=postgres-0
NS=production

# Restart the Pod
kubectl delete pod "$POD" -n "$NS"

The Pod is restarted; the kubelet remounts the volume.

Production discipline

A read-only filesystem is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the storage layer, identify the cause, apply the remediation. The storage is the cluster’s data; the remediation is the storage integrity recovery.

• Check the filesystem. mount | grep /data inside the Pod shows the ro flag; the liveness probe will not, so the symptom is invisible until someone looks.
• Check the kernel logs. dmesg | tail -50 is the kernel’s view of why it remounted, and it is the evidence a Pod restart discards.
• Check the storage backend and the PVC. A backend error, a network partition, or a detached volume all surface as the same ro mount, and only the backend distinguishes them.

Quiz