Drain with local data — emptyDir, hostPath, and the data-loss risk

The drain is destructive for Pods with local data. The emptyDir volumes are deleted; the hostPath volumes are orphaned on the node; the local persistent volumes are preserved but unavailable. This lesson walks the drain’s interaction with local data, the data-loss risk, and the design patterns.

The local data types

flowchart TD
    A[Pod with local data] --> B{Volume type?}
    B -->|emptyDir| C[Data lost on eviction]
    B -->|hostPath| D[Data preserved on node]
    B -->|local PV| E[Data preserved on node]
    C --> F[--delete-emptydir-data needed]
    D --> G[Data lost on node replacement]
    E --> G
    G --> H[Backup required]

The local data types that the drain affects:

• emptyDir: a directory that is created when the Pod is created and deleted when the Pod is deleted. The data is lost when the Pod is evicted.
• hostPath: a directory or file on the node’s filesystem. The data is preserved on the node when the Pod is evicted.
• local PV: a persistent volume that is bound to a node. The volume survives the eviction, and its node affinity pins the replacement Pod to the same node.

The three types have different data-loss behaviours:

The drain’s default behaviour is to reject the operation when the Pod has an emptyDir volume. The Pod’s data is lost; the operator must approve the loss.

The drain’s behaviour with emptyDir

The drain’s default behaviour:

# Substitute your own node name:
NODE=node-1

kubectl drain "$NODE"

error when evicting pod "default/cache-1": cannot evict pod with emptyDir volumes (use --delete-emptydir-data to ignore)

The drain rejects the operation. The Pod’s emptyDir volume is preserved; the Pod is not evicted.

The drain’s destructive behaviour:

# Substitute your own node name:
NODE=node-1

kubectl drain "$NODE" --delete-emptydir-data

node/node-1 cordoned
evicting pod "default/cache-1"
pod "default/cache-1" evicted

The drain is allowed. The Pod’s emptyDir volume is deleted; the Pod is evicted.

The drain’s behaviour with hostPath

The drain’s behaviour with hostPath:

# Substitute your own node name:
NODE=node-1

kubectl drain "$NODE"

The drain evicts the Pod. The hostPath data is preserved on the node. The Pod can be re-created on the same node (the data is still there).

The drain’s behaviour with hostPath when the node is replaced:

# The node is replaced.
# The new node does not have the hostPath data.
# The Pod is re-created on the new node without the data.

The hostPath data is lost when the node is replaced. The data is bound to the node’s filesystem; the new node does not have the data.

The production rule: hostPath is not a substitute for persistent storage. The data is bound to the node; the data is lost when the node is replaced.

The drain’s behaviour with local PVs

The drain’s behaviour with local PVs:

# Substitute your own node name:
NODE=node-1

kubectl drain "$NODE"

The drain evicts the Pod. The local PV is preserved on the node. The Pod can be re-created on the same node (the PV is still there).

The drain’s behaviour with local PVs when the node is replaced:

# The node is replaced.
# The new node does not have the local PV.
# The Pod is re-created on the new node without the PV.

The local PV is lost when the node is replaced. The PV is bound to the node; the new node does not have the PV.

The production rule: local PVs are not a substitute for network storage. Replacing the node leaves the PV unschedulable, and the Pod stays Pending with a volume node affinity conflict.

The drain’s pattern with local data

The drain’s pattern with local data:

1. Identify the local data type. The Pod’s spec shows the volumes.
2. Choose the right flag. The --delete-emptydir-data flag is destructive; the default is to reject.
3. Drain the node. The drain evicts the Pods.
4. Verify the local data. The data is preserved on the node (hostPath) or lost (emptyDir).
5. Uncordon the node. The node is uncordoned.

The drain’s pattern is the cluster’s protection against the data-loss. The operator must approve the loss.

The design patterns for local data

The design patterns for Pods that survive the drain:

• Use network storage for persistent data. The cluster’s network storage (CSI, NFS, Ceph) is preserved across the drain.
• Use emptyDir for temporary data. The emptyDir volume is deleted when the Pod is evicted; the data is intentionally lost.
• Use hostPath for node-local data. The hostPath data is preserved on the node; the data is bound to the node.
• Use local PVs for node-local persistent data. The local PV is preserved on the node; the data is bound to the node.

The design patterns are the operator’s responsibility. The production rule is to design the workload to survive the drain.

The anti-patterns

The drain’s anti-patterns:

• Storing data in emptyDir. The emptyDir volume is deleted when the Pod is evicted; the data is lost.
• Storing data in hostPath. The hostPath data is bound to the node; the data is lost when the node is replaced.
• Storing data in local PVs for cluster-wide workloads. The local PV is bound to the node; the data is lost when the node is replaced.
• Not backing up local data. The local data is bound to the node; the data is lost when the node is replaced.

The anti-patterns are the cluster’s silent failures. Each one works through every drain until the node is replaced, and the loss surfaces only then.

The recovery from data loss

The recovery from data loss:

1. Identify the lost data. The Pod’s events show the eviction.
2. Recover from backup. The cluster’s backup is the primary recovery.
3. Recover from the cluster’s storage. The cluster’s CSI snapshot is the secondary recovery.
4. Recover from the application’s replication. The application’s replication is the tertiary recovery.

The recovery is the cluster’s protection against the data loss. The production rule is to have a backup strategy for the data.

Quiz

Production discipline

• The drain is destructive for Pods with local data. The emptyDir volumes are deleted; the hostPath volumes are orphaned; the local PVs are preserved but unavailable.
• Use --delete-emptydir-data carefully. The flag is destructive; the operator should use it carefully.
• Use network storage for persistent data. The cluster’s network storage is preserved across the drain.
• Use emptyDir for temporary data. The emptyDir volume is deleted when the Pod is evicted; the data is intentionally lost.
• Design the workload to survive the drain. The design patterns are the operator’s responsibility.
• Audit the data at every release. The data is the cluster’s most valuable asset; the audit catches the failures.
• Test the drain in non-production. A staging cluster that mirrors production is the right place to test the drain.
• Monitor the data loss. The cluster’s alerts should fire on the data loss; the data is the cluster’s most valuable asset.