Attach failures — multipath, device limits, and the kernel-level diagnostics

Volume attach failures are a node-level storage incident that affects the Pod’s ability to mount the volume. The cause is one of several categories: the backend API, the instance type, AZ mismatch, multipath, or kernel-level device limits. This lesson walks the diagnostic.

The attach operation

Attach is ControllerPublishVolume (the CSI controller operation). The kube-controller-manager calls it; the CSI controller plugin executes it; the volume becomes visible to the node at the kernel level.

For cloud-block storage:

• EBS: ec2:AttachVolume API call; the EBS volume is attached to the EC2 instance.
• GCE PD: disks.attach API call; the PD is attached to the GCE instance.
• Ceph RBD: rbd map command; the RBD image is mapped to a block device on the node.

The attach can fail at any stage: the API call, the backend processing, the kernel’s device registration.

Cause 1: multipath misconfiguration

Multipath is the kernel’s mechanism for handling multiple paths to the same device. Cloud-block storage (EBS, GCE PD) can have multiple paths via the storage networking; multipath consolidates them into one device.

# Check multipath status
multipathd show maps status
# or
multipath -ll

If multipath is misconfigured:

• The device may not appear at the expected path (/dev/xvdba vs /dev/mapper/mpath0).
• The kubelet cannot find the device; the mount fails.

The production rule:

• EBS: enable multipath only if you need it (EBS multi-attach). For single-attach, multipath can cause confusion.
• Configure /etc/multipath.conf with the vendor-specific settings.

Cause 2: device limits

The kernel has limits on the number of block devices:

# Check the kernel's device limit
cat /proc/sys/fs/file-max
cat /sys/block/*/queue/max_segments

A node that has many volumes attached can hit these limits. New attaches fail with “no space left on device” or “device busy.”

The fix:

• Increase the kernel limits (requires node reboot or tuning).
• Reduce the number of volumes per node (via Pod density controls).
• Use a StorageClass that supports more volumes per node (e.g., instance store for ephemeral).

Cause 3: instance type does not support the volume

Some instance types do not support all volume types:

• EBS: some instance types do not support io2, or have a maximum number of EBS volumes per instance.
• GCE PD: some instance types have a maximum number of PDs.

# AWS: check instance type limits
aws ec2 describe-instance-type-offerings \
  --location-type availability-zone \
  --filters Name=instance-type,Values=m5.large

The fix: use a different instance type, or use a StorageClass that the instance type supports.

Cause 4: AZ mismatch

The volume is in a different AZ than the instance:

# AWS: the volume's AZ
aws ec2 describe-volumes --volume-ids vol-xxx \
  --query 'Volumes[0].AvailabilityZone'
# us-east-1a

# The instance's AZ
aws ec2 describe-instances --instance-ids i-xxx \
  --query 'Reservations[0].Instances[0].Placement.AvailabilityZone'
# us-east-1b

The fix: use WaitForFirstConsumer so the volume is created in the Pod’s AZ.

The diagnostic ladder

flowchart TD
    A[Attach failure] --> B[kubectl describe pod]
    B --> C{kubelet events?}
    C -->|FailedAttach| D[CSI controller logs]
    C -->|kubelet attach error| E[kernel logs: dmesg]
    D --> F{API error?}
    F -->|yes| G[Backend issue: API, IAM, quota]
    F -->|no| H{multipath issue?}
    H -->|yes| I[Configure multipath]
    H -->|no| J[Device limits]
    E --> K{Kernel error?}
    K -->|yes| L[Kernel-level issue]

The production fix

For each cause:

Quiz

Production discipline

• Attach failures are usually API or kernel issues. The diagnostic checks both.
• Multipath is needed only for multi-attach. Misconfigure is a common cause of attach failures.
• Device limits are a SLO. A node that hits the limit cannot attach more volumes.
• AZ mismatch is the most common cause. WaitForFirstConsumer is the fix.
• Document the instance type’s limits. EBS volume count, EBS volume type, AZ.