Ephemeral and persistent storage - the cloud storage model

Cloud storage has two main types: ephemeral (lost on termination) and persistent (survives termination). This lesson covers the tradeoffs and the patterns.

Ephemeral storage

Ephemeral storage is lost when the instance terminates:

• EC2 instance store: local SSD on the host. Fast but ephemeral.
• Container writable layer: in containers, the writable layer is ephemeral.
• tmpfs: RAM-backed filesystem. Very fast, very ephemeral.

Use for:

• Caches (Redis, etcd).
• Temporary processing.
• Buffers and queues (with persistence elsewhere).
• Anything that can be rebuilt from persistent storage.

Persistent storage

Persistent storage survives instance termination:

• EBS volumes (AWS): block storage attached to EC2.
• Azure Disks: block storage for Azure VMs.
• Persistent Disk (GCP): block storage for GCP.
• Object storage (S3, GCS, Azure Blob): for files, backups, large data.

Persistent storage is the right choice for:

• Databases.
• Application state.
• User uploads.
• Configuration that must survive restart.

Block storage lifecycle

Creating and attaching a volume is an API call. Formatting it is a local, irreversible one. Keep the two steps separated by an identification step.

# 1. Create a volume. Record the VolumeId it returns
aws ec2 create-volume --size 100 --availability-zone us-east-1a

# 2. Attach it. --device is a request, not a guarantee
aws ec2 attach-volume --volume-id vol-0123456789abcdef0 \
  --instance-id i-0abcdef1234567890 --device /dev/sdf

Resolve the volume ID to a device before touching it. The NVMe serial is the volume ID with the dash removed, and the kernel publishes a stable symlink built from it:

# 3. Positively identify the device behind vol-0123456789abcdef0
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS,SERIAL
sudo nvme list                       # SN column = volume ID, dash removed

ls -l /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0123456789abcdef0
# -> ../../nvme1n1   (today; do not hard-code that)

The other providers publish the same kind of stable path. Azure maps the LUN you passed to az vm disk attach under /dev/disk/azure/scsi1/lun0. GCP maps the device name you passed to gcloud compute instances attach-disk under /dev/disk/by-id/google-DEVICE_NAME. In all three cases the stable path is derived from the identifier you supplied to the API, so it cannot drift the way a kernel enumeration order can.

# 4. Bind the stable path to a variable, then refuse to format
#    anything that is not empty
DEV=/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0123456789abcdef0

if sudo blkid "$DEV"; then
  echo "REFUSING: $DEV already carries a filesystem signature"
elif [ -n "$(lsblk -no MOUNTPOINTS "$DEV" | tr -d ' \n')" ]; then
  echo "REFUSING: $DEV is mounted"
else
  sudo mkfs.xfs "$DEV"
fi

blkid prints nothing and exits non-zero on a genuinely blank device, so the if only proceeds when the device is empty. mkfs.xfs has its own safety net - without -f it refuses a device where it suspects a filesystem or a partition table - but cryptsetup luksFormat only asks a generic confirmation that it asks for every device, and -q suppresses even that. Neither tool can tell you that you picked the wrong device. Only step 3 can.

# 5. Mount by UUID, never by device node
UUID=$(sudo blkid -s UUID -o value "$DEV")
sudo mkdir -p /mnt/data
echo "UUID=$UUID /mnt/data xfs defaults,nofail 0 0" | sudo tee -a /etc/fstab
sudo systemctl daemon-reload
sudo mount /mnt/data

nofail keeps the instance bootable if the volume is detached. The fsck pass is 0 because XFS does not use fsck at boot. The UUID is written to the filesystem at mkfs time and travels with the data, so it survives the renumbering that broke the device node.

The volume survives instance termination. The data is preserved.

Snapshots

Block storage has snapshot support:

# AWS EBS snapshot
aws ec2 create-snapshot --volume-id vol-xxx --description "before-upgrade"

# Restore from snapshot
aws ec2 create-volume --snapshot-id snap-xxx --availability-zone us-east-1a

Snapshots are point-in-time copies. They are incremental (only changed blocks are stored). For databases, use application-consistent snapshots.

Encryption

Encrypt block storage at rest:

# AWS EBS encryption. Set at creation; it cannot be turned on in place
aws ec2 create-volume --encrypted --kms-key-id alias/my-key \
  --size 100 --availability-zone us-east-1a

LUKS goes on the same positively identified device, and behind the same guard. Never point luksFormat at a bare /dev/nvme* node:

DEV=/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0123456789abcdef0

# Confirm the symlink still resolves, and that the target is empty
readlink -f "$DEV"

if sudo blkid "$DEV"; then
  echo "REFUSING: $DEV already carries a filesystem signature"
else
  sudo cryptsetup luksFormat "$DEV"
fi

sudo cryptsetup open "$DEV" mydata
sudo mkfs.xfs /dev/mapper/mydata

# Back the header up immediately, before any data is written
sudo cryptsetup luksHeaderBackup "$DEV" \
  --header-backup-file /root/luks-header-$(date +%F).img
sudo chmod 600 /root/luks-header-$(date +%F).img

Copy that header image somewhere off the instance, and treat it as a secret. It contains the keyslot area, so anyone holding both the image and a passphrase can decrypt the volume - but without it a corrupted header means the data is gone regardless of how many passphrases you know. The trade is worth making; the storage location is what needs the care.

Cloud-provider encryption is at the storage layer. LUKS is at the OS layer. Both are recommended for sensitive data.

Knowledge check