Network and shared storage — NFS, CIFS, Ceph, iSCSI · Docker & Containers

Local volume storage on a single host cannot survive host failure. For stateful workloads that need to survive, the volume must live on shared storage. The standard choices are NFS, CIFS, Ceph, and iSCSI.

Before any of them: how Docker actually gets there.

It is `mount(8)`, not a volume driver

What --opt type, o and device really do

nfs, cifs, ceph, rbd and iscsi are not Docker volume driver names. There is no --driver nfs. What exists is the built-in local driver, which accepts three options and hands them to mount(8) on the host:

Option	Becomes
`--opt type=nfs`	`mount -t nfs`
`--opt o=addr=...,rw`	`mount -o addr=...,rw`
`--opt device=:/srv/export`	the mount source argument

That is the whole mechanism. The consequence is important in both directions.

Anything the host kernel can mount, the local driver can mount. If mount -t ceph works on the host, --opt type=ceph works in a volume. Docker documents nfs and cifs; the rest is a property of your kernel, not of Docker.

Anything the host kernel cannot mount does not become possible by writing it in a volume option. There is no type=iscsi and no type=rbd, because neither iSCSI nor RBD is a filesystem. They are block transports. You log in to the target, a block device appears, you put a filesystem on it, and that is what gets mounted.

The mount happens lazily: docker volume create with these options validates almost nothing. The first docker run that uses the volume is when mount(8) is called and when a wrong address, a missing export or a bad credential finally surfaces.

Read-only / Safeprove it by hand first— Always mount it manually on the host before wrapping it in a volume. The error messages are far better.

NFS_SERVER=192.0.2.10
NFS_EXPORT=/srv/exports/appdata

mkdir -p /mnt/nfs-test
mount -t nfs -o "addr=$NFS_SERVER,rw,nfsvers=4.2" "$NFS_SERVER:$NFS_EXPORT" /mnt/nfs-test

findmnt -no FSTYPE,OPTIONS /mnt/nfs-test
umount /mnt/nfs-test

NFS

Read-only / Safenfs volume— Creating the volume touches nothing. The mount is attempted on first use.

NFS_SERVER=192.0.2.10
NFS_EXPORT=/srv/exports/appdata

docker volume create --driver local --opt type=nfs --opt o="addr=$NFS_SERVER,rw,nfsvers=4.2,hard,timeo=600,retrans=2" --opt device=":$NFS_EXPORT" appdata-nfs

Note the leading colon on device. The mount -t nfs source is host:/path, and because the address is already in o=addr=, the host part is left empty — :/srv/exports/appdata. Omitting that colon is the most common reason a freshly created NFS volume fails on first use.

The mount options that decide how failures behave

Option	Effect	When you want it
`hard` (default)	Requests retry indefinitely	Almost always
`soft`	Requests fail after `retrans` retries	Almost never — see below
`nfsvers=4.2`	Pin the protocol version	Always, so a server change cannot silently downgrade you
`timeo=600`	Retransmit timeout in deciseconds (60 s)	With `hard`, to slow the retry storm
`nolock`	Disable NLM locking	Only when you have read the warning below
`noac`	Disable attribute caching	Rarely; large correctness-for-throughput trade
`ro`	Read-only	Whenever the workload does not write

If nfsvers is unspecified the client tries 4.2 and negotiates down until the server agrees. That is convenient and it means a server maintenance window can leave you on NFSv3 without anyone noticing, which changes the locking model underneath a running workload. Pin it.

soft can corrupt data silently, and nolock removes the protection you think you have

Two options in that table are load-bearing and both are routinely copied from a blog post without their warnings.

soft. The nfs(5) man page states it plainly: “A so-called ‘soft’ timeout can cause silent data corruption in certain cases.” A soft mount returns an error to the application after a few retries. An application that treats a failed write as transient, or that does not check the return value of write(2) at all, carries on with a file that has a hole in it. hard hangs instead, and a hang is a symptom you can see. Choose the visible failure.

nolock. Again from nfs(5): with nolock, “applications can lock files, but such locks provide exclusion only against other applications running on the same client.”

Read that twice. The locks still succeed. flock returns 0. fcntl(F_SETLK) returns 0. Nothing errors. They simply do not coordinate with any other host. Every safety mechanism your application relies on to avoid two writers is still reporting success while providing nothing.

nolock is a legitimate option in exactly one common case — the nfs(5) note that NLM locking must be disabled when mounting /var over NFS — and it is a catastrophe anywhere a second machine might open the same file.

Worth knowing where this spreads from: the NFS example in Docker’s own Compose volume reference reads o: "addr=10.40.0.199,nolock,soft,rw". It is an illustration of the driver_opts syntax, not a recommendation for a mount option set, but it is the string most people copy first. Read the options you paste.

Two containers, one database file, one NFS export

This is the incident this lesson exists for, and it looks completely reasonable in a Compose file.

A service uses SQLite, or a Lucene index, or a bare BoltDB file. It runs on an NFS-backed volume so it survives the host. It works. Then you scale it to two replicas on two hosts for availability, and both open the same file.

What happens next is not a crash. Both instances start. Both serve traffic. Both cache pages of that file in their own memory, because the NFS client caches attributes and the database caches its own pages on top. Each writes back what it believes the file contains. The final state is whichever process flushed last, interleaved at page granularity.

The symptom arrives days later as a corrupt index, a row that reverted, or a database disk image is malformed. By then the backups contain the corruption too.

Why the first diagnosis is usually wrong: everyone looks at the network. The NFS server is healthy, latency is fine, there are no errors in dmesg and none in the application log — because nothing failed. The storage did exactly what it was asked. The bug is that two processes were asked.

Single-writer file databases are single-writer. If you need two replicas, you need a database that speaks a network protocol (Postgres, MySQL) or a coordination layer, not a shared file. And if you genuinely have one writer, prove it: a lease, a lock file you actually verify, or a scheduler constraint that pins the service to one host.

CIFS / SMB

For Windows file servers, or Linux hosts sharing storage with Windows.

Read-only / Safecifs volume— Credentials in a command line are visible in shell history and to any process reading /proc. Use a credentials file.

SMB_HOST=198.51.100.20
SMB_SHARE=appdata

docker volume create --driver local --opt type=cifs --opt o="addr=$SMB_HOST,credentials=/etc/docker-smb.creds,vers=3.1.1,uid=1001,gid=1001,file_mode=0640,dir_mode=0750" --opt device="//$SMB_HOST/$SMB_SHARE" appdata-cifs

CIFS differs from NFS in a way that matters here: the server does not carry Unix ownership, so uid, gid, file_mode and dir_mode are client-side fictions applied to everything on the mount. Every file appears owned by whatever uid= you passed. This neatly sidesteps the UID mismatch problem from the permissions lesson — and it means the permission bits you see are not enforcing anything on the server, which is a different problem if you were relying on them.

Put the credentials in a root-owned 0600 file, never in o=:

username=svc-appdata
password=REPLACE_ME
domain=EXAMPLE

Ceph and iSCSI: block storage reaches a container differently

Neither of these is a volume option, and treating them as one is the error the shape of the CLI invites.

Ceph RBD. The host maps the image to a block device, you put a filesystem on it once, and the container gets a filesystem:

Destructiverbd map— mkfs is run ONCE, on first provisioning. Running it again destroys the image contents.

POOL=containers
IMG=appdata

# Host side: map the RBD image to a block device
rbd map "$POOL/$IMG" --name client.docker
DEV=$(rbd showmapped --format json | python3 -c 'import json,sys;print([m["device"] for m in json.load(sys.stdin).values()][0])')

# ONCE, at provisioning time only:
# mkfs.ext4 "$DEV"

mkdir -p /mnt/appdata
mount "$DEV" /mnt/appdata

The container then bind-mounts /mnt/appdata, or you wrap the already-mounted path in a local volume with type=none and o=bind.

iSCSI. Same shape, one layer lower: iscsiadm discovers and logs in to the target, a /dev/sd* appears, multipath aggregates the paths, and you mount a filesystem on it.

The operational point for both: the block device is exclusive. An RBD image or an iSCSI LUN with a non-clustered filesystem (ext4, xfs) mounted read-write on two hosts at once is not a sharing model, it is immediate corruption — worse than the NFS case, because there is no network filesystem layer even attempting coherence. Clustered filesystems (GFS2, OCFS2) exist for exactly this and bring a cluster manager with them.

CephFS is the file-level Ceph interface, and because the host kernel can mount it, the local driver can too:

docker volume create --driver local \
  --opt type=ceph \
  --opt o=name=docker,secretfile=/etc/ceph/docker.secret \
  --opt device=192.0.2.30:6789:/volumes/appdata \
  appdata-cephfs

Docker does not document type=ceph; it works because mount -t ceph works on that host. Verify it by hand first, as above, and treat the kernel client version as part of your compatibility matrix.

Choosing

Storage	Sharing model	Good for	Bad for
Local NVMe / SSD	None	Databases, anything latency-sensitive	Surviving host loss
NFSv4.2	Many readers and writers, real locking	Uploads, exports, shared config, media	Single-writer file databases
CIFS/SMB 3.x	Many readers and writers, oplocks	Windows interop, document shares	POSIX semantics, high IOPS
CephFS	Many readers and writers, POSIX	Shared files with no single server	Small-cluster simplicity
Ceph RBD	One host at a time	Databases needing HA and failover	Concurrent multi-host mounts
iSCSI	One host at a time (without a clustered FS)	Enterprise SAN integration	Concurrent multi-host mounts

The column that matters is the middle one, not throughput. Pick the sharing model your workload’s consistency requirements need, then worry about performance.

What happens when the NFS server disappears mid-write?

With the default hard mount, the container’s I/O blocks. Not errors — blocks. The process goes into uninterruptible sleep (state D) and stays there until the server returns.

The cascade is what makes this a page:

The process cannot be killed. SIGKILL does not interrupt uninterruptible sleep, so docker stop waits its ten seconds and then docker kill also does nothing. The container sits in Removal In Progress indefinitely.
Health checks time out, so the orchestrator tries to restart the container it cannot stop.
Anything that walks the filesystem hangs with it. df, a monitoring agent collecting disk metrics, a backup job, ls in the wrong terminal — each one adds another stuck process. Hosts that have “run out of processes” during an NFS outage got there this way.
docker ps itself can hang if the daemon touches the mount.

The diagnosis is ps state D plus a stack that names NFS:

ps -eo pid,stat,wchan:32,comm | awk '$2 ~ /D/'

and cat /proc/$PID/stack on one of them.

Recovery is to bring the server back, or umount -f -l the mount point so the kernel stops waiting. Nothing else works, and the processes only exit after the mount goes away.

The temptation is to switch to soft so it fails fast. Do not. soft converts a visible hang into invisible corruption. If you need to fail fast, use hard with a short timeo and retrans, keep the mount out of the path of monitoring agents, and design the application to be restarted rather than to survive.

Verification that can fail

Read-only / Safeverify a network volume— Confirms the mount really happened, with the options you asked for, before the workload depends on it.

VOL=appdata-nfs

# 1. Does it mount at all? (Volume creation does not test this.)
docker run --rm -v "$VOL":/data:ro alpine:3.20 ls /data

# 2. Are the options the ones you set? findmnt reads the kernel, not your intent.
findmnt -t nfs,nfs4,cifs,ceph -no TARGET,SOURCE,OPTIONS | grep -F "$(docker volume inspect "$VOL" --format '{{.Mountpoint}}')"

# 3. Is it actually writable end to end?
docker run --rm -v "$VOL":/data alpine:3.20 sh -c 'dd if=/dev/zero of=/data/.iotest bs=1M count=8 conv=fsync && rm /data/.iotest'

Read-only / Safefindmnt on a working NFS volume— What the kernel recorded, not what you asked for. Read vers= and hard here, every time.

$ findmnt -t nfs4 -no TARGET,SOURCE,OPTIONS

/var/lib/docker/volumes/appdata-nfs/_data 192.0.2.10:/srv/exports/appdata rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,local_lock=none,addr=192.0.2.10

Illustrative output

Step 2 is the one people skip and the one that catches the real errors: an o= string with a typo in an option name is silently ignored by some filesystem types, so a mount you believe is hard and nfsvers=4.2 can be neither. findmnt shows what the kernel recorded, which is the only version that counts.

In that output, vers=4.2 and hard are the two tokens to look for. If vers=3 appears, the client negotiated down and the locking model is not the one you designed against. If soft appears, or local_lock is anything other than none, someone pasted an option set without reading it.

Step 3 uses conv=fsync deliberately. Without it the write lands in the page cache and returns success even when the server is unreachable.

Knowledge check

Why this matters

Multi-host production almost always needs shared storage, and the choice is usually made by what infrastructure already exists. That part is fine.

What goes wrong is that “shared storage” gets read as “storage that can be shared by anything”. It cannot. NFS and CephFS share files between many writers and have a locking protocol to make that safe; RBD and iSCSI hand one host a block device and corrupt immediately if two take it. And a single-writer file database — SQLite, an index file, an embedded key-value store — is single-writer no matter how capable the storage underneath it is.

Decide the sharing model before the protocol, mount it by hand before you wrap it in a volume, and check with findmnt that the options you asked for are the options the kernel recorded.

It is mount(8), not a volume driver

NFS