CIFS/SMB and directory storage · Proxmox VE

Directory and CIFS are the backends people add without a design review. One is “a path”, the other is “the share the Windows team already gave us”. Both are legitimate answers to real problems, and both have a specific way of going wrong that the other backends do not.

They belong together because CIFS is the directory backend, extended with the options needed to mount a share. Everything about layout, formats and snapshots applies to both.

The directory backend

One required property — path, an absolute filesystem path — and a predefined layout underneath it:

Content type	Subdirectory
VM images	`images/<VMID>/`
ISO images	`template/iso/`
Container templates	`template/cache/`
Backup files	`dump/`
Snippets	`snippets/`
Import	`import/`

That layout is why a directory storage’s volume IDs carry a path: local:141/vm-141-disk-1.qcow2 resolves to <path>/images/141/vm-141-disk-1.qcow2.

It is overridable. content-dirs takes a comma-separated list of vtype=path entries, each path relative to the storage mount point — useful when an existing share already has a layout you cannot change, or when you want backups on a different filesystem from images.

Configuration changea directory storage with a custom layout— Changes cluster configuration. content-dirs overrides where each content type lives beneath the storage path. Every path is relative to the storage mount point, not absolute.

pvesm add dir bulk-store \
--path /srv/bulk \
--content backup,iso,vztmpl \
--content-dirs backup=archives,iso=media/iso \
--prune-backups keep-daily=7,keep-weekly=4

pvesm status --storage bulk-store

Two more properties earn a decision:

preallocation — off, metadata, falloc or full, defaulting to metadata and treated as off for raw images. The documentation notes that on network storage with large qcow2 images, choosing off can help prevent timeouts, which is a real consideration on a slow NFS or CIFS target.

shared — as always, an assertion. On a directory backend it is almost always wrong; the exception is a directory whose path is a mount of something genuinely shared, in which case the shared backend’s own plugin is usually the better choice.

The failure that fills the root filesystem

This is the one to internalise, because it is silent, common, and the diagnosis looks nothing like the cause.

You add a disk, mount it at /srv/bulk, and add a directory storage pointing there. It works. Months later the node reboots, the mount fails — a changed UUID, a dead disk, an fstab typo, a systemd ordering problem — and /srv/bulk is now an ordinary empty directory on the root filesystem.

Proxmox VE does not notice. pvesm add defaults to create-base-path yes and create-subdirs yes, so the storage layer creates the directory tree it expects and carries on. Backups run. They write to /srv/bulk/dump on the root filesystem.

The backup job that filled /

A node with a 96 GB root and a 4 TB data disk. The data disk failed to mount after a firmware update.

The nightly backup job ran, wrote to what it believed was the data disk, and filled the root filesystem at about 3 a.m. Then everything that needs to write — pvedaemon, pveproxy, logging, /etc/pve operations that need local temp space — began failing in ways that read as a total node failure.

The alerts that fired were about the API being unreachable and about services restarting. Nothing said “backup went to the wrong place”, because from the storage layer’s point of view nothing went wrong: the path existed, it was writable, and the write succeeded.

The tell, once you look, is one command: df -h /srv/bulk showing the root filesystem rather than the data disk. That is worth putting at the top of any runbook for a node that has unexpectedly run out of root space.

Configuration changemake the storage fail instead of filling the root filesystem— Changes cluster configuration. With create-base-path and create-subdirs off, PVE will not create the tree, so if the filesystem is not mounted and the path is absent the storage fails to activate — a loud, correct failure. This only works if the mount point directory does not exist when unmounted.

pvesm set bulk-store --create-base-path 0 --create-subdirs 0

pvesm status --storage bulk-store

Read-only / Safeprove the storage is on the filesystem you think it is— Read-only. Three checks that take five seconds and answer the question no dashboard answers: is this path a mount point, which device backs it, and how much space does the storage layer believe it has. Run them on every node after any storage or boot change.

STORE_PATH=/srv/bulk

findmnt --target "$STORE_PATH"

df -h "$STORE_PATH"

pvesm status --storage bulk-store

Read-only / Safe

$ findmnt --target /srv/bulk

TARGET SOURCE    FSTYPE OPTIONS
/      /dev/mapper/pve-root ext4 rw,relatime,errors=remount-ro

The CIFS backend

CIFS is the directory backend plus a mount. Its extra properties:

Property	Meaning
`server`	IP or DNS name. An IP avoids DNS lookup delays
`share`	The share name. Discover them with `pvesm scan cifs <address>`
`username`	Defaults to `guest`
`password`	Stored at `/etc/pve/priv/storage/<STORAGE-ID>.pw`, readable only by root
`domain`	User domain or workgroup
`smbversion`	Defaults to `3`. SMB1 is unsupported for security reasons
`subdir`	A subdirectory of the share to use, defaulting to the share root
`path`	The local mount point, defaulting to `/mnt/pve/<STORAGE_ID>/`
`options`	Extra `mount.cifs` options; PVE already sets soft mode, credentials and domain

Configuration changeadd a CIFS storage for backups— Changes cluster configuration. The password is written to /etc/pve/priv/storage/<ID>.pw with root-only permissions, and is therefore replicated to every node through pmxcfs. Passing it on the command line means it also lands in your shell history.

pvesm add cifs backup-smb \
--server 192.0.2.40 \
--share backups \
--username svc-pve \
--domain EXAMPLE \
--smbversion 3 \
--subdir /site-a \
--content backup \
--prune-backups keep-daily=14,keep-monthly=6

pvesm status --storage backup-smb

Where the credential lives, and what that implies

The password goes to /etc/pve/priv/storage/<STORAGE-ID>.pw. Three consequences worth being deliberate about:

It is in /etc/pve, so it is replicated to every node in the cluster, and it is in whatever backs up /etc/pve. Anyone with root on any node has it.

It is a service account’s password, in reusable form. Give the CIFS storage its own account, scoped to that share, with no interactive login and no rights anywhere else. If the same account has access to the file server’s other shares, a compromised node is a compromised file server.

Rotation is a two-step change. Change it on the file server and in PVE at the same time, or the storage goes inactive between the two — which, if the storage holds backups, means a silently failed backup window. Schedule it, do not improvise it.

Snapshots, and what qcow2 actually buys

Neither backend has snapshots of its own. The documentation is direct: most filesystems do not support snapshots out of the box, so the directory backend uses qcow2’s internal snapshot capability instead; and CIFS does not support snapshots at the storage level, but qcow2 backing files give you snapshots and cloning.

So on both backends:

raw — no snapshots, no linked clones, best performance.
qcow2 — snapshots and clones, thin allocation, at a performance cost and with one important caveat.

Internal qcow2 snapshots block a running VM

The storage documentation states plainly that creating and deleting internal qcow2 snapshots blocks a running VM and is not an efficient operation.

On a 40 GB disk over a fast local filesystem, that pause is short enough to go unnoticed. On a 2 TB disk over a CIFS share on a busy file server, deleting a snapshot can stall the guest for long enough that an application clusters itself out, a health check fails, or a database replica is declared dead.

Two habits that follow:

Do not schedule snapshot deletion during business hours on file-backed storage, and never as part of an automated pre-backup step on a large guest.
Do not accumulate snapshots on qcow2. Each one adds work to the deletion and to every read, and the pause when you finally clean up is proportional to what you have kept.

If the workload needs frequent, cheap, non-blocking snapshots, the backend is wrong: ZFS, LVM-thin and RBD all do this without pausing the guest.

Under the hood

Why is CIFS acceptable for backups and questionable for running VM disks?

Three properties of the protocol, in increasing order of how much they matter.

Latency. Every I/O is a network round trip to a file server, and file protocols have more per-operation overhead than block protocols. Sequential streaming — which is what a backup is — hides this well. Small random I/O, which is what a database is, does not.

Soft mounts. PVE sets soft mode automatically for CIFS. A soft mount returns an error to the caller when the server does not respond, rather than blocking forever as a hard mount would. For a backup that is the right trade: a failed backup is better than a wedged process. For a running guest’s disk it means an I/O error surfaces inside the guest during a server hiccup, and guest filesystems respond to I/O errors by remounting read-only or worse.

Locking semantics. SMB’s locking is not POSIX locking, and its behaviour across implementations — a Windows server, a NAS appliance, Samba on Linux — varies. Proxmox VE’s own layer prevents two nodes opening the same guest disk, so this is not a routine hazard. It becomes one exactly where a disk image is reachable by something outside PVE’s coordination: another cluster pointed at the same share, a NAS-side deduplication or antivirus process, or a well-intentioned backup agent on the file server.

The summary that holds up: CIFS is a good target for content that is written once and read rarely — backups, ISOs, container templates. It is a workable target for VM disks where the workload is light and the alternative is nothing. It is a poor target for anything latency-sensitive, and the failure mode is inside the guest rather than in PVE, which makes it hard to attribute.

When a file-backed backend is the right answer

Right:

ISOs and container templates. Shared, rarely written, and every node needs them. A CIFS or NFS share is exactly correct and needs no more thought.
Snippets. Cloud-Init and hook scripts, which must be on a storage with the snippets content type and are tiny.
A secondary backup target. A CIFS share on separate hardware is a real second copy, and the 3-2-1 argument in Part XIII cares more about it being separate than about it being fast.
Local scratch on one node. Import staging, a one-off conversion, a test guest.

A shortcut you will regret:

Production VM disks on CIFS, chosen because the share existed. The latency and the soft-mount error behaviour will find you.
A directory storage on the root filesystem with no capacity alarm. local ships this way and it is fine for ISOs; it becomes an outage when someone points backups at it.
shared 1 on a directory storage to enable migration. This is the silent data-divergence failure from the storage model lesson, in its most common form.
A directory storage as the primary backup target when PBS is available. PBS gives deduplication, incremental backup and verification; a directory of vzdump archives gives none of them.

What happens if the CIFS server goes away while backups are running?

During the backup. With soft mode, the operation fails rather than hanging indefinitely. The vzdump task ends in error, the task log names the storage, and the notification system reports a failed job — assuming notifications are configured, which is the assumption worth checking now rather than then.

Afterwards. The mount stays in a broken state and pvesm status shows the storage inactive. PVE re-attempts the mount, so a server that comes back usually recovers by itself. A server that comes back with a changed password, a changed share name or a different SMB version does not, and the error will say permission denied rather than anything about the change.

The failure you have to look for. Backups that fail loudly are the good case. The bad case is a backup job whose target storage has been inactive for three weeks while nobody read the notification. Backup failure alerting is necessary and not sufficient: what you actually need is an alert on the age of the newest successful backup per guest, which fires whether the job failed, never ran, or ran against nothing.

Recovery steps in order:

pvesm status --storage <id> on the affected node, to confirm inactive rather than slow.
findmnt /mnt/pve/<id> to see whether a stale mount is present, and umount it if the server is confirmed gone.
Test the credentials independently — pvesm scan cifs <server> --username <u> --domain <d> will prompt and will tell you whether it is authentication or reachability.
Re-check smbversion. A file server hardened to require SMB 3.1.1, or one that has disabled the dialect PVE negotiated, fails as an authentication or protocol error rather than as a version message.
Once recovered, verify a restore, not just that the storage is back. A storage that mounts is not evidence that the archives on it are complete.

Key takeaways

CIFS is the directory backend plus a mount, so the layout, formats and snapshot behaviour are the same for both.
The standard layout is images/<VMID>/, template/iso/, template/cache/, dump/, snippets/, import/, and content-dirs overrides it with vtype=path entries relative to the storage mount point.
create-base-path and create-subdirs default to yes, which is why an unmounted filesystem results in PVE creating the tree on the root filesystem and writing there silently. Set both to 0 where the path only exists when mounted, and check with findmnt --target.
CIFS credentials live at /etc/pve/priv/storage/<STORAGE-ID>.pw, root-only but replicated to every node. Use a dedicated, narrowly scoped service account, and treat rotation as a coordinated change.
smbversion defaults to 3 and SMB1 is unsupported for security reasons.
Neither backend has storage-level snapshots. qcow2 supplies them, and creating or deleting an internal qcow2 snapshot blocks a running VM, so do not accumulate them and do not delete them during business hours.
PVE sets CIFS mounts to soft mode: a server hiccup produces an I/O error inside the guest rather than a hang. Good for backups, bad for disks.
Alert on the age of the newest successful backup, not only on job failure.

Knowledge check

Why this matters

These two backends do not fail interestingly. Nothing crashes, no daemon restarts, no cluster event appears in the log. A path exists and is writable, so the write succeeds — into the wrong filesystem, or onto a share nobody has checked in a month.

That makes them a good exercise in a habit worth generalising: verify the thing, not the configuration. pvesm status tells you what PVE believes. findmnt --target tells you what the kernel believes. The age of the newest successful backup tells you what actually happened. Those three answer different questions, and only the last one is about outcomes.

Directory and CIFS are genuinely useful and there is no reason to avoid them. They just need the one check the block backends give you for free — that the storage is where you think it is.