iSCSI: targets, initiators, and multipath for redundancy

iSCSI is the shared-storage workhorse for enterprise environments that need block-level access. A VM sees iSCSI as a local disk; in reality, its I/O traverses a TCP network to a storage array. With multipath, two network paths to two storage controllers give you both bandwidth and redundancy.

This lesson walks through a real iSCSI setup end-to-end: target configuration, initiator connection, multipath, and the operational gotchas that matter at scale.

Mental model

iSCSI has three layers:

1. Target — the storage server. Exposes one or more LUNs (logical units) over the network. Each LUN looks like a block device.
2. Initiator — the consumer (your PVE node). Connects to the target, sees the LUN as a local /dev/sdX device.
3. IQN — iSCSI Qualified Name. Every target and initiator has a unique IQN for authentication. Format: iqn.YYYY-MM.reverse-domain:identifier.

Authentication is per-target using CHAP (Challenge-Handshake Authentication Protocol). The initiator proves it knows a shared secret; the target authenticates back. Mutual CHAP authenticates both sides.

Target configuration with targetcli

targetcli is the standard Linux iSCSI target management tool:

apt install -y targetcli-fb
systemctl enable --now target
targetcli

Inside the targetcli shell, configure a single LUN with two paths:

# Create a block backend (the actual storage)
/backstores/block create name=disk1 dev=/dev/sdb

# Create an iSCSI target (the network endpoint)
/iscsi create iqn.2024-01.com.example:pve.target01

# Create a LUN under the target
/iscsi/iqn.2024-01.com.example:pve.target01/tpg1/luns create \
  name=lun0 storage_object=/backstores/block/disk1

# Allow the PVE initiator to connect
/iscsi/iqn.2024-01.com.example:pve.target01/tpg1/acls create \
  iqn.2024-01.com.example:pve.node01

# Set a CHAP secret
/iscsi/iqn.2024-01.com.example:pve.target01/tpg1/acls/iqn.2024-01.com.example:pve.node01 \
  set auth userid=node01 password=secretpasswordhere

# Enable the target port
/iscsi/iqn.2024-01.com.example:pve.target01/tpg1/portals create 0.0.0.0

# Save and exit
saveconfig
exit

The ACL iqn.2024-01.com.example:pve.node01 must match the initiator’s IQN on the PVE node (see below). Without an ACL, the target rejects all connections.

For multipath, run a second target portal on a different IP:

/iscsi/iqn.2024-01.com.example:pve.target01/tpg1/portals create 10.0.5.10

Both portals serve the same LUN; the initiator connects to both and spreads I/O.

Initiator configuration

On each PVE node:

apt install -y open-iscsi multipath-tools

# Set the initiator IQN
echo "InitiatorName=iqn.2024-01.com.example:pve.node01" > /etc/iscsi/initiatorname.iscsi

# Discover targets
iscsiadm -m discovery -t sendtargets -p 10.0.4.10
iscsiadm -m discovery -t sendtargets -p 10.0.5.10

# Log in to both targets with CHAP
iscsiadm -m node -T iqn.2024-01.com.example:pve.target01 -p 10.0.4.10 \
  --op update -n node.session.auth.authmethod -v CHAP
iscsiadm -m node -T iqn.2024-01.com.example:pve.target01 -p 10.0.4.10 \
  --op update -n node.session.auth.username -v node01
iscsiadm -m node -T iqn.2024-01.com.example:pve.target01 -p 10.0.4.10 \
  --op update -n node.session.auth.password -v secretpasswordhere

iscsiadm -m node -T iqn.2024-01.com.example:pve.target01 -p 10.0.5.10 \
  --op update -n node.session.auth.authmethod -v CHAP
# ... repeat for both portals

# Log in
iscsiadm -m node -T iqn.2024-01.com.example:pve.target01 -p 10.0.4.10 --login
iscsiadm -m node -T iqn.2024-01.com.example:pve.target01 -p 10.0.5.10 --login

# Make sessions persistent across reboots
iscsiadm -m node -T iqn.2024-01.com.example:pve.target01 -p 10.0.4.10 \
  --op update -n node.startup -v automatic
iscsiadm -m node -T iqn.2024-01.com.example:pve.target01 -p 10.0.5.10 \
  --op update -n node.startup -v automatic

# Make sessions survive network blips (replace=immediate)
iscsiadm -m node -T iqn.2024-01.com.example:pve.target01 -p 10.0.4.10 \
  --op update -n node.session.timeo.replacement_timeout -v 5
iscsiadm -m node -T iqn.2024-01.com.example:pve.target01 -p 10.0.5.10 \
  --op update -n node.session.timeo.replacement_timeout -v 5

After this, lsblk should show the same LUN twice (or more if multipath is active):

lsblk
# NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
# sda           8:0    0   500G  0 disk
# sdb           8:16   0   500G  0 disk
# Both /dev/sda and /dev/sdb point to the same underlying LUN

Multipath configuration

Multipath aggregates the two physical paths into one logical device:

# /etc/multipath.conf
defaults {
  user_friendly_names yes
  path_grouping_policy multibus
  path_selector "round-robin 0"
  failback immediate
  no_path_retry 5
  rr_min_io 100
}

# Blacklist the local disk so multipath doesn't claim it
blacklist {
  devnode "sda$"
}

# Per-device settings for the iSCSI LUN
devices {
  device {
    vendor "LIO-ORG"
    product "disk1"
    path_grouping_policy multibus
    path_selector "round-robin 0"
    path_checker tur
    features "1 queue_if_no_path"
  }
}

Reload:

systemctl restart multipathd
multipath -ll
# Expect: two paths (10.0.4.10 and 10.0.5.10) both active

The VM sees one device, typically /dev/mapper/mpathN. PVE references this in storage configuration:

pvesm add iscsi shared-iscsi --portal 10.0.4.10,10.0.5.10 \
  --target iqn.2024-01.com.example:pve.target01 \
  --content images,rootdir

Performance tuning

iSCSI performance depends on four knobs:

1. MTU / jumbo frames

iSCSI benefits hugely from MTU 9000 on the storage network. Each request carries more data per round trip.

# On the storage network interface
ip link set ens4f0 mtu 9000

# Verify
ip link show ens4f0

Both initiator and target must have MTU 9000, and all switches in between must support and be configured for jumbo frames.

2. Multiple sessions per LUN

With multipath, you get N paths. To get N× throughput on a single LUN, increase the number of sessions per path:

# In /etc/iscsi/iscsid.conf
node.session.nr_sessions = 4

This creates 4 sessions per target portal, each with its own TCP connection. With 2 portals and 4 sessions each, you have 8 sessions — and 8× the queue depth available to the VM.

3. Queue depth on the initiator

# /etc/iscsi/iscsid.conf
node.session.iscsi.MaxQueueDepth = 128

The default 32 is too low for VM workloads. 128 matches typical NVMe queue depths.

4. Read-ahead on the LUN

For sequential workloads (backups, large file copies):

# Set read-ahead to 1024 sectors (512 KB) or higher
blockdev --setra 1024 /dev/mapper/mpath0

For random workloads (database VMs), leave it at default.

Adding the LUN to Proxmox

After the LUN is visible on every node via multipath, add it to PVE storage:

# Create a filesystem on the LUN
mkfs.xfs /dev/mapper/mpath0

# Mount it temporarily to verify
mount /dev/mapper/mpath0 /mnt

# Add as a PVE storage backend
pvesm add dir shared-iscsi --path /mnt/pve/shared-iscsi \
  --content images,rootdir --is_mountpoint yes

# Or for direct LUN use
pvesm add lvm shared-lvm --vgname vg-shared /dev/mapper/mpath0

For a cluster-wide LUN that hosts VMs shared between nodes (live migration requires shared storage), LVM is the right layer:

pvesm add lvm shared-lvm --vgname vg-shared /dev/mapper/mpath0 --shared yes

Diagnostics

When iSCSI is slow or dropping paths:

# Active sessions
iscsiadm -m session -P 1

# Session stats (per-connection latency, throughput)
iscsiadm -m session -s | grep -E 'tmo|retransmit|stats'

# Per-path I/O stats
iostat -dx 1 /dev/mapper/mpath0
ls -la /dev/disk/by-path/

# Multipath state
multipath -ll
multipathd -k'show status'
multipathd -k'remap paths'

# Recent kernel errors
dmesg --since '1 hour ago' | grep -i 'iscsi\|multipath\|connection'

Production considerations

• Always use multipath. Single-path iSCSI is a single point of failure: one cable, one switch port, one storage controller. With two of each, you survive any single failure.
• Tune the storage network, not the management network. iSCSI benefits from MTU 9000, dedicated VLAN, and a separate physical switch. Mixing iSCSI with the management network means a noisy backup saturates the link and stalls VM I/O.
• Watch queue depth. With many VMs on one LUN, the queue depth per VM matters. 128 is a good default; lower it to 32 if you see latency spikes.
• Test path failover. Pull one cable and confirm the VM keeps running. Pull the other. Verify both paths can fail independently without I/O errors.

Common mistakes

• Single-path iSCSI in production. The link WILL fail. The failover WILL be slow. The VM WILL freeze.
• Forgetting nr_sessions. With 1 session per target portal and default queue depth, a busy VM can saturate the queue and stall.
• No CHAP configured. Anyone with network access can read your VM disks. Even on a private VLAN, set up CHAP.
• Sharing the management network. iSCSI traffic competes with the GUI and the cluster corosync traffic. Separate network or QoS.

Key takeaways

• iSCSI target on the storage server, initiator on PVE, CHAP for auth.
• Multipath is mandatory for production: two paths, two portals, one logical device.
• MTU 9000 + multiple sessions per LUN = the throughput you expect.
• Test failover regularly: pull a cable, verify the VM keeps running.

Knowledge check