← All break/fix scenarios in Proxmox VE
iSCSI sessions keep dropping and LVM storage goes read-only
Reported symptoms
- ●VMs on iSCSI-backed storage freeze for several seconds, then recover
- ●dmesg shows "connection login retries" and "session reopens"
- ●multipath -ll shows paths alternating between "active" and "failed"
- ●LV on the iSCSI LUN may be marked as read-only by the kernel
Evidence
- · dmesg | grep -i iscsi shows session timeouts and NOP-Out failures
- · iscsiadm -m session -P 1 shows transient "abs" (absent) state
- · iptables -L -n | grep 3260 (target portal port) shows no rules
- · Switch port counters show no errors (so not physical)
Diagnosis and resolutionclick to reveal
Root cause
The iSCSI session is being terminated and re-established repeatedly. Common causes: - Firewall (iptables / nftables) blocking iscsi port 3260 - TCP window scaling issue between initiator and target - MTU mismatch (jumbo frames) - Target portal overload - Path down but only one path configured (no multipath)
Remediation
1. Check current session state: `iscsiadm -m session -P 1` 2. If using multipath, confirm both paths are configured: `multipath -ll` (should show 2 paths) 3. Check MTU end-to-end: - All NICs, switches, and the target must match - iSCSI typically uses 9000 MTU for jumbo frames 4. Check firewall: `iptables -L -n | grep 3260` Add rules if needed: `iptables -A INPUT -p tcp --dport 3260 -j ACCEPT` 5. Tune the iSCSI session for timeouts: `iscsiadm -m node -T <target> -o update -n node.session.timeo.replacement_timeout -v 30` 6. If the LV was marked read-only, recover: `lvchange -p rw /dev/<vg>/<lv>` 7. Re-establish sessions if needed: `iscsiadm -m node -T <target> --login`
Verification
- `iscsiadm -m session -P 1` shows both sessions stable - `multipath -ll` shows 2 active paths - VMs on the iSCSI LUN respond normally - `dmesg` shows no new iscsi errors - LVM is read-write (`lvdisplay` shows "write" status)
Prevention
- Configure multipath with at least 2 paths per LUN - Tune replacement_timeout to be larger than the longest expected network blip - Use dedicated NICs for iSCSI traffic - Configure firewall rules to allow iSCSI explicitly