Skip to main content
RunBook Academy

← All break/fix scenarios in Linux

advancedNetworking~40 min

Break/Fix: half the peers cannot reach the host and the firewall is not the reason

Reported symptoms

  • ●Roughly half the clients cannot reach a service on this host, and the same half fails every time
  • ●The other half works perfectly, with no packet loss and normal latency
  • ●Which clients fail does not correlate with subnet, VLAN, firewall zone or DNS
  • ●From the host, `ping` to some peers gets no reply while `ping` to others is clean
  • ●`ip link show bond0` reports the bond UP and RUNNING
  • ●`cat /proc/net/bonding/bond0` reports `MII Status: up` for both slaves

Evidence

  • · `cat /proc/net/bonding/bond0` reports `MII Polling Interval (ms): 0`
  • · The same file reports `MII Status: up` for both `eth0` and `eth1`
  • · `sudo ethtool eth1 | grep "Link detected"` reports `Link detected: no`
  • · `ip -s link show eth1` shows TX packets climbing steadily and RX packets frozen
  • · `sudo ethtool -S eth1 | grep -i carrier` shows a non-zero carrier-down counter
  • · `sudo dmesg -T | grep eth1` shows a single `NIC Link is Down` six days ago and nothing since
  • · `sudo tcpdump -ni bond0 host 192.0.2.77` shows replies being transmitted that never arrive at the peer
  • · `cat /sys/class/net/bond0/bonding/mode` reports `balance-xor 2` with `xmit_hash_policy` `layer2`
Diagnosis and resolutionclick to reveal

Root cause

The bond has no link monitoring configured at all: `miimon` is 0 and no ARP monitor is set, so the bonding driver never polls the slaves and simply reports whatever state each was in when it was enslaved. `eth1` lost carrier six days ago — the driver logged it once — and the bond never learned, so it still lists the slave as up and still schedules traffic onto it. In `balance-xor` with a layer2 hash, each peer MAC address is hashed to a fixed slave, so a given peer always uses the same NIC. Peers that hash to the live NIC work perfectly; peers that hash to the dead one blackhole completely and consistently. Ingress is unaffected because the switch only forwards over its own live port, so requests arrive and replies vanish — which reads exactly like a firewall or ACL problem and sends the investigation to the network team.

Remediation

Restore service first by removing the dead slave from the bond, which forces every flow onto the working NIC: `echo -eth1 | sudo tee /sys/class/net/bond0/bonding/slaves`. Then fix the actual defect, which is not the NIC — it is that the bond had no way to notice. Set a polling interval at runtime with `echo 100 | sudo tee /sys/class/net/bond0/bonding/miimon` and, critically, persist it in whatever manages the interface (netplan `mii-monitor-interval`, NetworkManager `bond.options`, or `BONDING_OPTS`) — a runtime write is lost at the next reboot and the fault returns looking new. Repair the physical fault on eth1, re-enslave it, and confirm the bond now reports its state correctly.

Verification

`cat /proc/net/bonding/bond0` must report a non-zero `MII Polling Interval (ms)` and a `MII Status` per slave that matches `sudo ethtool ethN | grep "Link detected"` for each. The test that can actually fail is a deliberate one: run `sudo ip link set eth1 down`, and within a second `/proc/net/bonding/bond0` must show that slave as `MII Status: down` and `dmesg` must log the bonding event — then `sudo ip link set eth1 up` and confirm it returns. Re-test reachability from a client that previously failed and confirm it now works. Finally reboot, or re-apply the network configuration, and re-check the polling interval; a value that only exists in sysfs has not been fixed.

Prevention

Assert `miimon` fleet-wide. A bond without link monitoring is not redundant — it is two NICs and a hash function — and the kernel bonding documentation says exactly that. Monitor per-slave link state rather than the bond's own UP flag, which stays UP while every slave is dead. Prefer 802.3ad with LACP where the switch supports it, because LACP exchanges keepalives and detects a peer that has stopped participating even when carrier is still asserted, and still configure `miimon` alongside it. Test failover as a routine exercise: down a slave, confirm the bond notices and traffic continues, bring it back. That five-minute drill is the only thing that distinguishes a working bond from one that has never been asked to do its job.

Reported symptoms

  • Since a change window six days ago, about half the application servers cannot reach the API on svc01. The other half are fine.
  • It is completely consistent. The same hosts fail every time; the same hosts succeed every time. Retrying never helps and never hurts.
  • The failing hosts are spread across three subnets and two VLANs. Working hosts are in the same subnets and VLANs. There is no pattern the network team can find in their ACLs.
  • svc01 itself looks perfectly healthy: the service is listening, the bond is UP, and curl from svc01 to some peers works.
  • The firewall team confirms no rule changed. The network team confirms no ACL changed. Everyone is confident it is not them.

Evidence provided

$ ip link show bond0
5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether 52:54:00:a1:b2:c3 brd ff:ff:ff:ff:ff:ff

$ cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.1.0

Bonding Mode: load balancing (xor)
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 52:54:00:a1:b2:c3

Slave Interface: eth1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 52:54:00:a1:b2:c4

$ sudo ethtool eth0 | grep 'Link detected'
	Link detected: yes

$ sudo ethtool eth1 | grep 'Link detected'
	Link detected: no

$ ip -s link show eth1
7: eth1: <BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN mode DEFAULT
    RX:  bytes packets errors dropped  missed   mcast
       4182993021  8811204      0       0       0       0
    TX:  bytes packets errors dropped carrier collsns
      91827364512 74119883      0       0   74119883      0

$ sudo ethtool -S eth1 | grep -iE 'carrier|link'
     tx_carrier_errors: 74119883
     link_down_events: 1

$ sudo dmesg -T | grep eth1
[Wed Aug  5 02:14:07 2026] ixgbe 0000:04:00.1 eth1: NIC Link is Down

$ cat /sys/class/net/bond0/bonding/mode /sys/class/net/bond0/bonding/xmit_hash_policy /sys/class/net/bond0/bonding/miimon
balance-xor 2
layer2 0
0

Work the evidence before reading on

Four observations, each of which is individually unremarkable and together are the whole answer:

  1. /proc/net/bonding/bond0 says MII Status: up for eth1. ethtool eth1 says Link detected: no. Both are printed by the kernel. Both cannot be right.
  2. Link Failure Count: 0 on a slave whose driver logged NIC Link is Down six days ago.
  3. ip -s link show eth1 shows TX packets in the tens of millions climbing right now, RX frozen, and a carrier-error count equal to the TX packet count.
  4. MII Polling Interval (ms): 0.

Before continuing, answer: who is supposed to notice that a slave has lost carrier, and how often does the file above say they check?

Root cause

1. The bonding driver only knows what it asks

The bond does not receive a notification when a slave loses carrier and act on it. It polls, on a timer, and miimon is that timer in milliseconds. There is a second mechanism — the ARP monitor, configured with arp_interval and arp_ip_target — which probes a peer address instead.

With miimon=0 and no arp_interval, neither runs. The driver performs no link monitoring whatsoever. It reports each slave’s state as it was when the slave was enslaved, and it schedules transmit across every slave in that list forever.

The kernel’s own bonding documentation is unusually direct about this: it is critical that either miimon, or arp_interval together with arp_ip_target, is specified — otherwise serious network degradation will occur when a link fails. This scenario is that sentence happening.

That counter increments when the bond observes a slave transition. The bond observed nothing, so it counted nothing. The failure is recorded elsewhere:

  • ethtool eth1 — the driver’s live query of the PHY: Link detected: no
  • ethtool -S eth1 — link_down_events: 1
  • dmesg — one line, six days ago
  • ip -s link show eth1 — carrier errors equal to TX packets, because every single frame handed to that NIC is discarded for want of carrier

Four independent sources say the link is down. The one file everybody looks at says it is up.

3. Why exactly half the peers, consistently

balance-xor with xmit_hash_policy layer2 selects the transmitting slave by XOR-ing the source and destination MAC addresses and taking the result modulo the number of slaves. With two slaves, that is a coin flip per destination MAC — but a deterministic one. A given peer always hashes to the same slave.

So the fleet splits cleanly and permanently:

  • Peers hashing to eth0: perfect service.
  • Peers hashing to eth1: every transmitted frame discarded at the NIC.

Because the hash is on MAC addresses and all off-subnet traffic goes via the router’s MAC, the split falls along lines that have nothing to do with subnet, VLAN or firewall zone — which is precisely why the network team could find no pattern.

4. Why it looks like a firewall rule

Ingress works. The switch has its own view of the link, its port to eth1 is down, and it therefore forwards every frame to the host over the live port. Requests arrive normally.

Replies go out through the bond, get hashed, and half of them are dropped inside the host before ever reaching a cable.

The result is a one-way path — traffic in, nothing out — for a fixed set of peers, unchanging over days. That is the exact signature of an ACL, and it is what everyone reasonably concluded.

Resolution

  1. Confirm the contradiction explicitly before changing anything, so the fix is aimed at the right thing. Compare /proc/net/bonding/bond0 against ethtool for every slave
  2. Restore service by removing the dead slave. With one slave in the bond, every flow hashes to the survivor and the failing half recovers immediately:
  3. `` echo -eth1 | sudo tee /sys/class/net/bond0/bonding/slaves ``
  4. Enable link monitoring at runtime. This is the actual fix and it takes effect immediately:
  5. `` echo 100 | sudo tee /sys/class/net/bond0/bonding/miimon cat /proc/net/bonding/bond0 | grep -i polling ``
  6. Persist it. A sysfs write does not survive a reboot, and a bond that regresses to miimon=0 at the next maintenance window produces an incident nobody connects to this one. On netplan:
  7. `` network: bonds: bond0: interfaces: [eth0, eth1] parameters: mode: balance-xor mii-monitor-interval: 100 ``
  8. Or with NetworkManager:
  9. `` sudo nmcli connection modify bond0 bond.options "mode=balance-xor,miimon=100" ``
  10. Apply and re-read the live state, not the file you edited. sudo netplan apply or sudo nmcli connection up bond0, then confirm the polling interval in /proc/net/bonding/bond0
  11. Fix the physical fault. Carrier down for six days is a cable, an optic, or a switch port — not a Linux problem. Get it repaired rather than working around it
  12. Re-enslave eth1 once the link is genuinely up, and confirm the bond now reports both slaves correctly:
  13. `` echo +eth1 | sudo tee /sys/class/net/bond0/bonding/slaves ``
  14. Audit every other bonded host in the fleet. This configuration came from a template, and templates are used more than once

Verification

  1. Monitoring is on. grep -i "polling interval" /proc/net/bonding/bond0 reports a non-zero value
  2. The bond agrees with the hardware. For each slave, the MII Status in /proc/net/bonding/bond0 matches sudo ethtool ethN | grep "Link detected". They disagreed for six days; agreement is the whole point
  3. The bond actually detects a failure. This is the verification that can fail, and it is the only one that proves anything:
  4. `` sudo ip link set eth1 down sleep 1 grep -A2 "Slave Interface: eth1" /proc/net/bonding/bond0 sudo ip link set eth1 up ``
  5. The bond reports the transition. sudo dmesg -T | tail shows a bonding line naming the slave, and Link Failure Count for that slave has incremented. A count that stays at 0 through a deliberate down means monitoring is still not running
  6. Traffic survives the failure. Run a continuous ping from a peer that previously failed while you down the slave; it should continue uninterrupted or lose at most a packet or two
  7. The previously broken peers work. Test from at least three of the hosts that were failing, not just one — with a hash-based mode, a single successful test may just be a peer that always hashed to the good slave
  8. It survives a reboot. Reboot the host, or re-apply the network configuration from scratch, and re-check the polling interval. This is where a runtime-only fix is exposed
  9. The fleet is clean. Confirm every bonded host reports a non-zero polling interval, rather than assuming the template was only used here

Prevention

  • Assert miimon (or the ARP monitor) on every bond, everywhere, as a configuration-management check. A bond without a monitor is not redundant hardware — it is two NICs sharing a hash function, and it fails worse than a single NIC would.
  • Monitor per-slave carrier, not the bond’s UP flag. /sys/class/net/<slave>/carrier and the Link Failure Count per slave are the signals that matter; the master’s operstate is not.
  • Alert on a slave count below the expected number. This host ran for six days at half capacity with nothing firing.
  • Alert on tx_carrier_errors from ethtool -S. A counter climbing by millions per day is a NIC transmitting into nothing, and it is unambiguous.
  • Drill failover. Down a slave on a schedule in a maintenance window, confirm the bond reacts, bring it back. Redundancy that has never been exercised is a hypothesis.
  • Treat a “half the clients fail, consistently, with no ACL pattern” report as a load-balancing hash symptom until proven otherwise. Bonds, ECMP routes and LAGs all produce it, and it never looks like the network device it actually is.