Skip to main content
RunBook Academy

← All break/fix scenarios in VyOS

advancedvyos-interface~30 min

Bond Interface Flapping

Reported symptoms

  • ●Several times a week, every downstream service behind the distribution router stops at once for 40 to 90 seconds and then returns on its own
  • ●The OSPF adjacency on `bond0.20` drops, the VRRP pair on `bond0.30` both transition, DHCP relay stops answering, and the remote syslog feed gaps - all within the same second
  • ●Monitoring records `bond0` down and then up; the alert has always cleared itself before anyone can log in
  • ●The switch team reports both switches healthy, the MLAG peer-link healthy, and the port-channel never down - only one member port bouncing
  • ●Inspected afterwards, `/proc/net/bonding/bond0` shows both members up with matching aggregator IDs and nothing obviously wrong
  • ●There is no correlation with traffic volume, time of day, temperature or any change ticket
  • ●An engineer who happened to be on the console during one event saw `bond0` and every `bond0.N` sub-interface reporting no carrier while `eth1` was still up with a synchronised LACP partner

Evidence

  • · `cat /proc/net/bonding/bond0` shows mode IEEE 802.3ad, LACP active on, LACP rate slow, and Link Failure Count 23 on eth2 against 0 on eth1
  • · `show configuration commands | match bonding` shows two member interfaces and `min-links '2'`
  • · The kernel log shows repeated `bond0: link status definitely down for interface eth2` followed 40 to 90 seconds later by `definitely up for interface eth2`
  • · During an event, `ip link show bond0` reports NO-CARRIER and state DOWN while `eth1` is up and its LACP partner is synchronised
  • · During the same event, every `bond0.N` VLAN sub-interface reports state LOWERLAYERDOWN
  • · `ip -s link show eth2` shows a non-zero carrier counter; the same counter on eth1 is zero
  • · `ethtool eth2` between events reports link detected yes, 1000Mb/s, full duplex - the member looks perfect whenever anybody checks it
Diagnosis and resolutionclick to reveal

Root cause

There are two findings and only one of them is the incident. The ordinary one is that eth2's physical path is unreliable: its Link Failure Count is 23 against zero on eth1, its carrier counter is non-zero, and the kernel log records the member's link dropping and returning. That is a cable, an optic or a switch port, and on a redundant bond it should have cost half the aggregate bandwidth and nothing else. The incident is the configuration that converted it into a total outage: `min-links` is set to 2 on a bond with two members. The Linux bonding driver asserts carrier on the master only while the number of aggregated members is at least `min-links`, so on a two-member bond that value means the bond is up only while both members are up. The first member fault takes the master to NO-CARRIER even though a healthy member with a synchronised partner is still there, and every VLAN sub-interface stacked on the bond follows the master into LOWERLAYERDOWN - which is every downstream service the router carries, all failing in the same second, from causes that appear entirely unrelated to each other. It is also why the switch team was telling the truth: from their side one port bounced and the port-channel stayed up on the other. The setting was almost certainly added to express "this bond needs both of its links", and that is exactly what it does. It just expresses it as "take the bond down rather than run degraded", which is the opposite of what a redundant uplink exists for.

Remediation

Set `min-links` to 1 so that a single member fault costs bandwidth rather than carrier, then chase the physical trigger separately. The order matters and so does the separation: apply `set interfaces bonding bond0 min-links '1'`, commit and save, and expect that a change under `interfaces bonding` may bounce the bond on commit, so treat it as a service-affecting change on a router whose users have already had several outages this week. If the bond happens to be degraded when you apply it, carrier returns immediately, which is the good outcome but means the change window and the incident window are the same window. Consider `lacp-rate fast` in the same change only if you have measured how long the member takes to rejoin the aggregator after carrier returns; changing two things at once means you will not know which one shortened the outage. Then give the switch team the timestamps of the Link Failure Count transitions and ask specifically whether that port was error-disabled and auto-recovered, and have the optic and patch replaced if it was not. One trap to avoid: do not "temporarily" remove the failing member from the bond while `min-links` is still 2, because a one-member bond under `min-links 2` never asserts carrier at all and you will convert an intermittent outage into a permanent one. If none of this can be done now, the honest hold is to shift the downstream services to the standby router by VRRP priority, take this router out of the forwarding path, and fix both findings in a window - with a named owner and an end time, not "when someone gets to it".

Verification

Prove the amplifier is gone by testing it rather than by reading the configuration back. In a window, take one member down deliberately - disable the switch port, or `set interfaces ethernet eth2 disable` - and require that `ip link show bond0` stays up with carrier, that every `bond0.N` sub-interface stays up, that the OSPF adjacency on `bond0.20` survives, and that `/proc/net/bonding/bond0` shows one aggregated member and a bond that is still running. A `min-links` value nobody has ever exercised is a guess, and this is the check that can fail. Re-enable the member and confirm it rejoins the same aggregator: matching Aggregator ID on both members and the same partner MAC on both. Time that rejoin and write the number down, because it is the part of the outage that `lacp-rate` governs. Verifying the physical trigger is a different exercise on a different timescale: record the current Link Failure Count for eth2, then compare it after a period longer than the longest gap you have observed between events. The fault was already intermittent, so one quiet afternoon is not evidence.

Prevention

Treat `min-links` as a statement about how the bond should fail, and write the reason in the interface description so the next reviewer sees the intent alongside the value. On a two-member bond, any value above 1 removes the redundancy the second member was bought for; the legitimate uses are bonds wide enough that running on one member would be worse than running on none, and they are rare enough to deserve a comment. Alert on the things that change before the outage rather than on the outage: the per-member Link Failure Count delta and the count of aggregated members are both leading indicators, and both were available for weeks in this incident while the only alert anybody had was a bond-state alert that cleared before a human could read it. Test the failure the redundancy exists for, at build time and after every change to the bond: pull one member in a window and watch what survives. Keep the switch side in the interface inventory - switch hostname, port, aggregation mode, LACP rate - because half of this bond is configured by another team. Finally, read "several unrelated services failed in the same second" as a question about the stack rather than about the services. Things that share nothing except a physical path fail together when the shared physical path fails, and the fastest way to find the shared thing is to ask what all the broken services are standing on.

The incident

dist-01 is the distribution router for a floor of the building. It has one uplink: bond0, two members, 802.3ad to an MLAG pair, and seven VLAN sub-interfaces on top of it carrying the OSPF adjacency to the core, a VRRP pair with its partner router, DHCP relay, the remote syslog feed and four user VLANs.

Several times a week, for between forty and ninety seconds, all of it stops. It has been happening for about a month.

The tickets read like four separate faults:

  • The core team has an OSPF adjacency alert on bond0.20.
  • The VRRP pair on bond0.30 transitions, both routers log it, and the service team has been chasing a “VRRP instability” ticket.
  • Users on two VLANs report brief drops; DHCP renewals fail during the window.
  • The logging platform has a gap-detection alert firing on this router’s feed.

Four teams, four tickets, four investigations, and the only thing the four have in common is that they clear before anyone can look. The bond-state alert fires and resolves inside the same polling interval often enough that half the events are not even recorded.

The switch team has been asked twice and has answered the same way twice: both switches are healthy, the MLAG peer-link is healthy, the port-channel to dist-01 has not gone down, and one member port bounces occasionally.

Evidence

Read-only / Safetaken between events: everything is up, and two fields are not ordinary
vyos@dist-01:~$ sudo cat /proc/net/bonding/bond0
Bonding Mode: IEEE 802.3ad Dynamic Link Aggregation
LACP active: on
LACP rate: slow
Min links: 2

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Aggregator ID: 1

Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 23
Aggregator ID: 1

Illustrative output

Read-only / Safetwo members, and a floor of two
vyos@dist-01:~$ show configuration commands | match bonding
set interfaces bonding bond0 mode '802.3ad'
set interfaces bonding bond0 hash-policy 'layer3+4'
set interfaces bonding bond0 min-links '2'
set interfaces bonding bond0 member interface 'eth1'
set interfaces bonding bond0 member interface 'eth2'
set interfaces bonding bond0 mtu '9000'

Illustrative output

Read-only / Safeone member, repeatedly, and never the other one
vyos@dist-01:~$ journalctl -k --since -7d | grep bond0
bond0: link status definitely down for interface eth2
bond0: link status definitely up for interface eth2
bond0: link status definitely down for interface eth2
bond0: link status definitely up for interface eth2

Illustrative output

This is the capture taken from the console during an event, and it is the piece nobody had until an engineer happened to be logged in:

vyos@dist-01:~$ ip link show bond0
5: bond0: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 9000 state DOWN

vyos@dist-01:~$ ip link show bond0.20
9: bond0.20@bond0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9000 state LOWERLAYERDOWN

vyos@dist-01:~$ ip link show eth1
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 state UP

And the per-member counters:

vyos@dist-01:~$ ip -s link show eth2
4: eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 state UP
    RX:  bytes  packets  errors  dropped  missed  mcast
    TX:  bytes  packets  errors  dropped  carrier  collsns
                              0        0       23        0

The carrier column on the transmit line counts carrier-loss events on that member. On eth1 the same column reads zero.

Work the evidence before reading on

Four facts have to be true at once, and only one explanation makes them all true.

  1. Four services that share no software, no protocol and no configuration failed in the same second. Repeatedly.
  2. The switch never lost the port-channel. Their statement is not a deflection; their side genuinely stayed up.
  3. During the event, eth1 was up with a synchronised partner, and bond0 had no carrier anyway.
  4. Between events everything reads healthy, except two numbers that nobody had compared: a Link Failure Count of 23 on one member and 0 on the other.

Fact 3 is the one to sit with. A two-member 802.3ad bond with one healthy member should be a bond running at half capacity. Before continuing: what would have to be configured for a bond with a live, aggregated member to refuse to assert carrier - and where in the evidence above is it?

Root cause

1. The trigger is ordinary and is not the incident

eth2’s physical path is unreliable. Link Failure Count 23 against 0 on eth1, a non-zero carrier counter, and a kernel log that records the member’s link dropping and returning, always the same member. That is a cable, an optic, or a switch port that is being error-disabled and auto-recovered - and it is a fault the bond exists to absorb. On a healthy redundant uplink this costs half the aggregate bandwidth for the duration and nothing else. Nobody would have opened four tickets.

The bonding driver asserts carrier on the master only while the number of aggregated members is at least min-links. Below that, the master goes NO-CARRIER regardless of how healthy the remaining members are.

On a bond with two members, min-links 2 therefore means: the bond is up only while both members are up. There is no degraded state. The first member fault is a total loss of the uplink.

That single node explains every symptom:

  • Why four unrelated services fail together. Every bond0.N VLAN sub-interface is stacked on bond0, and a VLAN interface follows its lower device’s carrier. When the master goes NO-CARRIER, all seven sub-interfaces go LOWERLAYERDOWN in the same instant. OSPF, VRRP, DHCP relay and syslog did not fail; the thing they were all standing on went away.
  • Why the switch team saw nothing. From the switch side one member port bounced and the port-channel stayed up on the other. The switch has no visibility into the host’s carrier policy, and nothing on its side went down. Both reports were accurate and they described different systems.
  • Why it looks fine afterwards. min-links is only visible when it bites. Between events the bond has two aggregated members, is above the floor, and reads perfectly healthy.

3. Why the outage lasts far longer than the bounce

The member is removed from the aggregator as soon as the driver sees the link go down, which the MII poll catches quickly. Coming back is not symmetric: the member has to complete an LACP exchange and reach the synchronised, collecting and distributing state before the driver counts it as aggregated again, and only then does the member count cross back over min-links and carrier return.

That asymmetry is why a sub-second physical bounce produces a forty-to-ninety-second service outage. It is also the part of the outage that lacp-rate influences - slow exchanges LACPDUs on a 30-second period against fast’s one second. Measure the rejoin time on your own bond before assuming a number for it; the point here is the shape, which is that recovery is bounded by protocol negotiation and not by the cable.

4. Why the intent was reasonable and the outcome was not

min-links 2 was almost certainly set to express “this bond needs both of its links”. That is a true statement about capacity, and min-links is the wrong place to put it. What the node actually says is “if you cannot have both links, have none” - and on the single uplink of a distribution router, “none” means the router is disconnected.

Resolution

  1. Confirm the floor before changing it. show configuration commands | match bonding must show min-links and the member count together - the value is only meaningful relative to how many members exist.
  2. Confirm you have a path to the router that does not cross bond0. This change may bounce the bond on commit, and the console or out-of-band path is what you will be using if it does.
  3. Apply the floor change on its own: configure, set interfaces bonding bond0 min-links 1, compare to confirm exactly one line, commit, save. Change nothing else in this commit.
  4. Confirm the bond is up with both members aggregated, matching Aggregator IDs and a partner MAC present on each member.
  5. Give the switch team the timestamps of the Link Failure Count transitions from the kernel log and ask specifically whether that port was error-disabled and auto-recovered, and for what reason. If it was not, the optic and the patch are the next things to replace.
  6. Leave lacp-rate alone for now. If the rejoin time you measure during verification is unacceptable, change it as a separate, later commit so you can attribute the improvement.
  7. Record in the bond description what the floor is and why, so the next reviewer sees the intent next to the value.

If the change cannot be made now, hold deliberately rather than by default: shift the downstream services to the standby router by VRRP priority, take dist-01 out of the forwarding path, and fix both findings in a window. Give the hold a named owner and an end time. An undecided hold on a router that drops every service four times a week is not a decision, it is the incident continuing.

Verification

  1. Fail a member on purpose, in a window. Disable the switch port for eth2, or set interfaces ethernet eth2 disable and commit.
  2. ip link show bond0 stays UP with carrier, and does not report NO-CARRIER. This is the check that can fail, and it is the only one that actually tests the change.
  3. Every bond0.N sub-interface stays up. Confirm at least the one carrying the OSPF adjacency and the one carrying VRRP, by name, rather than assuming they follow.
  4. The OSPF adjacency on bond0.20 stays full through the whole test, and the VRRP pair on bond0.30 does not transition.
  5. cat /proc/net/bonding/bond0 shows one aggregated member and a bond that is still running.
  6. Re-enable the member and time how long it takes to rejoin. Confirm both members show the same Aggregator ID and the same partner MAC afterwards, and write the rejoin time in the change record - it is the number that decides whether lacp-rate fast is worth a second change.
  7. Record the current Link Failure Count for eth2, then re-read it after a period longer than the longest gap between events you have observed. The trigger was intermittent, so a quiet afternoon proves nothing about whether the physical fault is fixed.
  8. Confirm the four original tickets stop. The proof that the diagnosis was right is that the OSPF, VRRP, DHCP and syslog alerts all stop together, because they were always one alert wearing four names.

Prevention

  • Write min-links down as a failure policy, not as a capacity number, and put the reason in the interface description. On a two-member bond any value above 1 deletes the redundancy the second member exists to provide. The legitimate uses are bonds wide enough that running on one member is worse than running on none, and they are rare enough to deserve a comment.
  • Alert on the leading indicators, not on the outage. Per-member Link Failure Count deltas and the count of aggregated members were both available for a month while the only alert anybody had cleared itself before a human could read it. An alert that resolves inside one polling interval is not monitoring, it is a rumour.
  • Test the failure the redundancy is for. Pull one member in a window at build time, and again after any change to the bond. Redundancy that has never been exercised is a design document, not a property of the system.
  • Keep the switch side in the interface inventory: switch hostname, port, aggregation mode, LACP rate. Half of an 802.3ad bond is configured by a team you have to phone, and the phone call goes better when both sides are describing the same ports.
  • Read simultaneity as a stack question. When services that share no code and no configuration fail in the same second, look for the lowest layer they share. Four teams spent a month investigating four protocols that were all standing on one carrier bit.
  • Separate the trigger from the amplifier in the write-up as well as in the fix. Closing this incident with “replaced a faulty optic” leaves the next flapping member free to do exactly the same thing.
  • vyos-x-01-lacp-concept — LACP negotiation, actor and partner, fast versus slow rate, and what min-links does.
  • vyos-x-04-bond-failure-modes — the eight production bond failure modes and the evidence each one leaves.
  • vyos-x-06-bond-validation — the host-side and switch-side validation command set used throughout this scenario.
  • vyos-x-03-switch-dependencies — the switch half of the bond, and why the inventory needs to name it.