Failover as the client sees it - the outage the cluster does not measure

pcs status says the resource moved in nineteen seconds. The service owner says the application was down for a quarter of an hour. Both are telling the truth, and the gap between them is where this lesson lives.

A cluster measures its own transition: stop here, start there, monitor returns OK. The client measures something else - the time until its next successful request. Those two numbers are related but not equal, and the second one is the one on the incident report.

Why a fenced node costs more than a graceful one

Consider a client holding an open TCP connection to a service on node1.

Graceful stop. The service exits, the kernel sends FIN, or the port closes and the kernel sends RST. The client’s next operation fails immediately with “connection reset” or “connection refused”. It reconnects, hits the load balancer or the virtual IP now on node2, and recovers in whatever its retry logic takes. Sub-second, usually.

Fenced node. The node is powered off. There is no kernel left to send anything. Packets sent to it are not refused - they are absorbed by a switch port that leads nowhere.

The client’s TCP stack does what TCP is supposed to do: it retransmits, with exponential backoff, for a very long time. On Linux the budget is net.ipv4.tcp_retries2, which defaults to 15 and works out at roughly fifteen minutes before the connection is declared dead.

sysctl net.ipv4.tcp_retries2

The fix is on the client side, in three places:

A request timeout. Every client library has one and it is usually unset. curl --max-time, a JDBC socketTimeout, an HTTP client read timeout. This is the single highest-value change.

TCP_USER_TIMEOUT. A socket option that caps how long transmitted data may go unacknowledged before the connection is aborted, regardless of tcp_retries2. Applications that set it detect a dead peer in seconds. Where the application cannot be changed, tcp_retries2 can be lowered host-wide - a value around 8 gives roughly a hundred seconds - but understand you are changing behaviour for every connection on the host, including ones crossing a genuinely lossy link.

TCP keepalive, for long-lived idle connections. The defaults are useless for this purpose: tcp_keepalive_time is 7200 seconds, so an idle pooled connection is not probed for two hours.

sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes

What happens to requests in flight

At the instant of failover, some requests have been sent and not answered. The client does not know which of three things happened:

1. The request never arrived.
2. It arrived, was processed, and the response was lost.
3. It arrived and was partially processed.

No protocol distinguishes these from the client side. This is not a cluster limitation - it is a property of unreliable networks, and it is why the retry decision belongs to the application semantics rather than to the transport.

Idempotent operations can simply be retried. GET, PUT and DELETE are idempotent by definition: doing them twice leaves the same result as doing them once. POST is not.

Non-idempotent operations need an idempotency key: the client generates a unique id per logical operation and sends it with every attempt; the server records completed ids and returns the original result for a repeat. Payment APIs do this because the alternative is charging twice, and the same reasoning applies to any operation with a side effect.

Connection pools hold dead connections

A pool exists to avoid reconnecting, which means it is structurally inclined to hand out a connection to a node that no longer exists. After a failover, the first request on each pooled connection fails.

The mitigations are all pool settings, and all of them are off by default in something you run:

• Validate on borrow - test the connection before handing it out. Costs a round trip; worth it for pools that outlive failovers.
• Maximum connection lifetime - retire connections after a few minutes so the pool naturally drains onto the new node.
• Bounded socket timeout - so a validation query against a blackholed peer does not itself hang for fifteen minutes.

Name resolution is a cache you do not control

If clients reach the service by name and the failover changes the address, the DNS TTL is added to your RTO - and only if every layer honours it. Resolver caches, container DNS caches, and language runtimes each add their own.

The JVM is the classic case: its address cache is controlled by networkaddress.cache.ttl, and in some configurations it has historically cached successful lookups for the life of the process. A Java client that resolved the old address at startup will never find the new one, however correct the DNS is.

This is why cluster designs move an address rather than a name: a virtual IP that relocates to the surviving node, announced with gratuitous ARP, changes nothing the client has cached. The mechanics belong to the load balancing and Keepalived parts of this course; the architectural point is that a design relying on DNS to fail over has an RTO set by the least well-behaved client.

Measuring the RTO that counts

Run this from a client machine, not from a cluster node, and trigger the failover while it runs:

$ while true; do
printf '%s ' "$(date -Ins)"
curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' \
  --max-time 2 http://192.0.2.50/health || echo FAIL
sleep 0.5
done

2026-08-11T10:22:14,102+00:00 200 0.004
2026-08-11T10:22:14,608+00:00 200 0.004
2026-08-11T10:22:15,114+00:00 FAIL
2026-08-11T10:22:17,120+00:00 FAIL
2026-08-11T10:22:19,126+00:00 FAIL
2026-08-11T10:22:21,131+00:00 200 0.006

Note that --max-time 2 is doing real work here. Without it, a single curl against the fenced node would sit in the TCP retransmission budget and the loop would print one line fifteen minutes later - which is exactly what the application does, and exactly why it is invisible in the cluster logs.

Test both kinds of failover, because they produce different numbers:

# Graceful: the service stops cleanly and connections are reset.
sudo pcs resource move app --wait=120
sudo pcs resource clear app

# Ungraceful: what an incident actually looks like.
sudo pcs stonith fence node1

Record both. The graceful number is what you can promise for planned maintenance. The ungraceful number is your real RTO.

Knowledge check