Skip to main content
RunBook Academy

← All break/fix scenarios in Linux

intermediateSecurity~25 min

Break/Fix: the key works from the office but not from the VPN

Reported symptoms

  • ●The same key and the same user authenticate from the office LAN but are rejected from the VPN pool
  • ●The client reports `Permission denied (publickey,keyboard-interactive)` after offering a key the server accepts
  • ●`sshd -t` reports no error and the unit restarts cleanly, so the change looked safe at deploy time
  • ●The server log shows the key being accepted as *partial* authentication, then the connection failing
  • ●Nothing in the affected user account, key, or home-directory permissions differs between the two paths

Evidence

  • · `ssh -vvv` from the VPN shows `Authenticated with partial success` after publickey, then `Permission denied`
  • · `journalctl -u ssh` shows `Postponed publickey for alice` followed by `Failed keyboard-interactive for alice`
  • · `sshd -T -C user=alice,host=vpn,addr=203.0.113.7` reports `authenticationmethods publickey,keyboard-interactive`
  • · `sshd -T -C user=alice,host=office,addr=192.0.2.5` reports `authenticationmethods publickey`
  • · Both queries report `kbdinteractiveauthentication no`
  • · `sshd -t -f /etc/ssh/sshd_config` exits 0 - the contradiction is not a syntax error
  • · The `Match Address 203.0.113.0/24` block was added last week to force MFA on remote logins
Diagnosis and resolutionclick to reveal

Root cause

A `Match Address 203.0.113.0/24` block sets `AuthenticationMethods publickey,keyboard-interactive` to require MFA for remote logins, but `KbdInteractiveAuthentication no` is set in the global section and the Match block never overrides it. `AuthenticationMethods` requires the client to complete *every* method in at least one of its lists; the second method is administratively disabled, so no list can ever be completed and every connection from that prefix fails after the key is accepted. The office LAN does not match the block, keeps the global `AuthenticationMethods publickey`, and is unaffected - which is why the fault looks like a client or key problem rather than a server one. `sshd -t` validates syntax only and cannot see the contradiction, so the change passed review and deployed cleanly.

Remediation

Decide which of the two the MFA requirement actually needs, then make the effective configuration say it. To require key + TOTP from the VPN, enable the second factor inside the same Match block (`KbdInteractiveAuthentication yes` plus a PAM stack that runs the TOTP module, and `UsePAM yes` globally). If MFA was not intended, drop the second method from `AuthenticationMethods`. Either way, confirm the result with `sshd -T -C` for both source networks before reloading, and keep an already-established session open while you reload so a mistake does not lock you out.

Verification

Run `sshd -T -C user=alice,host=vpn,addr=203.0.113.7` and confirm that every method named in `authenticationmethods` is also enabled in the same output. From a second terminal, authenticate from the VPN and confirm the session establishes; `ssh -vvv` should show each method completing rather than stopping at `partial success`. Re-test from the office prefix to confirm the Match block did not change the unmatched path. Confirm the original session is still usable before you close it.

Prevention

Treat `sshd -t` as a syntax check, not a review. Every change to `AuthenticationMethods` or to a `Match` block must be verified with `sshd -T -C` for each source network the block is meant to distinguish - that is the only command that renders the configuration the way sshd will actually apply it. Cross-check that every method listed in `AuthenticationMethods` is enabled in the same rendered output. Add the `-T -C` comparison to CI for the source prefixes that matter, and never reload sshd without a second established session to fall back to.

Reported symptoms

  • Alice authenticates from the office LAN with her key. The same key, the same account, from the VPN pool, is rejected.
  • The client message is Permission denied (publickey,keyboard-interactive) — after the server has already accepted the key.
  • sshd -t reported no error and the service restarted cleanly, so the change that introduced this looked safe.
  • Nothing about the account, the key, or ~/.ssh permissions differs between the two paths.

Evidence provided

$ ssh -vvv alice@server01.example.com
...
debug1: Offering public key: /home/alice/.ssh/id_ed25519 ED25519
debug1: Server accepts key: /home/alice/.ssh/id_ed25519 ED25519
Authenticated with partial success.
debug1: Authentications that can continue: keyboard-interactive
debug2: we did not send a packet, disable method
alice@server01.example.com: Permission denied (publickey,keyboard-interactive).

$ sudo journalctl -u ssh --since '10 minutes ago' | tail -4
Aug 11 09:14:02 server01 sshd[4411]: Postponed publickey for alice from 203.0.113.7 port 51422 ssh2 [preauth]
Aug 11 09:14:02 server01 sshd[4411]: Failed keyboard-interactive for alice from 203.0.113.7 port 51422 ssh2
Aug 11 09:14:02 server01 sshd[4411]: Connection closed by authenticating user alice 203.0.113.7 port 51422 [preauth]

$ sudo sshd -t -f /etc/ssh/sshd_config; echo "exit=$?"
exit=0

$ sudo sshd -T -C user=alice,host=vpn,addr=203.0.113.7 | grep -E 'authenticationmethods|kbdinteractive|passwordauth'
passwordauthentication no
kbdinteractiveauthentication no
authenticationmethods publickey,keyboard-interactive

$ sudo sshd -T -C user=alice,host=office,addr=192.0.2.5 | grep -E 'authenticationmethods|kbdinteractive|passwordauth'
passwordauthentication no
kbdinteractiveauthentication no
authenticationmethods publickey

$ sudo grep -n -A3 '^Match' /etc/ssh/sshd_config
41:Match Address 203.0.113.0/24
42-    AuthenticationMethods publickey,keyboard-interactive

Work the evidence before reading on

Three facts have to be reconciled:

  1. The server accepts the key — the log says Postponed publickey, not Failed publickey. So the key, the account and the file permissions are all fine. This is not an authorization problem.
  2. The two sshd -T -C runs differ in exactly one line, and that line names a method the very same output says is disabled.
  3. sshd -t is happy. Whatever is wrong, it is not a syntax error.

Write down what “partial success” means before continuing. It is the single most informative word in the whole transcript, and it rules out most of the causes an engineer reaches for first.

Root cause

1. AuthenticationMethods is a conjunction, not a menu

AuthenticationMethods takes one or more comma-separated lists. Successful authentication requires completing every method in at least one list. publickey,keyboard-interactive therefore means “key and keyboard-interactive”, not “key or keyboard-interactive”.

The key succeeds, which is why sshd reports Authenticated with partial success and asks for the next method. There is no next method available, so the connection dies.

2. The Match block requires a method the global section disabled

The intent was MFA for remote logins. The block supplies the requirement but not the mechanism:

KbdInteractiveAuthentication no      # global

Match Address 203.0.113.0/24
    AuthenticationMethods publickey,keyboard-interactive

Match blocks override only the keywords they actually contain. KbdInteractiveAuthentication is not one of them, so the global no still applies inside the block. The result is a requirement that can never be satisfied: sshd demands a method it will not run.

The office prefix does not match the block, keeps the global AuthenticationMethods publickey, and works perfectly — which is exactly why this reads as a client-side fault for the first hour of the incident.

3. Why the log looked like a user problem

Failed keyboard-interactive for alice names the user and a method the user never chose. An operator scanning the journal reads it as “alice failed to authenticate” and starts investigating alice. The preceding Postponed publickey line — the one that proves the server already accepted her — is the line that matters, and it does not contain the word “failed” so it does not survive a grep -i fail.

Resolution

  1. Keep an escape hatch open. Before touching sshd, open a second session and leave it connected. A reload does not drop established sessions, so this is your way back in if the next step is also wrong
  2. Decide what the requirement actually is. The Match block was added to force MFA on remote logins. Either implement MFA properly, or drop the requirement — do not leave a rule that demands an unavailable method
  3. If MFA is intended: enable the second factor inside the same Match block and give it something to run. Add KbdInteractiveAuthentication yes to the block, confirm UsePAM yes globally, and configure the TOTP module in the PAM stack for sshd:
  4. `` Match Address 203.0.113.0/24 AuthenticationMethods publickey,keyboard-interactive KbdInteractiveAuthentication yes ``
  5. If MFA is not intended: reduce the block to the methods that are actually enabled — AuthenticationMethods publickey — or remove the block entirely if it now says nothing the global section does not
  6. Verify before reloading. Render the configuration for both source networks and confirm every listed method is enabled:
  7. `` sudo sshd -T -C user=alice,host=vpn.example.com,addr=203.0.113.7 | grep -E 'authenticationmethods|kbdinteractive' sudo sshd -T -C user=alice,host=office.example.com,addr=192.0.2.5 | grep -E 'authenticationmethods|kbdinteractive' ``
  8. Reload, do not restart. sudo systemctl reload ssh re-reads the configuration without dropping established sessions. Then authenticate from the VPN in a third terminal, while both earlier sessions stay open

Verification

  1. Confirm the rendered configuration is self-consistent. For each source prefix, every method in authenticationmethods appears as enabled in the same sshd -T -C output
  2. Confirm the VPN path completes. ssh -vvv from the VPN shows each method completing and the session establishing, with no partial success dead end
  3. Confirm the office path is unchanged. The unmatched prefix still authenticates exactly as before — a Match block fix must not alter the path it does not match
  4. Confirm the MFA requirement is real, if that was the intent. Authenticate from the VPN with the key alone and confirm it is rejected. A requirement that no longer rejects anything is not MFA
  5. Close the escape-hatch sessions last, only after a fresh connection has succeeded end to end

Prevention

  • Verify every Match change with sshd -T -C for each source network the block is meant to distinguish. sshd -t cannot catch a contradiction and should never be the last check before a reload.
  • Check the invariant explicitly: every method named in AuthenticationMethods must be enabled in the same rendered output. This is cheap enough to run in CI against a set of representative -C tuples.
  • Put global settings above the first Match block. Anything below it belongs to that block whether you intended it to or not.
  • Reload rather than restart, and always hold an established session open across the change.
  • When triaging an SSH rejection, read the log line before the failure. Postponed <method> means that method succeeded, and it moves the investigation off the user and onto the configuration.