Skip to main content
RunBook Academy

← All break/fix scenarios in Linux

advancedIdentity / auth~35 min

Break/Fix: logins fail, apt refuses the repo, and none of the errors mention time

Reported symptoms

  • ●SSH single sign-on to one host fails with `Permission denied (gssapi-with-mic,publickey,password)` while the same account works everywhere else
  • ●`sudo` on that host reports an authentication failure for a password the user just used successfully elsewhere
  • ●`sudo apt update` fails with `Release file ... is not valid yet`
  • ●The monitoring agent stopped reporting, with the collector logging a certificate that is not yet valid
  • ●Searching the central log platform for the last 15 minutes returns nothing at all from the host
  • ●The account is not locked in the directory and the password has not been changed

Evidence

  • · `chronyc tracking` reports `Reference ID : 00000000 ()` and `Leap status : Not synchronised`
  • · `chronyc sources -v` shows every source with `Reach` 0 and a `?` state
  • · `timedatectl` reports `NTP service: active` and `System clock synchronized: no`
  • · `systemctl is-active chronyd` reports `active` — the check most runbooks stop at
  • · `date -u` on the failing host differs from `date -u` on a healthy peer by about 22 minutes
  • · `sudo kinit alice@EXAMPLE.COM` fails with `Clock skew too great while getting initial credentials`
  • · `chronyc tracking` reports a `Skew` in the tens of ppm and a `System time` offset of roughly 1320 seconds
  • · A firewall change nine days ago replaced the outbound NTP allow rule with one naming a decommissioned server
Diagnosis and resolutionclick to reveal

Root cause

The host's clock is 22 minutes behind real time and nothing is correcting it. chronyd is running — which is why every "is NTP up?" check passed — but its configured sources became unreachable when a firewall change nine days ago replaced the outbound rule for UDP 123, so it has had no reference to discipline the clock against. Kerberos rejects any ticket request whose timestamp is outside the KDC's tolerance, 300 seconds by default, and the 22-minute skew is far outside it. Every visible symptom is a consequence reported by the layer that noticed: SSH names GSSAPI, sudo names authentication, apt names a Release file, the collector names a certificate. None of them names the clock, because none of them knows the clock is the problem — from inside the host, the time is simply what the time is.

Remediation

Restore the host's ability to reach a time source first — correct the firewall rule or point `/etc/chrony/chrony.conf` at a reachable server — and confirm with `chronyc sources -v` that a source is being polled. Correcting reachability is not enough on its own: chronyd only steps the clock during the first few updates after start-up, per its `makestep` directive, so a long-running daemon will slew a 22-minute offset at a rate measured in seconds per day. Force the correction with `sudo chronyc makestep`, then restart the services that cached a bad notion of time — sssd and the monitoring agent — because a running process holding an expired ticket will not recover on its own.

Verification

`chronyc tracking` must report `Leap status : Normal`, a real reference ID rather than `00000000`, and a `System time` offset in milliseconds rather than seconds. `timedatectl` must report `System clock synchronized: yes`. Compare `date -u` against a known-good host and confirm they agree to the second. Then exercise the paths that failed, because those are the checks that can fail: `kinit alice@EXAMPLE.COM` must succeed and `klist` must show a ticket whose valid-from time is now; SSH single sign-on must complete; `sudo apt update` must succeed; and the host must reappear in the central log platform for the current time window rather than 22 minutes in the past.

Prevention

Monitor the measured offset, not the daemon. `systemctl is-active chronyd` answers a question nobody is asking; alert instead when `chronyc tracking` reports `Leap status` other than `Normal`, or when the absolute system-time offset exceeds a threshold well below the Kerberos tolerance — 60 seconds gives four minutes of warning before authentication starts failing. Configure at least three independent sources so one unreachable server is a degradation rather than an outage, and include UDP 123 to the time servers in the standing firewall test suite. Sweep the fleet for offset regularly: hosts sitting at four minutes of skew are working today and will page someone the moment they cross five.

Reported symptoms

Four separate tickets were opened over two days, by four different people, against four different systems.

  • Identity team: “SSO to app07 is broken. Permission denied (gssapi-with-mic,publickey,password). The account is fine — it works on every other host.”
  • Application team: “sudo on app07 rejects my password. I used the same password on app06 thirty seconds earlier.”
  • Platform team: “apt update on app07 fails: Release file for https://mirror.example.com/... is not valid yet. Did someone break the mirror?”
  • Observability team: “app07 stopped reporting. The collector is logging a client certificate that is not yet valid. Has its cert been reissued?”

Nobody connected them, because no two of them are about the same subsystem.

Evidence provided

$ ssh -o GSSAPIAuthentication=yes alice@app07.example.com
alice@app07.example.com: Permission denied (gssapi-with-mic,publickey,password).

$ sudo apt update
Get:1 https://mirror.example.com/debian bookworm InRelease [151 kB]
Reading package lists... Done
E: Release file for https://mirror.example.com/debian/dists/bookworm/InRelease is not valid yet (invalid for another 13min 41s). Updates for this repository will not be applied.

$ systemctl is-active chronyd
active

$ timedatectl
               Local time: Tue 2026-08-11 08:56:12 UTC
           Universal time: Tue 2026-08-11 08:56:12 UTC
                 RTC time: Tue 2026-08-11 08:56:11
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

$ chronyc tracking
Reference ID    : 00000000 ()
Stratum         : 0
Ref time (UTC)  : Thu Jan 01 00:00:00 1970
System time     : 0.000000000 seconds fast of NTP time
Last offset     : +0.000000000 seconds
RMS offset      : 0.000000000 seconds
Frequency       : 12.417 ppm slow
Residual freq   : +0.000 ppm
Skew            : 0.000 ppm
Root delay      : 1.000000000 seconds
Root dispersion : 1.000000000 seconds
Update interval : 0.0 seconds
Leap status     : Not synchronised

$ chronyc sources -v
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? ntp1.example.com              0   10     0     -     +0ns[   +0ns] +/-    0ns
^? ntp2.example.com              0   10     0     -     +0ns[   +0ns] +/-    0ns

$ date -u; ssh app06.example.com date -u
Tue Aug 11 08:56:20 UTC 2026
Tue Aug 11 09:18:33 UTC 2026

$ kinit alice@EXAMPLE.COM
kinit: Clock skew too great while getting initial credentials

Work the evidence before reading on

Two lines in that transcript are the entire scenario, and both are easy to skim past:

  1. timedatectl says NTP service: active on one line and System clock synchronized: no on the next. Those are answers to two completely different questions.
  2. chronyc tracking reports System time : 0.000000000 seconds fast of NTP time — which reads like perfection and means nothing at all.

Before continuing, work out why chrony reports a zero offset on a host that is 22 minutes wrong, and what Reference ID : 00000000 implies about that number.

Root cause

1. The daemon is running. That is not the same as the clock being right

systemctl is-active chronyd returns active and stops there. It tells you a process exists. It says nothing about whether that process has ever successfully spoken to a time server.

The three questions, and the command that answers each:

QuestionCommandAnswer here
Is the daemon running?systemctl is-active chronydyes
Is it synchronised?chronyc tracking (Leap status)Not synchronised
How wrong is the clock?date -u against a known-good host22 minutes

Most runbooks check the first one.

2. A zero offset from an unsynchronised daemon is not a measurement

System time : 0.000000000 seconds fast of NTP time looks like the healthiest possible reading. It is what chrony reports when it has no reference at all: with Reference ID : 00000000, Stratum 0, and a Ref time of the Unix epoch, there is no NTP time to be fast or slow of. The zero is a placeholder, not a result.

The line that actually carries the state is the last one: Leap status : Not synchronised.

3. Why the sources went away

chronyc sources -v shows both servers with Reach 0 and a ? state, meaning no poll has ever been answered. Nine days ago a firewall change rewrote the outbound rule for UDP 123 to name a decommissioned server, so every poll since has been dropped silently. chronyd does not fail, log loudly, or exit when its sources are unreachable — it keeps polling and keeps the clock free-running on the local oscillator, which drifts at roughly 12 ppm here. Twelve parts per million is about one second per day, so nine days of drift is not 22 minutes.

The 22 minutes came from somewhere else: the host was restored from a snapshot six days ago, came up with a stale RTC, and chronyd — with no reachable source — had nothing to correct it against. The firewall change did not cause the skew. It removed the mechanism that would have fixed it in the first second of boot.

4. Why every error names something else

Kerberos includes a timestamp in the pre-authentication data so a captured request cannot be replayed later. The KDC rejects anything outside its tolerance — 300 seconds by default in both MIT Kerberos and Active Directory. At 1320 seconds of skew, every ticket request from this host is refused.

What each layer then reports:

  • sshd offers GSSAPI, the ticket acquisition fails, it falls through every remaining method and prints the generic Permission denied (gssapi-with-mic,publickey,password).
  • sudo, backed by the same Kerberos PAM stack, reports an authentication failure. To the user this is indistinguishable from a wrong password.
  • apt compares the Valid-Until and Date fields in the signed InRelease file against the local clock. A file signed 8 minutes ago in real time is dated 8 minutes in this host’s future, so apt correctly refuses it as not yet valid.
  • The monitoring agent’s TLS handshake presents a certificate whose notBefore is a recent reissue. From a clock 22 minutes behind, that certificate does not exist yet.

Each message is accurate about what that layer observed. None of them can see the cause, because from inside the host the clock is simply the clock.

Resolution

  1. Establish the skew as a number, from outside the host. date -u on the suspect and on a known-good peer, run back to back. Do not trust the suspect host to tell you it is wrong
  2. Read the state, not the service. chronyc tracking — the Leap status line is the verdict. chronyc sources -v — Reach 0 on every source means nothing has ever answered
  3. Fix reachability first. Confirm the path to the time servers is open before touching the clock:
  4. `` # Substitute your own values before running: NTP_SERVER=ntp1.example.com chronyc sources -v sudo timeout 5 chronyd -Q "server $NTP_SERVER iburst" ``
  5. Correct the firewall rule or the configured sources, then restart chronyd and confirm Reach starts climbing. A source with a non-zero Reach is the first real progress
  6. Step the clock deliberately. chronyd will not step a long-running system on its own — makestep normally applies only to the first few updates after start-up. Without this the 22 minutes would be slewed away over months:
  7. `` sudo chronyc makestep chronyc tracking ``
  8. Restart the services that cached a bad time. sssd holds Kerberos tickets and negative-cache entries; the monitoring agent holds a TLS session it believes is invalid:
  9. `` sudo systemctl restart sssd sudo sss_cache -E ``
  10. Re-run the four failing operations by hand before telling anyone it is fixed. Each of the four tickets needs its own confirmation
  11. Sweep the rest of the fleet. The firewall rule was fleet-wide. Other hosts are almost certainly drifting and have not yet crossed the 300-second bound

Verification

  1. chrony is genuinely synchronised. chronyc tracking shows Leap status : Normal, a real Reference ID rather than 00000000, a non-zero stratum, and a System time offset in milliseconds
  2. The kernel agrees. timedatectl reports System clock synchronized: yes
  3. The clock matches an independent reference. date -u on this host and on a known-good peer agree to the second
  4. Kerberos works. kinit alice@EXAMPLE.COM succeeds and klist shows a ticket whose valid-from time is the current time, not a time in the past
  5. Each reported symptom is individually cleared. SSO login succeeds; sudo accepts the password; sudo apt update completes; the monitoring agent reconnects and the collector logs no certificate error
  6. The host reappears in the current log window. Search the central platform for the last five minutes and confirm records from this host arrive with correct timestamps
  7. The fleet is checked, not assumed. Collect chronyc tracking from every host and assert the absolute offset is under a second. Any host reporting Not synchronised is the next incident
  8. The firewall path is tested, not inspected. Confirm the NTP rule works by observing Reach climb to 377 on a source, rather than by reading the rule and agreeing with it

Prevention

  • Alert on the offset, never on the daemon. A useful pair of checks: Leap status is Normal, and the absolute value of the tracking offset is under 60 seconds. Both fail long before Kerberos does.
  • Configure at least three independent sources, from at least two networks, so one unreachable server degrades accuracy instead of eliminating it. pool directives with maxsources do this cleanly.
  • Put UDP 123 to the time servers in the standing firewall test suite, and test it by observing reachability rather than by reading rules. This outage was a firewall change whose blast radius nobody modelled because time is invisible until it is not.
  • Sweep fleet-wide offset on a schedule. A host at four minutes of skew passes every check and is one warm afternoon away from failing authentication.
  • When a host is restored from a snapshot or migrated, treat the clock as suspect and verify it explicitly as part of the return-to-service checklist.
  • Log the skew. If the central logging platform recorded both event time and receive time, this would have been a five-minute diagnosis on the first ticket.