Skip to main content
RunBook Academy

← All break/fix scenarios in Linux

intermediateIdentity / auth~30 min

Break/Fix: the whole team lost sudo overnight and visudo -c says the config is fine

Reported symptoms

  • ●Every member of the ops team lost sudo on all 200 hosts at the same time
  • ●`sudo -l` reports `Sorry, user alice may not run sudo on host01`
  • ●Other teams whose rules live in `/etc/sudoers` itself are unaffected
  • ●Group membership is intact: `id alice` lists the `ops` group on every host
  • ●`sudo visudo -c` reports `parsed OK` and lists no errors
  • ●The rule file is present in `/etc/sudoers.d/`, owned by root, mode 0440, and contains exactly the right rule

Evidence

  • · `id alice` lists `ops` and `getent group ops` returns the expected members
  • · `ls -l /etc/sudoers.d/` shows `-r--r----- 1 root root ... 10-ops-team.conf`
  • · `sudo grep -r ops /etc/sudoers.d/` prints the intended `%ops ALL=(ALL:ALL) ALL` rule
  • · `sudo visudo -c` prints `/etc/sudoers: parsed OK` and names every drop-in it read — but not `10-ops-team.conf`
  • · `sudo -l -U alice` run as root reports that alice may not run sudo, which is the authoritative answer
  • · `sudo -ll -U alice` shows no matching entries from any drop-in
  • · The configuration-management commit 14 hours ago renamed `10-ops-team` to `10-ops-team.conf`
  • · `journalctl -u sssd` is clean and no directory or PAM change was deployed
Diagnosis and resolutionclick to reveal

Root cause

sudo deliberately ignores files inside an `#includedir` directory whose names contain a dot or end in a tilde. The rule exists so that package manager leftovers such as `.dpkg-dist`, `.rpmnew` and editor backups are not silently loaded as policy, and it applies to any dot, anywhere in the name. A configuration-management change renamed the drop-in from `10-ops-team` to `10-ops-team.conf` to match a new naming convention, and from that moment sudo skipped it on every host. Nothing complains: the file is present, correctly owned, correctly permissioned and syntactically valid, and it is simply never read. `visudo -c` reinforces the illusion, because it validates the files sudo would actually read and therefore reports `parsed OK` on a policy that no longer contains the rule — it is answering "is what sudo reads valid" rather than "does sudo read what I wrote".

Remediation

Rename the file to a name with no dot and no trailing tilde — `sudo mv /etc/sudoers.d/10-ops-team.conf /etc/sudoers.d/10-ops-team` — and confirm the policy changed with `sudo -l -U alice` rather than with `visudo -c`. Getting a root shell to do that is the harder half when nobody has sudo: use an existing root session if one is open, the console or out-of-band management, a break-glass account, or the configuration-management agent, which already runs as root and is the only practical route to 200 hosts. Fix the template that produced the name in the same change, otherwise the next run renames it back. Deploy to one canary host and verify there before the fleet.

Verification

`sudo -l -U alice` run as root must list the expected commands — this is the rendered policy and it is the only check that would have caught the fault. Have a real ops-team member run `sudo -n true` and confirm it exits 0; a policy that renders correctly but does not work means the problem is elsewhere. `sudo visudo -c` must still report `parsed OK` and must now name the drop-in among the files it read. Assert the filename rule directly with `ls /etc/sudoers.d/ | grep -E "\\.|~$"`, which must return nothing. Then re-run the configuration-management agent and repeat the checks, because a fix the next converge undoes is not a fix.

Prevention

Verify sudo changes with `sudo -l -U user`, never with `visudo -c` alone: one renders the policy, the other only says that whatever was read parses. Edit drop-ins with `sudo visudo -f /etc/sudoers.d/name` so a syntax error is refused before it is written. Enforce the naming rule in CI — every file in `sudoers.d` must match a plain alphanumeric-with-dashes pattern — because the failure is silent and no runtime check will catch it. Always hold an open root session while changing sudo policy, and canary authorisation changes to one host before the fleet. Keep a documented, audited break-glass path and test it on a schedule; the time to discover that the console password is wrong is not while 200 hosts have no administrators.

Reported symptoms

  • At 08:00 every member of the ops team found that sudo no longer worked, on all 200 hosts, simultaneously.
  • The DBA team is unaffected. Their rules live directly in /etc/sudoers.
  • The first hypothesis was the directory: a group sync had run overnight and everyone assumed ops membership had been dropped. It had not.
  • The second hypothesis was SSSD or PAM. Both are clean and unchanged.
  • sudo visudo -c was run on three hosts. All three reported parsed OK, so sudoers was ruled out.
  • The rule file is right there in /etc/sudoers.d/, with the correct content, owner and mode.

Evidence provided

$ sudo -l
[sudo] password for alice:
Sorry, user alice may not run sudo on host01.

$ id alice
uid=1841(alice) gid=1841(alice) groups=1841(alice),4001(ops),4009(oncall)

$ getent group ops
ops:*:4001:alice,bob,carla,dan,erin

$ ls -l /etc/sudoers.d/
total 12
-r--r----- 1 root root  118 Aug 10 18:41 10-ops-team.conf
-r--r----- 1 root root   87 Mar 02 09:15 20-dba-readonly
-r--r----- 1 root root   44 Jan 18 11:03 90-cloud-init

$ sudo grep -r ops /etc/sudoers.d/
/etc/sudoers.d/10-ops-team.conf:%ops ALL=(ALL:ALL) ALL

$ sudo visudo -c
/etc/sudoers: parsed OK
/etc/sudoers.d/20-dba-readonly: parsed OK
/etc/sudoers.d/90-cloud-init: parsed OK

$ sudo -l -U alice
User alice is not allowed to run sudo on host01.

$ sudo -ll -U bob
User bob is not allowed to run sudo on host01.

$ git -C /srv/cfg log --oneline -1 -- roles/base/files/sudoers.d/
c41f8a2 style: standardise drop-in filenames on the .conf suffix

Work the evidence before reading on

The answer is visible in the transcript, in a place that reads as formatting rather than as content. Three things to line up:

  1. visudo -c prints one line per file it validated. Count the files it names. Now count the files in ls -l /etc/sudoers.d/.
  2. The one that is missing from visudo -c is the one whose rule is missing from the policy.
  3. Look at what is different about that filename compared to the other two.

Before continuing: visudo -c said parsed OK. What exact question was it answering?

Root cause

1. sudo ignores drop-in filenames containing a dot

/etc/sudoers ends with an include directive:

@includedir /etc/sudoers.d

(Older files write this as #includedir /etc/sudoers.d. Despite appearances that is a directive, not a comment — sudo predates the convention, and @ was added later as the unambiguous spelling. Both work; neither is disabled by the leading #.)

When sudo reads that directory it deliberately skips any file whose name contains a . anywhere, or ends in ~. From sudoers(5): files in the included directory that end in ~ or that contain a . character are ignored.

The reason is sound. Package upgrades leave behind 10-ops-team.dpkg-dist, 10-ops-team.rpmnew, 10-ops-team.rpmsave; editors leave 10-ops-team~ and .10-ops-team.swp. Loading any of those as live authorisation policy would be a serious security bug. So sudo refuses to read them.

The rule makes no exception for a dot you meant. 10-ops-team.conf contains a dot. It is skipped.

2. visudo -c validated exactly what sudo reads

This is the part that cost the team two hours, and it is not a bug.

visudo -c walks the same include directory using the same skip rules and validates each file it decides to read. It reported parsed OK for three files. It said nothing about the fourth, because as far as sudo is concerned there is no fourth.

Read carefully, the output is the answer: the file you are worried about is absent from a list of everything that was read. Read quickly, parsed OK is a green light and the investigation moves on.

The distinction generalises:

CommandQuestion it answers
visudo -cDoes everything sudo reads parse?
sudo -l -U aliceWhat is alice actually allowed to do?

Only the second one can detect a rule that was never loaded.

3. Why it hit every host at once, and only one team

The rename came from configuration management, so it converged across the fleet overnight. The DBA rules live in /etc/sudoers itself and were never subject to the include rules, which is why exactly one team lost access — and why the incident looked like a directory-group problem specific to ops.

Resolution

  1. Get a root shell. Nobody has sudo, so work down this list: an already-open root session on any host; the console or out-of-band management; a documented break-glass account; the cloud provider serial console or run-command service; single-user mode via the boot loader as a last resort
  2. For 200 hosts, the configuration-management agent is the answer. Puppet, Salt, Chef and an Ansible pull-mode agent all already run as root and do not use sudo. Fixing the template and letting the fleet converge is faster and safer than 200 console sessions
  3. Confirm the diagnosis on one host before changing anything else. Compare the file list from visudo -c against ls /etc/sudoers.d/; the missing name is the fault
  4. Rename the file. This is the whole fix:
  5. `` sudo mv /etc/sudoers.d/10-ops-team.conf /etc/sudoers.d/10-ops-team ``
  6. Verify with the rendered policy, not the file:
  7. `` sudo -l -U alice sudo visudo -c ``
  8. Fix the template that produced the name, in the same change. Otherwise the next converge renames it straight back and the incident recurs at a time nobody will connect to this one
  9. Canary before the fleet. Deploy to one host, have a real ops-team member run sudo -n true there, and only then roll out
  10. Roll out and re-verify a sample, rather than trusting that convergence implies correctness
  11. Add the naming assertion to CI while the incident is fresh, so the class is closed rather than the instance

Verification

  1. The rendered policy contains the rule. sudo -l -U alice, run as root, lists the expected commands. This is the check that can fail and the one that was missing from the change process
  2. A real user can actually use it. Have an ops-team member run sudo -n true and confirm exit status 0. A policy that renders correctly but fails in practice points at PAM or the directory, which is a different incident
  3. visudo now reads the file. sudo visudo -c reports parsed OK and names 10-ops-team in its output. Its presence in that list is the point, not the parsed OK
  4. No filename in the directory can be skipped. ls /etc/sudoers.d/ | grep -E "\.|~$" returns nothing. Any output is a file sudo is ignoring, whether or not you meant it to
  5. Permissions are still right. ls -l /etc/sudoers.d/10-ops-team shows root:root and mode 0440. sudo refuses a world-writable or wrongly-owned drop-in — a different silent-ish failure with a different fix
  6. Convergence does not undo it. Re-run the configuration-management agent, then repeat the first two checks. This is where a hand-fix that skipped the template is exposed
  7. The fleet is sampled, not assumed. Run sudo -l -U alice on a random handful of hosts across different roles and regions
  8. The break-glass path was exercised and still works. If you used it during this incident, note whether it behaved as documented; if you did not need it, test it separately

Prevention

  • Verify with sudo -l -U user. visudo -c answers a narrower question than most people think, and in this failure it answers it correctly and unhelpfully.
  • Name drop-ins with letters, digits, dashes and underscores only. No dots, no .conf, no .disabled, no trailing tilde. Enforce it in CI, because nothing at runtime will.
  • Never disable a rule by renaming it. Remove the file, or comment out its contents with visudo -f. A rename can silently disable a rule you wanted, and equally silently enable one you did not.
  • Edit with visudo -f so a syntax error is refused rather than written.
  • Canary authorisation changes. Any change that can remove administrative access from a fleet should reach one host before it reaches all of them.
  • Maintain and test a break-glass path. This incident was survivable because the configuration-management agent runs as root; without it, 200 console sessions is a very long morning.
  • When a change lands via configuration management, fix the source, not the host. A hand-fixed host is a host waiting for the next converge.