Break/Fix: 403 on a file with correct ownership, and ausearch finds nothing · Break/Fix · Linux

Reported symptoms

Every URL under /reports/ returns 403. The rest of the site is fine.
ls -l shows the files owned by apache:apache, mode 0644, in a directory that is 0755 apache:apache.
sudo -u apache cat /var/www/html/reports/q3.pdf prints the file, so the web server user can read it.
The application log shows the web server’s own 403, not an application error.
Someone checked for SELinux denials with ausearch -m avc -ts recent, got no matches, and moved the investigation to the application team.

Evidence provided

$ ls -l /var/www/html/reports/q3.pdf
-rw-r--r--. 1 apache apache 184320 Aug  2 04:12 /var/www/html/reports/q3.pdf

$ sudo -u apache cat /var/www/html/reports/q3.pdf > /dev/null; echo "exit=$?"
exit=0

$ getfacl /var/www/html/reports/q3.pdf 2>/dev/null | tail -4
user::rw-
group::r--
other::r--

$ ls -lZ /var/www/html/
drwxr-xr-x. 2 apache apache system_u:object_r:httpd_sys_content_t:s0   4096 Jul 14 10:02 assets
drwxr-xr-x. 2 apache apache unconfined_u:object_r:user_home_t:s0       4096 Aug  2 04:12 reports

$ getenforce
Enforcing

$ sudo ausearch -m avc -ts recent
<no matches>

$ systemctl is-active auditd
inactive

$ sudo journalctl -k --since '10 min ago' | grep -i avc | tail -2
Aug 11 09:22:07 web01 kernel: audit: type=1400 audit(1786000927.114:812): avc:  denied  { read } for  pid=1422 comm="httpd" name="q3.pdf" dev="dm-0" ino=918273 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0

$ matchpathcon /var/www/html/reports/q3.pdf
/var/www/html/reports/q3.pdf	system_u:object_r:httpd_sys_content_t:s0

Work the evidence before reading on

Four things in that transcript are worth pausing on:

sudo -u apache cat succeeds. The web server user can read the file. So whatever is blocking the web server is not the web server’s user identity.
The ls -l output ends each mode string with a dot: -rw-r--r--. That dot is not a typo and it is not part of the mode.
ausearch says <no matches>. journalctl -k says the opposite, loudly. Both are telling the truth.
matchpathcon prints a context. Compare it to the one in ls -lZ.

Answer this before continuing: why does sudo -u apache cat work when httpd cannot read the same file as the same user?

Root cause

1. The dot in `ls -l` is the clue everyone scrolls past

A trailing . on the mode string means the file has an SELinux security context. A trailing + means it has a POSIX ACL. Both are printed by default, both are one character wide, and both are routinely read as punctuation.

Once you see the dot, ls -lZ is the obvious next command — and it shows the whole answer in one line:

drwxr-xr-x. apache apache system_u:object_r:httpd_sys_content_t:s0   assets   <- works
drwxr-xr-x. apache apache unconfined_u:object_r:user_home_t:s0       reports  <- 403

2. DAC and MAC are two independent checks

Linux runs discretionary access control (owner, group, mode, ACLs) and mandatory access control (SELinux) as separate gates. A read has to pass both. Everything the team checked belongs to the first gate, and the first gate is wide open.

sudo -u apache cat succeeds because that shell runs in the unconfined_t domain, which policy allows to read user_home_t. The web server runs in the httpd_t domain, which does not. Same user, same file, different domain, different answer — and that is the single most useful thing to internalise about MAC: the process’s domain matters as much as its uid.

Confirm which domain the service is actually running in — it is not the one your shell is in:

ps -eZ | grep httpd | head -2

Because you cannot easily borrow another domain from a shell, the only faithful reproduction is to make the real service do the read: request the URL and watch the kernel log.

3. `mv` preserves labels; `cp` does not

The deploy script changed nine days ago:

# Before - correct labels, slower
cp -r /home/deploy/staging/reports/. /var/www/html/reports/

# After - fast, and wrong
mv /home/deploy/staging/reports/* /var/www/html/reports/

cp creates a new inode at the destination, and the kernel applies the type transition rule for the destination directory — so the copy comes out httpd_sys_content_t. mv within the same filesystem is a rename: no new inode, no transition, the file keeps the user_home_t label it was born with under /home.

The same trap exists in several familiar tools:

Tool	Label behaviour
`cp`	Destination default (correct by accident)
`cp -a`, `cp --preserve=context`	Source label preserved
`mv` (same filesystem)	Source label preserved
`mv` (across filesystems)	Behaves as a copy — destination default
`tar -x`	Destination default unless `--selinux`
`rsync`	Destination default unless `-X`
`install`	Destination default

4. `ausearch` was telling the truth

ausearch reads /var/log/audit/audit.log, which is written by auditd. With auditd stopped, the kernel has nowhere to send audit records except the kernel ring buffer, so they surface in dmesg and the journal as type=1400 ... avc: denied and never reach the file ausearch reads.

<no matches> from ausearch on a host where auditd is not running is no evidence at all. It is the most expensive false negative in SELinux troubleshooting, because it reads exactly like a clean result.

setenforce 0 is a diagnostic, not a fix

sudo setenforce 0 will make the 403 disappear in under a second, and that is genuinely useful — it confirms the hypothesis in one step. Two things go wrong when it is left in place:

The host is now unprotected, and so is every host the change gets copied to. setenforce 0 is not persistent, so the fault returns at the next reboot looking like a brand-new incident.
Worse is the persistent version, SELINUX=permissive in /etc/selinux/config. In permissive mode denials are logged but not enforced, so the host keeps working while accumulating exactly the evidence you would need — and nobody reads it because nothing is broken.

Use it to confirm, then set it straight back:

sudo setenforce 0
# confirm the request now succeeds, then immediately:
sudo setenforce 1
getenforce

Resolution

Confirm the hypothesis where the denial actually happens. Watch sudo journalctl -kf in one terminal while you request the URL in another; a fresh avc: denied line arriving with each request is the confirmation. Reading the file from your own shell will not reproduce it, because your shell is not in the confined domain
Start auditd first, so every subsequent step is visible in the place you will look next time:
``sudo systemctl enable --now auditd``
Ask policy what the label should be. matchpathcon /var/www/html/reports/q3.pdf prints the expected context; sudo matchpathcon -V on a path reports actual against expected and is the form worth learning
If the path is one the default policy already knows — anything under /var/www on a stock policy — relabel it directly:
``sudo restorecon -Rv /var/www/html/reports``
If the path is outside the default policy, register the rule before relabelling, or the next full relabel will undo your work:
``sudo semanage fcontext -a -t httpd_sys_content_t '/srv/reports(/.*)?' sudo restorecon -Rv /srv/reports``
Fix the deploy script, which is the actual defect. Either go back to cp, or keep mv and relabel afterwards:
``mv /home/deploy/staging/reports/* /var/www/html/reports/ restorecon -Rv /var/www/html/reports``
Confirm enforcing mode is still on. If the diagnostic step used setenforce 0, put it back and verify with getenforce
**Do not reach for audit2allow here.** It generates a policy module permitting httpd_t to read user_home_t — which would let the web server serve every user home directory on the box. The label is wrong; the policy is right

Verification

Labels are correct and policy agrees. ls -lZ /var/www/html/reports/ shows httpd_sys_content_t throughout, and sudo matchpathcon -V /var/www/html/reports/q3.pdf reports verified rather than a mismatch
Enforcing is on. getenforce returns Enforcing. A 200 in permissive mode proves nothing
The request actually succeeds. curl -sS -o /dev/null -w '%{http_code}\n' https://web01.example.com/reports/q3.pdf returns 200
No new denials. With auditd running, sudo ausearch -m avc -ts recent stays empty while the page is loaded repeatedly. This is the check that could not work before and can now
The audit path itself works. Deliberately provoke a denial on a scratch file, confirm ausearch reports it, and clean up. An audit pipeline nobody has tested is the reason this incident lasted a day
The next deploy does not undo it. Run the deploy script and re-check ls -lZ. If the labels regress, the script fix did not land
It survives a relabel. sudo restorecon -Rvn /var/www/html/reports in dry-run mode should propose no changes; if it proposes reverting your labels, the semanage fcontext rule is missing

When ausearch works but still shows nothing

Once auditd is running and ausearch still reports no matches for a denial you are certain is happening, the next suspect is a dontaudit rule. Policy uses these to silence denials that are expected and harmless — usually probing access that applications perform and recover from — and they suppress the audit record without suppressing the denial.

Turn them off temporarily, reproduce, then turn them back on:

sudo semodule -DB      # disable dontaudit, rebuild policy
# reproduce the failure, then read the log
sudo semodule -B       # restore dontaudit rules

Leave dontaudit disabled only for as long as the reproduction takes. With it off, an ordinary host generates a substantial volume of uninteresting AVC records, which is the reason those rules exist.

Prevention

Run auditd on every enforcing host and monitor that it is running. An SELinux host without auditd enforces silently, which is the worst of both worlds.
Treat ausearch returning nothing as inconclusive until you have confirmed auditd is active. Cross-check journalctl -k | grep avc, which works whether or not auditd is running.
Register file contexts with semanage fcontext for every non-default path, then restorecon. A label set by chcon alone is temporary: it is correct until the next restorecon -R or automatic relabel silently reverts it.
Make deployment tooling label-aware and assert on it afterwards. A one-line restorecon -Rv at the end of a deploy costs nothing and removes the entire class.
Never ship SELINUX=permissive as a remediation. If a policy genuinely needs extending, extend it with a reviewed module — and read what audit2allow proposes before applying it, because it will happily generate a rule far broader than the problem.

What if the same symptom appears on an AppArmor host?

Debian and Ubuntu ship AppArmor rather than SELinux, and the same “permissions are correct but access is denied” shape appears — with different tools and a different underlying model.

AppArmor confines by path, not by label, so there is nothing on the file to inspect: ls -lZ shows no useful context and the trailing dot is absent. The evidence lives entirely in the kernel log:

sudo journalctl -k | grep -i 'apparmor="DENIED"'
sudo aa-status

The denial line names the profile and the path it refused. Because confinement follows the path, the equivalent of this scenario is not a mislabelled file but a relocated directory: move the document root from /var/www to /srv/www and the profile no longer covers it.

To investigate, put the single profile into complain mode rather than disabling AppArmor globally:

sudo aa-complain /etc/apparmor.d/usr.sbin.apache2

Then fix the profile — usually by adding the path rule and reloading with apparmor_parser -r — and return it to enforce with aa-enforce. The transferable habit is identical: check whether a mandatory access control layer is refusing the operation before concluding that a correct-looking ls -l means the permissions are fine.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. The dot in ls -l is the clue everyone scrolls past

2. DAC and MAC are two independent checks

3. mv preserves labels; cp does not

4. ausearch was telling the truth

Resolution

Verification

Prevention

1. The dot in `ls -l` is the clue everyone scrolls past

3. `mv` preserves labels; `cp` does not

4. `ausearch` was telling the truth