Symptom, cause and contributing factor — separating what the user sees from what the firewall did

A firewall incident arrives in three layers. The user reports a symptom — vague, often incorrect, sometimes misdirected. The firewall shows a different symptom — concrete, technical, sometimes obscured by tooling. The cause is the thing that produced the firewall symptom, and it is usually one or two steps removed from what the user described. The operator’s job is to walk from the user-reported symptom through the firewall-observed symptom to the cause, in that order, without skipping a layer or jumping to conclusions.

This lesson covers the three layers, the discipline of keeping them separate on paper, and the trap of treating a contributing factor as the actual cause.

The three layers

The user says “email is broken”. The operator must not assume that means SMTP is broken — email could mean SMTP, IMAP, POP3, webmail, an API the user’s app uses, or any combination. The operator’s first move is to refine the symptom: which protocol, which port, which client, which server, when did it last work.

Walking from symptom to cause

The discipline is to make each layer explicit before moving to the next. The operator writes the user-observed symptom on the incident note, then walks to the firewall and gathers the firewall-observed symptom, then walks back to the console and forms a cause hypothesis. Each step is a separate act, not a single gestalt guess.

A useful pattern:

1. Write the user-observed symptom verbatim. Include who reported it, when, and how. “User jdoe reports Outlook on workstation 192.0.2.50 cannot send mail since 02:14.”
2. Refine with one clarifying question. The minimum question that disambiguates the protocols and ports. “Is it sending, receiving, or both? Which server?”
3. Reproduce or capture. Generate the same traffic from the workstation (or a known-good one) and capture at the firewall. “tcpdump on LAN shows SYN to imap.example.com:993 with no SYN-ACK.”
4. Check the firewall’s own view. pfctl -s state | grep, pfctl -s rules | grep, the routing table, the NAT table. The firewall’s view is the firewall-observed symptom.
5. Form the cause hypothesis. One sentence. “The outbound NAT rule for the LAN does not match IMAPS because the rule was changed to permit SMTP only.”
6. Test the hypothesis. What evidence would refute it? Look for that evidence. If absent, the hypothesis stands.

Each step produces a written entry in the incident note. The next operator reading the note can reconstruct the trail.

Contributing factors vs cause

A contributing factor is a condition that makes the cause worse or more likely. The cause is the specific change or condition that produced the firewall symptom. They are not the same thing, and the difference matters:

• A change to the outbound NAT rule is the cause of a missing NAT translation for IMAPS.
• The fact that the rule change happened during a maintenance window is a contributing factor.
• The fact that the change was not peer-reviewed is a contributing factor.
• The fact that there was no verification step after the change is a contributing factor.

The fix for the cause is to restore the rule. The fix for the contributing factors is to add a pre-commit peer review, a post-change verification checklist, and a documented change procedure. Both fixes are needed to prevent recurrence, but they are different fixes.

The incident note format

The minimum content for an incident note that distinguishes the three layers:

[User-observed symptom]
  Source: jdoe
  Time: 02:14 UTC
  Verbatim: "I can't send email since this morning."
  Refined: POP3 from workstation 192.0.2.50 to pop.example.com fails; SMTP and webmail work.

[Firewall-observed symptom]
  tcpdump on LAN shows SYN from 192.0.2.50:51234 to pop.example.com:995
  with no SYN-ACK.
  pfctl -s state: no entry for 192.0.2.50 → pop.example.com:995.
  pfctl -s rules: LAN → any on 110, 143, 587, 993 allowed; 995 missing.

[Hypotheses]
  1. Rule missing for POP3S — supported by pfctl output.
  2. Routing failure upstream — refuted by working SMTP to same provider.
  3. Server-side outage — refuted by working SMTP and webmail.

[Cause]
  Outbound firewall rule on LAN was scoped to 110, 143, 587, 993;
  POP3S on 995 was not added when IMAPS was added.

[Contributing factors]
  - Rule change during maintenance window did not include 995.
  - No post-change verification from a test client.

[Remediation]
  Added POP3S (995) to the LAN outbound rule.
  Verification: workstation sends/receives mail; pfctl state shows entries.

[Prevention]
  Peer review of rule changes before apply.
  Verification checklist including a known-good client test.

The note is longer than “fixed the rule”, but the next on-call operator can read it and understand exactly what happened, why, and how to prevent recurrence. The note is the document that turns an incident into institutional memory.

$ pfctl -s state | grep 192.0.2.50 | head -5

all tcp 192.0.2.50:51234 -> 198.51.100.10:587  ESTABLISHED:ESTABLISHED
all tcp 198.51.100.10:587 <- 192.0.2.50:51234  ESTABLISHED:ESTABLISHED
all tcp 192.0.2.50:51235 -> 198.51.100.10:993  ESTABLISHED:ESTABLISHED
all tcp 198.51.100.10:993 <- 192.0.2.50:51235  ESTABLISHED:ESTABLISHED

The vocabulary

A useful set of distinctions to keep in the incident vocabulary:

• Symptom — what someone observed. “Email is broken.”
• Refined symptom — the disambiguated version with protocol, port, host, time. “POP3S from 192.0.2.50 fails since 02:14.”
• Evidence — an output from a command. “pfctl -s state has no entry for 192.0.2.50:995.”
• Hypothesis — a candidate explanation. “Rule missing for POP3S.”
• Cause — the verified hypothesis. “Rule was scoped to 110, 143, 587, 993; 995 was omitted.”
• Contributing factor — a condition that enabled the cause. “Rule change was not peer-reviewed; no post-change verification.”
• Remediation — the fix that addresses the cause. “Add 995 to the rule.”
• Prevention — the fix that addresses the contributing factors. “Add peer review and post-change verification to the change procedure.”

A sentence in the incident note that uses these terms precisely is easier to read and easier to act on than a sentence that uses them interchangeably.

Summary

• Three layers: user-observed symptom, firewall-observed symptom, cause. Walk them in order; do not skip.
• A contributing factor is not a cause. The cause is the specific change or condition; the contributing factor is the condition that enabled it.
• The incident note format separates the layers on paper. The format is the discipline.
• When the user symptom and the firewall symptom disagree, the firewall symptom wins.
• Precise vocabulary (symptom, evidence, hypothesis, cause, contributing factor) keeps the investigation honest.