Spotting Risky Network Changes

A change to a network resource touches every consumer of that network resource. A change to a security group touches every instance attached to the group. A change to a route table touches every subnet using the table. A change to a network ACL touches every subnet associated with the ACL. The blast radius of a network change is rarely one resource; it is the topology behind the resource.

Network changes are also the changes most likely to be approved without being read, because the HCL is short and the changes look like “simple rule updates”. The plan output for a single rule change is small; the impact is large. The reviewer’s job is to connect the small diff to the large impact before approving.

What to look for in the diff

Six patterns in the network diff deserve a closer read. None of them are obvious from the line count; all of them are obvious from the line content.

1. The 0.0.0.0/0 ingress

A security group rule that opens a port from 0.0.0.0/0 is the single most common network misstep in production. The pattern shows up in three shapes:

# Shape A: a new rule adds public ingress
+ aws_security_group_rule "ingress_https" {
    type        = "ingress"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
}

# Shape B: an existing rule widens the source
~ aws_security_group_rule "ingress_https" {
    ~ cidr_blocks = ["10.0.0.0/8"] -> ["0.0.0.0/0"]
}

# Shape C: a port range is opened widely
+ aws_security_group_rule "ingress_ssh" {
    from_port   = 0
    to_port     = 65535
    cidr_blocks = ["0.0.0.0/0"]
}

The review posture differs by shape:

• Shape A on a load balancer’s SG is a normal pattern for public HTTP/HTTPS. The rule passes if the load balancer is the only resource in the SG and a WAF is in front.
• Shape A on a database SG is a critical production incident. Database ports (3306, 5432, 27017, 6379, 1433, 9200) should never accept 0.0.0.0/0.
• Shape B is the dangerous one because the previous rule may have been narrow (10.0.0.0/8) and the new rule is wide (0.0.0.0/0). The blast radius increased by a factor of the source range.
• Shape C is the open-everything pattern that some authors add as a temporary measure and forget to remove. The fix is never to merge it; the fix is to add the specific port range required.

2. The port broadening

A rule that previously allowed port 22 from a bastion and now allows port 22 from 0.0.0.0/0 is a port widening with no narrowing. A rule that previously allowed port 22 from 10.0.0.0/24 and now allows ports 22-30000 from the same source is a port widening with a port-range widening. The diff looks small; the blast radius increased.

~ aws_security_group_rule "app_to_db" {
    ~ from_port   = "5432" -> "0"
    ~ to_port     = "5432" -> "65535"
}

The reviewer checks that the change is justified by a ticket. If the ticket is “open the database ports for debugging”, the reviewer rejects and reopens the ticket with a different scope.

3. The rule removal that looks like no-op

A ~ to remove an ingress rule (or an egress rule) is often missed. The arrow points to null, which is the removal; the line is short; the reviewer scrolls past.

~ aws_security_group_rule "egress_db" {
    ~ cidr_blocks = ["10.10.0.0/16"] -> []
}

An empty cidr_blocks is the same as no rule. If the previous rule was the only thing allowing the database subnet to reach the application subnet, removing it breaks connectivity. The reviewer checks that the removal is intended and that a compensating rule is added.

4. The route-table change

A change to a route table can be the highest-impact network change in the diff. A route that points to a new transit gateway, a new peering, or a new NAT gateway redistributes traffic. A route that is removed breaks connectivity for every subnet on the table.

~ aws_route_table "private" {
    ~ route {
        ~ cidr_block = "0.0.0.0/0"
        ~ gateway_id = "igw-0abc" -> "tgw-0def"
      }
}

The diff says the default route’s gateway changed. The effect is that every subnet on the table now sends internet-bound traffic through the transit gateway instead of the internet gateway. That may be the intent (the egress now flows through a packet inspector); it may also be a typo (the author intended to update a different route). The reviewer confirms the intent and the change window.

5. The CIDR overlap

A workspace that adds a CIDR overlapping an existing VPC’s CIDR is the silent failure mode. Both VPCs are created; both peering attachments are accepted; the routing is symmetric; the first packet from one VPC to the other arrives at the wrong VTEP.

+ module "network_v2" {
    + cidr_block = "10.0.0.0/16"
}

The review catches the overlap by reading the allocation table. If 10.0.0.0/16 is already assigned to prod-platform-us-east-2, this PR is rejected even if the plan succeeds.

6. The NACL statelessly wider rule

A NACL rule that opens a wide ephemeral port range (1024-65535) is sometimes necessary for stateful return traffic. The reviewer confirms the rule is on the right NACL (the one for the data subnet, not the public one), the protocol is tcp (not all), and the rule is paired with a narrower rule for the specific return path.

The cost of a permissive change

Each pattern has a quantifiable cost. Naming the cost in the PR comment is the discipline.

Pattern                  Cost
-------                  ----
0.0.0.0/0 ingress on     Credential-stuffing target;
SSH/RDP                  typical brute-force time to
                         compromise: hours for an
                         internet-facing SSH port

0.0.0.0/0 ingress on     Direct data exfiltration path;
DB ports                 no application-layer
                         authorisation on the DB protocol

Route-table change       Connectivity break for every
                         subnet on the table; full
                         change window required

NACL stateless wide      Asymmetric routing; some return
                         traffic dropped; intermittent
                         failures that look like
                         application bugs

CIDR overlap             Silent misrouting; packets
                         delivered to the wrong VPC;
                         debugging takes hours

The cost is reviewed, not invented. A permissive SG change does not “feel expensive” until the first incident; the discipline is to call out the cost before the incident.

Change windows for network changes

Network changes are not apply-anytime changes. The right change window depends on what is changing.

The narrower/wider distinction is the rule. A rule that narrows traffic (removes an ingress, restricts a source, removes a route) is safe by default; the cost of narrowing is at most a connection failure that the next plan will detect. A rule that widens traffic is the risk; widenings require a sign-off and a change window.

The right approval workflow

A network PR is approved by two reviewers: the standard PR reviewer and a network engineer (or a security engineer, depending on the change category).

PR opened
   |
   v
CODEOWNERS auto-assigns the network reviewer
   |
   v
plan runs; scanners run; comment posted
   |
   v
network reviewer walks the rubric:
  - which resources are affected
  - is the change a widening or a narrowing
  - what is the rollback
  - is the change window scheduled
  - is the change documented in the runbook
   |
   v
standard reviewer walks the standard rubric
   |
   v
both approve; change window reached; apply

The network reviewer’s rubric is distinct from the standard reviewer’s rubric. The standard reviewer reads the configuration for style and correctness; the network reviewer reads the configuration for blast radius and risk. Both are needed.

# CODEOWNERS entry for network files
/network/**                          @runbook/network-reviewers @runbook/iac-engineers
/modules/network-*/**                @runbook/network-reviewers @runbook/iac-engineers

The dual-team entry means: at least one reviewer from each team. The standard team approves the standard rubric; the network team approves the network rubric.

A reference PR for a network change

# PR #1042: Tighten web tier SG to bastion only

resource "aws_security_group_rule" "bastion_to_web" {
  type                     = "ingress"
  from_port                = 22
  to_port                  = 22
  protocol                 = "tcp"
  source_security_group_id = aws_security_group.bastion.id
  security_group_id        = aws_security_group.web.id
  description              = "SSH from bastion only"
}

# Removed: previous rule that allowed 0.0.0.0/0 on 22
# (will be deleted via - in plan)

terraform plan -out=tfplan
terraform show -json tfplan | jq '.resource_changes[] | select(.address | test("security_group"))'

{
  "address": "aws_security_group_rule.bastion_to_web",
  "actions": ["create"]
}
{
  "address": "aws_security_group_rule.legacy_open_ssh",
  "actions": ["delete"]
}
{
  "address": "aws_security_group.web",
  "actions": ["no-op"]
}

The plan shows one create and one delete; the SG itself is unchanged. The reviewer walks the rubric:

• Scope: 1 create, 1 delete, no replaces. Limited.
• Blast radius: 2 instances + 1 ALB are members of the SG. List them in the comment.
• Reversibility: The widening is a narrowing (delete the open-SSH rule), trivially reversible by re-creating the rule.
• Change ticket: SEC-1042 — “Disable public SSH on web tier.” Matches.
• Trust: The change matches the ticket, the blast radius is named, the rollback is documented.

Approve. Apply during the standard change window.

Validating against the live cloud

A network-change PR should be checked against the live cloud at review time, not only at apply time. The check verifies that the configuration matches reality at the moment of merge.

# Pull the live security groups for the workspace
aws ec2 describe-security-groups \
    --filters "Name=tag:Workspace,Values=prod-web" \
    --query 'SecurityGroups[*].{Id:GroupId,Ingress:IpPermissions}' \
    --output json > /tmp/live-sg.json

# Compare against the planned SG
terraform show -json tfplan | \
    jq '.resource_changes[] | select(.address | test("security_group"))' \
    > /tmp/planned-sg.json

A diff between the live SG and the planned SG that exceeds the changes in the PR is drift. The PR is correct against the configuration but the configuration is not correct against the cloud; the next apply will restore the configuration and undo the drift.

DRIFT: aws_security_group.web has additional ingress rule
  live:    tcp 22 from 0.0.0.0/0 (added via console)
  planned: tcp 22 from bastion only (in this PR)

The PR will apply; the live SG will be restored to
the configuration; the console-added rule will be
removed by the apply.

The reviewer notes the drift in the PR comment; the author confirms the intent to remove the console rule.

Production failure modes

1. The reviewer approved the 0.0.0.0/0 ingress because “it’s just on port 443”. Port 443 to a database SG is the same public exposure as port 22 to a database SG. The fix is the rubric walk; the port number does not override the source.

2. A route-table change was applied without a change window. An on-call engineer applied at 03:00 to fix an incident. The change broke a different subnet that shared the table. The fix is the change-window rule; not “apply anytime”.

3. A CIDR overlap was applied because the policy did not catch it. Two VPCs were peered with the same 10.0.0.0/16. Routing was symmetric; packets delivered to the wrong VPC. The fix is the allocation-table check in the policy; the policy rejects cidr_block not in the table.

4. A NACL rule was added for debugging and never removed. Six months later, an auditor found the rule; the SG was supposed to be private; the NACL provided a backdoor. The fix is the PR template expiration date; every temporary rule has one.

5. The PR widened the SSH source range and the reviewer approved without checking the new range against the bastion. The new range included the engineer’s home IP for “convenience during debugging”. The fix is the standard rubric: bastion-only SSH, no engineer home IP.

6. The live cloud check was skipped; the PR was merged against a stale cloud. The apply made a change that the reviewer thought was a no-op because the configuration matched the cloud six months earlier. The fix is the live-cloud validation at review time.

Security and performance

Security: a permissive SG rule is a public credential- stuffing target within minutes of being applied. Credential-stuffing bots scan the entire IPv4 address space on common ports within hours. An open SSH or RDP port is compromised in a typical timeframe of hours for an internet-facing host. An open database port is targeted by ransomware crews on a similar timeline.

Performance: the network diffs are short (one line per rule). The plan output is small. The performance cost is not in the plan; the cost is in the change window and the sign-off. Both are operational overhead, not tooling overhead.

Production guidance

• CODEOWNERS for every network file path. The network reviewer is part of the rubric, not optional.
• The blast radius is named in the PR comment. Every consumer of the SG, the route table, the NACL is listed.
• Wider changes require a sign-off and a change window. Narrower changes are applied anytime.
• Live-cloud validation at review time. The configuration matches reality or the PR is rejected.
• Temporary rules have an expiration date in the PR. No expiration, no merge.

What comes next

The next lesson is the state-touching PR review: terraform state mv, terraform state rm, terraform import, the audit trail, and the right rollback.

Verification

Take the last three PRs against network/** in the team’s repository. Walk each one against the rubric: which reviewer approved, what was the blast radius, was it a widening or a narrowing, was the change in a window, was the live cloud checked. Note the PRs that fail the rubric; those are the next incidents.