Vulnerability scanning — detecting known CVEs in images

Vulnerability scanning inspects an image’s filesystem for known CVEs. The scanner compares the image’s packages against a vulnerability database (NVD, GHSA, vendor advisories) and reports findings. This lesson covers the tools, the severity tiers, the CI/CD and admission integration, and the production triage workflow.

The CVE severity tiers

CVEs are scored by CVSS (Common Vulnerability Scoring System) and grouped into tiers:

A production cluster typically blocks deploys on Critical and High CVEs without a documented exception.

flowchart LR
    A[Scan image] --> B{Critical CVE?}
    B -->|yes| C[Block deploy]
    B -->|no| D{High CVE?}
    D -->|yes, no exception| E[Block deploy]
    D -->|no, exception| F[Deploy with warning]
    D -->|no| G[Allow deploy]

The scanning tools

The choice depends on the operational context: Trivy and Grype are the most common open-source choices; Snyk is the commercial leader.

Running a scan

# Trivy
trivy image myapp:v1.0
# = | severity | package | cve | fix |
# CRITICAL | openssl | CVE-2024-1234 | 3.0.13 |
# HIGH | curl | CVE-2024-5678 | 8.5.0 |

# Grype (using an SBOM)
grype sbom:./myapp.spdx.json
# NAME    INSTALLED  FIXED-IN  VULNERABILITY   SEVERITY
# openssl 3.0.10     3.0.13    CVE-2024-1234   Critical

The scanner produces a list of findings with the package, the installed version, the fixed version, and the CVE ID.

CI/CD integration

The scan runs in the CI pipeline and gates the build:

#!/bin/bash
# scan-image.sh — CI gate

trivy image --exit-code 1 --severity CRITICAL,HIGH myapp:v1.0
if [ $? -ne 0 ]; then
  echo "Critical or High CVE found; deploy blocked"
  exit 1
fi
trivy image --exit-code 0 --severity MEDIUM,LOW myapp:v1.0
echo "Medium/Low CVEs reported; deploy allowed"

--exit-code 1 makes the scan exit with failure on Critical/High CVEs; the CI pipeline fails; the deploy is blocked.

Admission integration

A ValidatingAdmissionPolicy rejects Pods whose images have Critical CVEs. The scanner runs as a sidecar (Kyverno’s verifyImages or a custom webhook):

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: deny-critical-cves
spec:
  validationFailureAction: Enforce
  rules:
  - name: critical-cves
    match:
      resources:
        kinds: ["Pod"]
    verifyImages:
    - imageReferences:
      - "registry.example.com/*"
      attestors:
      - entries:
        - keys:
            publicKeys: |-
              -----BEGIN PUBLIC KEY-----
              ...
              -----END PUBLIC KEY-----
      - name: vultr-scan  # Trivy server or similar

A Pod whose image has a Critical CVE is rejected at admission. The deployment is blocked before the workload runs.

flowchart LR
    A[Pod admission] --> B[Scan image]
    B --> C{Critical CVE?}
    C -->|yes| D[Reject]
    C -->|no| E{High CVE?}
    E -->|yes, no exception| D
    E -->|no| F[Allow]

Triage workflow

The team triages scanner findings:

1. Triage by severity. Critical and High are fixed first; Medium and Low are scheduled.
2. Check exploit availability. A CVE with a known exploit (CISA KEV) is fixed before a CVE without.
3. Check the package’s usage. A CVE in a package that the workload does not use is a false positive; document it.
4. Document exceptions. A CVE that cannot be fixed (no patch available) is documented with a deadline.

## Production failure modes

1. **Scanner database is stale.** The scanner misses
   recent CVEs. The fix is to update the database
   daily.
2. **CVEs are not remediated.** The team ignores the
   findings. The fix is to enforce the gating and
   assign ownership.
3. **No exception documentation.** A CVE that cannot
   be fixed is deployed without a record. The fix is
   to require an exception entry.
4. **Scanner reports false positives.** The team
   ignores all findings. The fix is to triage and
   dismiss false positives explicitly.

## Cross-course references

- The Linux course covers the package managers that
  the scanner inspects.
- The Observability course covers the SIEM
  integration for scanner alerts.

## Quiz

<Quiz
  id="kubernetes-q-lxiv-03-vuln-scanning"
  client:load
  questions={[
    {
      kind: "multiple_choice",
      id: "kubernetes-q-lxiv-03-severity",
      prompt: "Which CVE severity is the conventional cutoff for blocking a deploy?",
      options: [
        "Medium (CVSS 4.0+)",
        "Low (CVSS 0.1+)",
        "All severities equally",
        "Critical and High (CVSS 7.0+)"
      ],
      correct: 3,
      explanation: "Critical and High CVEs (CVSS 7.0+) are the conventional cutoff for blocking a deploy. Medium and Low are tracked but do not block. The exact cutoff depends on the organisation's risk tolerance; a security-sensitive cluster may block on Medium as well."
    },
    {
      kind: "true_false",
      id: "kubernetes-q-lxiv-03-false-positive",
      prompt: "A scanner finding for a CVE in a package that the workload does not actually use is a true positive — the deploy must be blocked.",
      correct: false,
      explanation: "A scanner finding is a true positive only if the package's vulnerable code path is reachable. A CVE in a library that the workload does not use is a false positive; the deploy can proceed. The triage workflow must distinguish true positives from false positives (and document the false positives for future audits)."
    },
    {
      kind: "scenario",
      id: "kubernetes-q-lxiv-03-stale-db",
      prompt: "Your CI pipeline scans every image with Trivy. The TrivyDB is refreshed weekly. A new Critical CVE (CVE-2024-9999) is announced on Monday; the next TrivyDB refresh is on Sunday. The vulnerability is in a widely-used library. Walk the response.",
      scenario: "CVE-2024-9999 is announced Monday. The TrivyDB refreshes Sunday. Six days of builds may pass without detection. The CVE is exploitable; an attacker who knows about it could target the cluster.",
      evidence_expected: [
        "TrivyDB is at the version from the previous Sunday",
        "CVE-2024-9999 is not in the scanner's database",
        "CI builds between Monday and Sunday do not detect the CVE",
        "The vulnerability is in a library the cluster uses"
      ],
      remediation_expected: [
        "Refresh the TrivyDB daily (or on every build) instead of weekly",
        "Subscribe to the CVE feed and alert on Critical CVEs in used libraries",
        "Re-scan recent builds against the updated database",
        "Document the workflow in the runbook"
      ],
      rollback_expected: [
        "There is no rollback for an undetected CVE — the CVE may have been exploited",
        "If a build with the vulnerable library was deployed, patch and re-deploy"
      ],
      explanation: "A scanner database that is refreshed weekly misses CVEs announced during the week. The fix is to refresh daily (or on every build). The vulnerability database is a critical component of the supply chain; stale data is a security gap. The operational discipline is to refresh on a known cadence and to re-scan recent builds when a Critical CVE is announced."
    },
    {
      kind: "short_answer",
      id: "kubernetes-q-lxiv-03-tools",
      prompt: "Name two open-source vulnerability scanners for container images and explain how each one is integrated into CI/CD.",
      expected_keywords: [
        "trivy",
        "grype",
        "CI",
        "exit-code",
        "severity"
      ],
      explanation: "Two open-source scanners: (1) Trivy — integrated with `trivy image --exit-code 1 --severity CRITICAL,HIGH <image>`; the CI pipeline fails on Critical/High CVEs. (2) Grype — integrated with `grype sbom:<sbom-file>` or `grype <image>`; can take an SBOM as input, which decouples SBOM generation from scanning. Both produce findings with severity, package, CVE ID, and fix version."
    }
  ]}
/>

## Production discipline

Vulnerability scanning is the operational control for
known CVEs. A defensible programme refreshes the
scanner database daily, scans every image in CI/CD,
gates the build on Critical and High CVEs, triages
findings, and documents exceptions. A cluster whose
images are scanned and gated has a vulnerability
programme that is auditable; a cluster whose images
are not scanned has a programme that is not.