Notification controller and webhooks — the alerting layer

notification-controller is the alerting layer of Flux. It owns three CRDs - Provider, Alert, and Receiver - that turn Flux events into Slack messages, Teams webhooks, Discord posts, or generic HTTP callbacks. The same controller exposes an inbound webhook endpoint that Git hosts call to trigger an immediate reconcile.

What notification-controller does

The controller has two faces:

• Outbound. Receives events from source-controller, kustomize-controller, and helm-controller. Matches each event against an Alert, identifies a Provider, fans the message out.
• Inbound. Exposes an HTTP endpoint (/hook/:guid) that Git hosts call on push. The endpoint writes an annotation that forces the next reconcile out of cycle.

flowchart LR
    SC["source-controller"] --> EV["events"]
    KC["kustomize-controller"] --> EV
    HC["helm-controller"] --> EV
    EV --> NC["notification-controller"]
    NC --> AL["Alert: matches event"]
    AL --> P["Provider"]
    P --> Slack["Slack"]
    P --> Teams["Teams"]
    P --> Discord["Discord"]
    P --> WH["Generic webhook"]
    GH["Git host"] -->|"/hook/:guid"| NC
    NC -->|"annotation"| GR["GitRepository / Kustomization"]

The outbound path replaces polling flux get kustomizations; the inbound path replaces waiting for the source interval.

The three CRDs

Provider declares a destination and its credentials:

apiVersion: notification.toolkit.fluxcd.io/v1beta2
kind: Provider
metadata:
  name: slack
  namespace: flux-system
spec:
  type: slack
  channel: prod-deploys
  address: https://slack.example.com
  secretRef:
    name: slack-webhook

The provider type determines the message format and required credentials. slack, msteams, discord, github, gitlab, generic are supported.

Alert declares which events go where:

apiVersion: notification.toolkit.fluxcd.io/v1beta2
kind: Alert
metadata:
  name: on-call
  namespace: flux-system
spec:
  providerRef:
    name: slack
  eventSeverity: error
  eventSources:
    - kind: Kustomization
      name: '*'
  exclusionList:
    - ".*pulled.*new.*image.*"

The alert matches every error event from every Kustomization and routes it to the slack provider. The exclusionList drops matching messages.

Receiver declares an inbound webhook endpoint and the resource it annotates:

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
  name: github
  namespace: flux-system
spec:
  type: github
  events:
    - "ping"
  secretRef:
    name: github-webhook-secret
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1
      kind: GitRepository
      name: apps

The receiver is exposed at http://notification-controller.flux-system.svc.cluster.local:8080/hook/$GUID. The Git host sends a ping (or push) event to that URL with the secret. The controller authenticates the call, identifies the resources in spec.resources, and writes a reconcile annotation on each.

End-to-end: source-controller to Slack

sequenceDiagram
    participant SC as source-controller
    participant NC as notification-controller
    participant A as Alert
    participant P as Provider
    participant Slack as Slack
    SC->>NC: error event
    NC->>A: match event
    A->>P: send to slack provider
    P->>Slack: POST /services/.../incoming-webhook
    Slack-->>NC: 200 OK

The controller writes every event to its audit log regardless of whether an Alert matched. flux logs against the notification-controller namespace shows every event.

Under the hood

The controller is small because the work is mostly protocol translation. Outbound: format the event as Slack JSON or generic webhook JSON. Inbound: validate the HMAC signature, identify the resources, write annotations.

Production discipline

1. eventSeverity: error is the production default for Alert. Info events flood Slack; errors need a human eye.
2. exclusionList is curated, not empty. A noisy stream is one operators mute; a missing event is one nobody catches.
3. The Receiver URL is exposed through an ingress with TLS and an allowlist. No public webhook without a source IP allowlist.

Cross-course references

• Kubernetes for Production Sysadmins - Part XXIII (Ingress and TLS) covers the ingress the Receiver URL terminates at.
• Git, CI/CD & GitOps for Infrastructure Engineers - Part LXXV (Pull-based Deployment) is the pattern this controller implements.

Quiz