Dashboard Test Cadence

The tier 1 service overview was last reviewed eighteen months ago. The reviewer had moved to a different team. The dashboard still opened during incidents. A panel that referenced http_requests_total had stopped rendering eight months earlier when the instrumentation team renamed the metric to http_server_requests_seconds_count. The panel silently showed “No data” while the surrounding panels rendered green. The dashboard was the wrong answer during a 03:00 incident that cost the team an hour of investigation before anyone noticed the panel had been empty for months.

This is what dashboard test cadence is for. Tests are useful only if they run on a rhythm. The rhythm catches drift between the dashboard’s intent and the data source’s reality. A CI step on every pull request catches drift at merge time. A monthly or quarterly review catches drift that accumulates without code changes (a renamed metric upstream, a deprecated label, a credential rotation that was not propagated). Without cadence, every test becomes a one-shot check that is right once and wrong forever after.

Dashboard test cadence is the per-tier rhythm at which correctness, variable, data source, and unit tests are run, results are reviewed, and findings are assigned to owners with a follow-up date.

What it is

The four test families introduced in the previous lessons (query correctness, variable validation, data source availability, unit consistency) are the inputs to the cadence. The cadence is the schedule on which the inputs are run and the outputs are acted upon:

  +----------------+--------------------------------+
  | Trigger        | Tests run                      |
  +----------------+--------------------------------+
  |  Pull request  |  All four tests, in CI,        |
  |                |  against staging               |
  +----------------+--------------------------------+
  |  Merge to main |  All four tests, in CI,        |
  |                |  against production            |
  +----------------+--------------------------------+
  |  Monthly       |  Tier 0 dashboards: all four   |
  |                |  tests; leadership review      |
  +----------------+--------------------------------+
  |  Quarterly     |  Tier 1 and tier 2 dashboards: |
  |                |  all four tests; owner review  |
  +----------------+--------------------------------+
  |  On close      |  Tier 3 dashboards: review;    |
  |                |  delete or promote to tier 2   |
  +----------------+--------------------------------+
  |  Ad-hoc        |  Any dashboard whose owner    |
  |                |  changes; any data source     |
  |                |  whose UID is renamed          |
  +----------------+--------------------------------+

The CI runs the tests continuously. The scheduled jobs catch what CI cannot catch: drift that accumulates without code changes (upstream metric renames, label-schema changes, data source plugin updates, credential expirations).

Dashboard test cadence has four properties:

1. Tier-appropriate. Tier 0 dashboards get a monthly cadence; tier 1 and tier 2 get quarterly; tier 3 gets on-close.
2. Owned. Every cadence entry has a named primary reviewer and a named backup. A cadence entry with one reviewer cancels itself the first time that person is unavailable.
3. Tracked. Findings from a cadence run go into a tracker with an owner and a follow-up date. Findings without owners are not findings; they are noise.
4. Closed. Findings from a cadence run are converted to follow-up pull requests within the same sprint they are filed. A cadence that produces findings that never close is a cadence that the team has stopped trusting.

A cadence that fails property 1 over-reviews or under-reviews. A cadence that fails property 2 cancels itself. A cadence that fails property 3 produces findings nobody acts on. A cadence that fails property 4 stops being a cadence.

Why a sysadmin cares

Three operational pains map directly to dashboard test cadence:

1. The lapsed cadence. A quarterly cadence that has not been honoured for eighteen months means the dashboards have drifted from the data sources. The drift is silent; the dashboard looks healthy; the on-call engineer trusts the wrong number.
2. The single-point-of-failure reviewer. A cadence entry that lists only one named reviewer cancels itself the first time that person is on holiday, on leave, or has moved teams. The cadence lapses; the dashboard drifts; the failure shape is invisible until an incident surfaces it.
3. The findings tracker nobody reads. A cadence run that produces a list of broken panels without an owner and a follow-up date is a list that gets forgotten by the next sprint. The cadence becomes a ritual; the findings are noise.

The wrong shape shows up as a dashboard that has not been reviewed in eighteen months while opening during every incident, or as a list of broken panels with no owner attached.

Cadence vs CI

CI and cadence are complementary. CI catches drift at merge time; cadence catches drift that accumulates without code changes. A dashboard that has not been touched in six months has not been re-tested by CI; the metrics it depends on may have been renamed. The cadence is the only check that finds this.

  +----------------+--------------------------------+
  | Trigger        | Catches                         |
  +----------------+--------------------------------+
  |  CI            |  Drift at merge time            |
  |                |  (new metric, new label, new    |
  |                |  panel expr)                    |
  +----------------+--------------------------------+
  |  Cadence       |  Drift that accumulates without |
  |                |  code changes (renamed metric   |
  |                |  upstream, deprecated label,    |
  |                |  credential rotation, plugin    |
  |                |  update)                        |
  +----------------+--------------------------------+

CI without cadence catches the easy drift and misses the slow drift. Cadence without CI catches the slow drift and misses the easy drift. Both are needed; the cadence is the one that finds the metric that was renamed eight months ago.

How it works

The cadence pipeline:

  scheduled job / pull request
        |
        v
  +---------------------+
  |  enumerate scope    |   tag-based: tier:0, tier:1, tier:2
  +---------------------+
        |
        v
  +---------------------+
  |  for each dashboard |
  |  in scope:          |
  |  run the four tests |   query correctness, variable,
  |                     |   data source, unit
  +---------------------+
        |
        v
  +---------------------+
  |  collect findings   |   broken panels, missing owners,
  |                     |   drift, missing units
  +---------------------+
        |
        v
  +---------------------+
  |  file findings      |   tracker issue per finding with
  |                     |   owner and follow-up date
  +---------------------+
        |
        v
  +---------------------+
  |  review meeting     |   named primary + backup reviewers;
  |                     |   findings walked; status updated
  +---------------------+
        |
        v
  +---------------------+
  |  close findings     |   findings converted to follow-up
  |                     |   pull requests within the sprint
  +---------------------+

How to configure it

The canonical pattern: a tier-based cadence calendar, a named reviewer list, a CI workflow with a schedule trigger, and a findings tracker.

The cadence calendar, in the team runbook:

# docs/runbooks/dashboard-cadence.yml
cadence:
  tier:0:
    when: 'first Monday of the month, 10:00 UTC'
    duration_minutes: 60
    primary_reviewer: 'sre-lead'
    backup_reviewer:  'engineering-lead'
    scope: 'tag = tier:0'
    tests: [query, variable, datasource, unit]
    output: 'tracker issue per finding'
    follow_up: 'within the same sprint'
  tier:1:
    when: 'first Monday of the quarter, 10:00 UTC'
    duration_minutes: 90
    primary_reviewer: 'service-owner'
    backup_reviewer:  'sre-lead'
    scope: 'tag = tier:1, owner = <team>'
    tests: [query, variable, datasource, unit]
    output: 'tracker issue per finding'
    follow_up: 'within the same sprint'
  tier:2:
    when: 'first Monday of the quarter, per component'
    duration_minutes: 45
    primary_reviewer: 'component-owner'
    backup_reviewer:  'service-owner'
    scope: 'tag = tier:2, scope = <system>'
    tests: [query, variable, datasource, unit]
    output: 'tracker issue per finding'
    follow_up: 'within the same sprint'
  tier:3:
    when: 'on incident close'
    duration_minutes: 15
    primary_reviewer: 'incident-author'
    backup_reviewer:  'on-call-lead'
    scope: 'tag = tier:3'
    tests: [query]
    output: 'delete or promote to tier:2'
    follow_up: 'before the next incident'

The CI workflow with a schedule trigger:

# .gitea/workflows/dashboard-cadence.yml
name: dashboard-cadence
on:
  schedule:
    - cron: '0 6 1 * *'   # first of the month
  workflow_dispatch:
jobs:
  cadence:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run cadence tests
        run: bash scripts/run-cadence.sh
      - name: File findings
        if: failure()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const log = fs.readFileSync('cadence.log', 'utf8');
            await github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: 'dashboard-cadence findings ' + new Date().toISOString().slice(0, 10),
              body: log,
              labels: ['dashboard-cadence', 'tier:0']
            });

The cadence run writes its findings to a file; the workflow files a tracker issue with the findings as the body. The tracker issue has a label for the cadence and a label for the tier.

The reviewer runbook:

  Per-cadence-run checklist:
    1. Open the cadence tracker issue. Walk every
       finding.
    2. For each finding: confirm an owner is named and
       a follow-up date is set.
    3. For each finding with a follow-up date in the
       past: confirm a follow-up pull request exists or
       is in flight.
    4. For dashboards without an owner tag: file a
       finding; the next review meeting assigns an
       owner.
    5. For dashboards with an owner tag whose owner
       has left the team: file a finding; the next
       review meeting reassigns the owner.
    6. Confirm the cadence calendar has a backup
       reviewer for every entry.

How to validate it

Five checks confirm the cadence discipline is live.

Severity: READ-ONLY.

# 1. Every dashboard in scope has a tier tag and an
#    owner tag. A dashboard without tags is not in
#    any cadence; that is a finding.
for tag in tier:0 tier:1 tier:2 tier:3; do
  count=$(curl -s -u admin:$ADMIN \
    "https://grafana.example.com/api/search?type=dash-db&tag=$tag" \
    | jq 'length')
  echo "$tag: $count dashboards"
done

# 2. The cadence CI workflow ran in the last cadence
#    window. A scheduled workflow that has not run is
#    a cadence that has lapsed.
gh run list --workflow dashboard-cadence \
  --limit 1 --json status,conclusion,createdAt
# [{"status":"completed","conclusion":"success",
#   "createdAt":"2026-08-01T06:00:00Z"}]

# 3. The findings tracker has open issues with the
#    dashboard-cadence label. A tracker with no open
#    issues is a tracker that has been ignored.
gh issue list --label dashboard-cadence \
  --state open --json number,title,createdAt | jq 'length'
# 4

# 4. The cadence calendar has backup reviewers for
#    every entry. A cadence entry with one reviewer
#    cancels itself.
yq '.cadence.* | {primary: .primary_reviewer,
                   backup:  .backup_reviewer}' \
  docs/runbooks/dashboard-cadence.yml

# 5. The cadence tests pass against the current
#    state. A cadence run that produces no findings
#    is a cadence run worth having.
bash scripts/run-cadence.sh
# (empty output on success)

How it can fail

Six failure shapes appear repeatedly with dashboard test cadence:

1. Cadence lapses. A scheduled CI workflow that has not run in three cycles is a cadence that has lapsed. Symptom: the cadence calendar is honoured in name but not in practice; dashboards drift without review.
2. Single-point-of-failure reviewer. A cadence entry with one named reviewer cancels itself the first time that reviewer is on leave. Symptom: the cadence skips a cycle; the next cycle’s findings include everything that drifted during the skip.
3. Findings without owners. A tracker issue with a dashboard-cadence label and no owner assigned is a finding that will not close. Symptom: the tracker accumulates open issues; the cadence becomes noise.
4. Findings without follow-up dates. A tracker issue without a follow-up date has no deadline; the work slides indefinitely. Symptom: the same findings recur in every cadence run.
5. CI skipped on pull request. A dashboard change merged without the dashboard-cadence CI workflow passing. Symptom: the merge introduced drift that the cadence would have caught but did not.
6. Owner left the team. The cadence entry lists a reviewer who has moved to a different team. Symptom: the cadence runs but the findings are assigned to a non-existent owner; nothing closes.

How to troubleshoot it

The diagnostic order:

1. Check the cadence calendar. Confirm the cadence is on the calendar with a primary and a backup reviewer. A cadence entry with one reviewer is a finding.
2. Check the CI workflow. Confirm the scheduled run is firing. A workflow that has not run in three cycles is a cadence that has lapsed.
3. Check the findings tracker. Confirm open findings have owners and follow-up dates. Findings without owners are noise; findings without dates are forever-open issues.
4. Check the dashboard tags. Confirm every dashboard has a tier tag and an owner tag. A dashboard without tags is not in any cadence; file a finding.
5. Check the reviewers. Confirm every cadence entry has a backup reviewer. A backup reviewer is the difference between a cadence that lapses once and a cadence that lapses every time the primary is unavailable.

Security implications

Dashboard test cadence has no direct security implications. The data sources, dashboards, and provisioning manifests are protected by the existing RBAC; the cadence run uses the same credentials as the CI step. The indirect risk is operational trust: a cadence that lapses is a cadence that nobody trusts, and a dashboard review that nobody trusts is worse than no dashboard review at all (because the team believes the dashboards are reviewed when they are not).

Performance implications

Dashboard test cadence has no direct performance implications on the dashboards themselves. The CI workflow is a scheduled job; the cost is the four test scripts running against staging and production. The indirect risk is review-meeting time: a cadence run that produces 50 findings takes 90 minutes to walk; a cadence run that produces 5 findings takes 15 minutes. Bounding the findings per run is a discipline the team sets in the runbook.

Production guidance

• Calendar the cadence. A scheduled CI workflow with no calendar invite is a workflow that lapses. The calendar invite has a primary and a backup; both accept the invite.
• Name backup reviewers. A cadence entry with one reviewer cancels itself the first time that reviewer is unavailable. Two named attendees (primary and backup) is the minimum.
• Track findings. A cadence run without a tracker is a cadence run that produces noise. Every finding has an owner and a follow-up date.
• Close findings fast. A cadence run that produces findings that do not close within the sprint is a cadence run that has lost credibility. The rule: every finding produces a follow-up pull request within the same sprint it is filed.
• Document the cadence in the runbook. The cadence calendar, the reviewer list, the runbook, and the findings tracker are all in the runbook. A cadence that lives in one person’s head is a cadence that lapses when that person leaves.
• Review the cadence itself. A quarterly cadence for dashboards needs a quarterly cadence for the cadence (a meta-review). The meta-review asks: are we reviewing the right dashboards at the right cadence? Is the runbook up to date? Are the reviewers still in role?

Verification

You should now be able to answer:

• What is the difference between CI-driven tests and a scheduled cadence?
• Which tier is reviewed monthly, and which is reviewed on incident close?
• Why does every cadence entry need a primary and a backup reviewer?
• What is the failure shape of a cadence run that produces findings without owners?
• How does the cadence calendar convert a finding into a follow-up pull request?

Quiz