Wallpaper Dashboards

A 14-inch monitor in the corner of the NOC is showing a dashboard called “Production - All Services (legacy)”. It has 47 panels. Every panel is green. The date in the corner of each panel shows that the last real change to the underlying query was in 2019. The person whose name appears in the dashboard description left the company in 2022.

When an alert fired at 03:14 that morning, the operator did not look at that dashboard. They looked at the per-service dashboard they had bookmarked, the one with their team name in the title. The wallpaper had become a piece of the office furniture.

A wallpaper dashboard is a dashboard whose primary function has become decoration. It exists, it loads, it has not been deleted, and no one is reading it. It survives because removing it would require a decision about who owns it, and no one is the owner.

What it is

A wallpaper dashboard in production operations is a Grafana dashboard that meets all four of these criteria:

1. No named owner. The dashboard has no team or individual listed in its description, no tag matching the team’s namespace, and no contact in the dashboard annotations.
2. No traffic. It has not been opened or shared as a link in more than 90 days. In Grafana 11.x the audit log records dashboard views against the user’s API token; the count is observable from the Grafana API.
3. No alerts. No alert rule references it. No alert message has a deep link to it. When something is on fire, no one is sent here.
4. Wide and shallow. Panel count exceeds what a single operator can scan in 30 seconds (typically more than 20 panels), and the panels do not drill into one another.

The contrast is the dashboards that survive. The surviving ones have an owner line, three to eight panels, and an edit date in the last quarter. They are linked from alerts. They are bookmarked.

Why a sysadmin cares

Three production costs come from wallpaper dashboards:

• Operator attention is a budget. When the on-call engineer scans twenty dashboards during an incident, each one needs to pay for its place. A wallpaper dashboard that looks identical to a working one trains the engineer to distrust the whole layout. The distrust spreads to real signals.
• Query load is real. Every time a dashboard is opened, Grafana 11.x evaluates every panel query at the panel’s configured interval against the active time range. A 40-panel dashboard opened three times a day by an audit screen pulls 120 panel evaluations per day per source. With Loki and Tempo in the mix, that fan-out compounds.
• Wallpaper hides signal. The most dangerous failure mode is not the dashboard that is all green. It is the dashboard that has one stale red panel buried under a “if red, ignore” comment from 2021. The next operator does not read the comment, and the red panel is the actual production issue.

How it works

The failure shape that produces wallpaper dashboards is well understood:

   Service is built
        |
   +----+----+
   |         |
 Panel    Panel
 added    added
   |         |
   v         v
 Dashboard  Dashboard
 grows       grows
   |         |
   +----+----+
        |
   Service is
   deprecated
        |
   +-----+
   |
 Dashboard
 is not
 removed
   |
   v
 Wallpaper

The steps that produce wallpaper are:

1. A service launches. A dashboard is created to monitor it.
2. The team adds panels as questions come up. The dashboard grows to 30+ panels.
3. The service is rewritten or replaced. The old dashboard is not pointed at the new service.
4. The author leaves or rotates to another team. The dashboard has no owner.
5. Grafana’s “starred dashboards” list moves on. The legacy dashboard loses its last bookmark.
6. Nothing deletes the dashboard because no one has the mandate.

The anti-pattern is sustained by two design omissions: no expiry/retirement policy, and no owner field that the team is required to maintain.

Under the hood

In Grafana 11.x the dashboard JSON lives in the database (or in the provisioning file system if provisioned). The schema has matured across versions and the relevant fields for wallpaper detection are:

• meta.createdBy — the user who created the dashboard.
• meta.updatedBy — the user who last saved a change.
• meta.updated — the timestamp of the last save.
• dashboard.tags — array of tag strings.
• dashboard.annotations.list — legacy alert annotations.
• dashboard.templating.list — variables, often a strong tell that the dashboard is unused if no recent edit touched them.
• dashboard.panels[*].id — the panel count is the count of top-level rows plus rows’ panels.

Grafana 11.x exposes both the HTTP API and the provisioning loader. The HTTP API is appropriate for inventory and audit; the provisioning loader is appropriate for new dashboards and for enforcing the discipline at write-time.

A dashboard provisioner such as Grafana’s own file-based provisioning, or a sidecar like grafana-schemas, can refuse to provision a dashboard that lacks an owner tag or whose panel count exceeds the configured ceiling.

How to configure it

The replacement discipline has three parts: an owner field, a panel-count ceiling, and a retirement pipeline.

Owner field

Tag every dashboard with the team name. Enforce this through provisioning rather than through policy:

# /etc/grafana/provisioning/dashboards/platform-team.yaml
apiVersion: 1
providers:
  - name: platform
    orgId: 1
    folder: Platform
    type: file
    disableDeletion: false
    updateIntervalSeconds: 60
    options:
      path: /var/lib/grafana/dashboards/platform
      foldersFromFilesStructure: true

# /var/lib/grafana/dashboards/platform/checkout-service.json
# Frontmatter-style block embedded in dashboard JSON
# (or as a separate annotation in the team's convention)
{
  "tags": ["team:checkout", "env:prod", "owner:oncall-checkout"],
  "title": "Checkout Service - Overview",
  "uid": "checkout-overview",
  "schemaVersion": 39,
  "panels": [ ... ],
  "templating": { "list": [] },
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "enable": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  }
}

Panel-count ceiling

Reject provisioning files with too many panels at the CI level. The check runs against the JSON before it lands:

# .github/workflows/dashboards-lint.yaml (excerpt)
- name: Enforce panel-count ceiling
  run: |
    for f in $(find dashboards -name '*.json'); do
      count=$(jq '.panels | map(.panels //= .panels, .. | objects | select(has("id"))) | length' "$f")
      if [ "$count" -gt 20 ]; then
        echo "::error::$f has $count panels; ceiling is 20"
        exit 1
      fi
      owner=$(jq -r '.tags[]? | select(startswith("owner:"))' "$f")
      if [ -z "$owner" ]; then
        echo "::error::$f has no owner:* tag"
        exit 1
      fi
    done

Retirement pipeline

A scheduled job runs weekly. For each dashboard:

1. Look up the owner tag.
2. Query the Grafana audit log for views in the last 90 days.
3. If view count is zero and the alert manager has no rules referencing the dashboard UID, mark it for retirement.
4. Notify the owner channel on Slack with a link to a one-click archive job.
5. If no response in 14 days, archive the dashboard (Grafana 11.x supports the isStarred=false archive state via the API; the dashboard is moved to a Trash folder and removed from listings).

How to validate it

# READ-ONLY. List dashboards older than 365 days with no
# owner:* tag using the Grafana HTTP API.
GRAFANA_URL="https://grafana.internal"
GRAFANA_TOKEN="${GRAFANA_API_TOKEN}"

curl -sS -H "Authorization: Bearer ${GRAFANA_TOKEN}" \
  "${GRAFANA_URL}/api/search?type=dash-db&limit=5000" \
  | jq -r '
      .[]
      | select((.tags // []) | map(select(startswith("owner:"))) | length == 0)
      | "\(.uid)\t\(.title)\t\(.folderTitle)"
    ' \
  | head -20

Expected output (illustrative):

legacy-jvm-heap  JVM Heap (legacy)   Trash
jenkins-build-duration  Jenkins build time (legacy)  Trash
opentsdb-rewrite-attempt  OpenTSDB bridge attempt  Trash

Validate the retirement pipeline by archiving a known wallpaper dashboard and confirming it disappears from api/search:

# CONFIGURATION. Archive a dashboard by UID.
curl -sS -X POST \
  -H "Authorization: Bearer ${GRAFANA_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"message": "Wallpaper; archived after 90 days of zero views."}' \
  "${GRAFANA_URL}/api/dashboards/uid/legacy-jvm-heap"

# READ-ONLY. Confirm the dashboard no longer appears.
curl -sS -H "Authorization: Bearer ${GRAFANA_TOKEN}" \
  "${GRAFANA_URL}/api/search?query=legacy-jvm-heap" \
  | jq 'length'
# 0

How it can fail

Six failure shapes attached to wallpaper dashboards:

1. The buried red panel. A panel that has been red for six months is the only signal of a real problem. Operators have trained themselves to skip the dashboard. The fix is owner re-attribution, not more panels.
2. The orphaned alert link. An alert rule’s runbook_url field references a dashboard that has been deleted or archived. The on-call engineer clicks the link during an incident and lands on a Grafana 404. Retirement must update the alert rule at the same time.
3. The high-cardinality panel. A 40-panel dashboard that fans out into 100+ label combinations per panel will time out evaluation in Prometheus when the active time range is six months. The dashboard looks broken; the underlying cause is the panel layout.
4. The provisioning split. A team uses both HTTP API dashboards and file-provisioned dashboards for the same service. The two copies drift. One of them is the wallpaper.
5. The browser memory crash. Grafana 11.x in a Chromium tab allocates one iframe per panel; 47 panels with repeat: 'instance' over 80 instances exceeds the tab memory budget. The operator’s browser tab dies when they open the dashboard.
6. The “shared screen” wall. A NOC screen shows a wallpaper dashboard 24/7. When a real signal appears on a different dashboard, the wall does not change colour. Operators stop noticing when the wall does change colour.

How to troubleshoot it

The diagnostic order when wallpaper has caused an incident:

1. Was the operator pointed at the wallpaper or the working dashboard? Inspect the alert rule’s runbook_url and any chat-ops deep link. If it points at an archived UID, fix the alert first.
2. Is the wallpaper actually up to date? A dashboard with no recent edit but a recent query result might still be technically correct. Confirm by opening the dashboard and checking the data source response times.
3. Is the working dashboard present and bookmarked? The replacement must already exist and be linked from the alert. Otherwise retirement turns an ignored dashboard into a missing one.
4. Are any panels red? Run curl -sS ... | jq over each panel query at the panel’s time range; flag any non-empty error result. Treat a red panel as a real signal regardless of how old the dashboard is.
5. Has the owner been notified? Open a ticket against the team channel that owns the owner:* tag. The retirement is owned by the same team that owns the dashboard.

Security implications

A wallpaper dashboard is, by definition, a dashboard no one looks at. That makes it an attractive place for sensitive data to leak. A panel added in 2018 that prints environment-variable values, a row of stats that contains a service-account email address, a Loki query that returns lines from a service that should have been removed: all of these survive in wallpaper indefinitely. The audit posture of the dashboard set is only as good as the worst wallpaper.

Two controls apply:

• Provisioning-based ACLs. Grafana 11.x supports folder-level permissions. Folder every dashboard into a folder owned by one team and apply the team’s role bindings. Wallpaper tends to land in the default folder; that is the worst place for it.
• Review on a cadence. A quarterly review walks the dashboard list, removes wallpaper, and forces the owner’s re-confirmation of the dashboard permissions.

Performance implications

The Grafana server evaluates every panel on every dashboard load and on a per-panel refresh cadence. A 47-panel dashboard opened ten times in a 30-minute incident pulls 470 evaluations against the data sources. Each evaluation is one Prometheus query, one LogQL query, or one Tempo query. With Loki and Tempo sources this fan-out is the largest source of dashboard-related query load on the data plane.

The browser side compounds it: each panel renders as a separate chart inside a CSS grid; each chart has its own subscription to the data source’s WebSocket / event stream. Chromium allocates non-trivial JavaScript heap per chart, and Grafana 11.x does not aggressively garbage-collect when the dashboard is in the background. Twenty minutes of an open wallpaper dashboard in the background consumes more heap than ten real dashboards sequenced properly.

Production guidance

• Cap panel counts at 20 for overview dashboards and 12 for per-service dashboards. Larger layouts do not pay for themselves.
• Require an owner:teamname tag on every provisioned dashboard. Reject provisioning files that lack it.
• Run the audit-log view count against every dashboard once a quarter. Anything under five views in 90 days is a candidate.
• Tie alert runbook_url to dashboard UID, not title. UIDs do not change when the title is edited.
• Treat retirement as part of the same change as any architectural change. New dashboard goes in, old dashboard goes out.

Verification

You should now be able to answer:

• What four criteria mark a dashboard as wallpaper?
• How does the Grafana 11.x audit log distinguish a working dashboard from one no one opens?
• Why is the alert rule’s runbook_url UID-pinned rather than title-pinned?
• What is the right order when retiring a wallpaper dashboard that an alert still points at?
• What is the operator-side cost of leaving a 47-panel wallpaper open during an incident?

Quiz