Tags and Filtering

A single dashboard renders cleanly in staging and silently returns “no data” in production. The dashboards are identical. The data source names are identical. The only difference is the data source UID — staging’s UID is prom-stg-eu and production’s UID is prom-prod-eu. The dashboard references the name, the variable query filters by a tag, and the environment never resolved.

The same dashboard imports a Loki data source whose name field is Loki (capitalised, with the explicit Grafana legacy form). A new data source with name: loki (lowercase, the canonical form) is added six months later. The alerting engine silently picks the wrong source for some rule groups. The lesson this time is about the metadata attached to a data source: tags, the BuiltIn marker, the legacy naming conventions that still affect some features, and the UID that governs the proxy URL.

What it is

A Grafana data source is more than a name, a URL, and a credential. It also carries:

• A UID — the stable identifier used in the proxy URL, in dashboards, and in derived-field links. UID changes break every reference.
• A name — the human-readable label in selectors. Names can be renamed safely; UIDs cannot.
• Tags — a list of strings in jsonData.tags. Tags are how a dashboard variable filters “show me the Prometheus for this environment”. Tags do not affect the proxy URL.
• A BuiltIn marker — Grafana’s own preinstalled data sources (TestData, Grafana Cloud integrations, the ---prefixed service data sources) carry this marker. The UI uses it to hide built-in sources from selectors by default.
• A legacy naming convention — Grafana’s oldest data source types (Loki, MSSQL, Elasticsearch, Prometheus, InfluxDB, MySQL, Postgres, Graphite) used capitalised display names. Grafana 11.x still recognises these forms in a few internal places. A new data source with the lowercase form is the canonical shape; the capitalised form is a compatibility alias.

Why a sysadmin cares

Data source metadata is the configuration surface where dashboards, alerting, and Explore decide which upstream they talk to. Four production shapes appear repeatedly:

• UID drift after a rename. An operator renames a data source from prom-prod-eu to prom-prod-eu-west because the region split. The name change is harmless. The UID change breaks every dashboard and derived-field reference in the platform.
• Tags that no dashboard reads. Tags are added for future-proofing (“we will filter by environment later”) and are never actually consumed. The metadata cost is paid on every reload; the benefit is zero.
• BuiltIn data sources leaking into selectors. A TestData data source with the BuiltIn marker is supposed to be hidden from selectors, but a panel that filters by tag = "prometheus" shows it anyway because the filter does not exclude BuiltIn sources.
• Legacy Loki and MSSQL names causing dual resolution. Two data sources with the same name (one capitalised, one lowercase) are treated as two distinct sources by Grafana 11.x, but a feature that resolves by display name picks one and a feature that resolves by internal type picks the other. Alerting rules silently pick the wrong one.

How it works: metadata in motion

   grafana.ini            provisioning yaml              runtime api
   -----------            ----------------              ------------
                          name: prom-prod-eu
                          uid:  prom-prod-eu           -----> GET /api/datasources
                          type: prometheus                       |
                          jsonData:                              v
                            tags: [prod, eu]        {
                            tlsAuthWithCACert:        "id": 1,
                            true                       "uid": "prom-prod-eu",
                          }                              "name": "prom-prod-eu",
                          secureJsonData:                 "type": "prometheus",
                            tlsCACert: |...                 "tags": ["prod", "eu"],
                            basicAuthPassword: ${...}       "isDefault": true
                          }                              }

Three observations:

1. Tags are stored on the data source, not on the dashboard. A dashboard variable query of the form prometheus (the data source type) and filter tag = prod returns the data sources that match both. Tags are how dashboards ask “give me the production Prometheus” without naming it.
2. UIDs govern the proxy URL. Every panel and Explore query resolves to /api/datasources/proxy/uid/<uid>/.... The UID is set at provisioning time and is stable for the life of the data source. The data source proxy is the authoritative answer to “which Grafana-configured upstream does this query target”.
3. BuiltIn is a UI-side convention. Grafana’s preinstalled data sources (TestData, Grafana Cloud integrations, the ---prefixed services) carry BuiltIn in their tags. The data source selector UI hides them by default. They remain reachable through their UID.

How to configure it

# /etc/grafana/provisioning/datasources/prometheus.yml
apiVersion: 1

datasources:
  - name: prom-prod-eu
    uid: prom-prod-eu          # stable; never reuse across instances
    type: prometheus
    access: proxy
    orgId: 1
    url: https://prom-prod-eu.internal:9090
    isDefault: true
    editable: false

    jsonData:
      # Tags drive the dashboard variable filter.
      # Convention: environment + region + role.
      tags:
        - prod
        - eu
        - metrics

      tlsAuth: false
      tlsAuthWithCACert: true
      tlsSkipVerify: false
      timeInterval: 15s
      httpMethod: POST

    secureJsonData:
      tlsCACert: |
        -----BEGIN CERTIFICATE-----
        MIIDazCCAlOgAwIBAgIUJx...
        -----END CERTIFICATE-----
      basicAuthPassword: ${PROM_PASSWORD}

  - name: prom-stg-eu
    uid: prom-stg-eu           # different UID; same shape
    type: prometheus
    access: proxy
    orgId: 1
    url: https://prom-stg-eu.internal:9090
    isDefault: false
    editable: false
    jsonData:
      tags:
        - staging
        - eu
        - metrics
      tlsAuth: false
      tlsAuthWithCACert: true
      timeInterval: 15s
      httpMethod: POST
    secureJsonData:
      tlsCACert: |
        -----BEGIN CERTIFICATE-----
        MIIDazCCAlOgAwIBAgIUJx...
        -----END CERTIFICATE-----
      basicAuthPassword: ${PROM_STG_PASSWORD}

The dashboard variable that consumes the tags:

# In a Grafana dashboard JSON, a variable of type "datasource":
{
  "name": "ds",
  "type": "datasource",
  "query": "prometheus",
  "filters": [
    { "tags": ["eu"] }
  ],
  "current": { "text": "prom-prod-eu", "value": "prom-prod-eu" }
}

A few production notes on the options:

• uid is set in YAML and never reused. Two data sources with the same UID is a provisioning error; Grafana rejects the second one with “data source with the same UID already exists”.
• Tags are strings, lowercase, hyphen-separated. Grafana does not enforce a format, but the convention env-region is the shape that most dashboards filter on.
• A tag that no variable filter consumes is dead metadata. Remove tags that nothing reads.
• isDefault selects the implicit target for new panels. Setting it on more than one Prometheus per Grafana instance produces surprising behaviour in Explore.
• editable: false prevents click-ops from drifting the metadata. Production data sources should not be editable in the UI.

How to validate it

# READ-ONLY: list all data sources with their tags.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  http://grafana.internal:3000/api/datasources | jq '.[] | {uid, name, type, tags}'
# [
#   {
#     "uid": "prom-prod-eu",
#     "name": "prom-prod-eu",
#     "type": "prometheus",
#     "tags": ["prod", "eu", "metrics"]
#   },
#   {
#     "uid": "prom-stg-eu",
#     "name": "prom-stg-eu",
#     "type": "prometheus",
#     "tags": ["staging", "eu", "metrics"]
#   }
# ]

# READ-ONLY: the proxy URL uses the UID, not the name.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  --data-urlencode 'query=up' \
  http://grafana.internal:3000/api/datasources/proxy/uid/prom-prod-eu/api/v1/query
# {"status":"success","data":{"resultType":"vector","result":[...]}}

# READ-ONLY: filter the API by tag (returns data sources whose
# tags include the requested value).
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  'http://grafana.internal:3000/api/datasources?tag=prod' | jq '.[] | .uid'
# "prom-prod-eu"

# CONFIGURATION: reload Grafana provisioning.
sudo systemctl reload grafana-server

A clean validation: the UID is set, the tags are present, the proxy URL works, and the data source selector in the UI shows the source when the dashboard variable filter matches.

How it can fail

The most expensive data-source metadata failure modes from real production incidents.

1. UID regenerated after a rename. An operator renames prom-prod-eu to prom-prod-eu-west and the UID is omitted from the YAML. Grafana regenerates the UID from the new name. Every dashboard that references the old UID renders “data source not found”. The symptom is widespread panel failures correlated with the rename commit.
2. Tags inconsistent across environments. Production carries tags: [prod, eu]; staging carries tags: [staging, eu]. A dashboard variable filters by eu and resolves correctly in both. A different variable filters by prod and resolves correctly in production but fails to match in staging because staging has no prod tag. The symptom is “dashboard works in prod, fails in staging”.
3. BuiltIn marker added by mistake. A new data source includes BuiltIn in its tag list. The data source selector UI hides it. Dashboards that reference the source by UID work; new panels cannot be created against it. The symptom is “the data source is configured but the UI refuses to show it”.
4. Legacy capitalised name conflicts with lowercase form. Two data sources exist: Loki (legacy form) and loki (canonical form). The alerting engine resolves by internal type and picks the lowercase; a feature that resolves by display name picks the capitalised. The symptom is “alerting routes to one source, dashboards route to the other”.
5. Tag added after the dashboard variable is written. A new tag metrics-gen2 is added to the data source after the dashboard variable filter is committed. The variable does not pick up the new tag because the filter is hard- coded. The symptom is “the data source has the right tags but the dashboard does not see them”.
6. Reused UID across instances. A staging data source is provisioned with the same UID as a production data source (a copy-paste error). Grafana rejects the second provisioning with an error. The symptom is “staging provisioning fails; the operator reverts; production silently drifts”.

How to troubleshoot it

The diagnostic order is “is the UID stable?”, “are the tags correct?”, “is the BuiltIn marker accidental?”, “is the legacy naming conflict resolved?”.

1. Confirm the UID. GET /api/datasources/uid/<uid> and GET /api/datasources (which lists every UID). If a dashboard is failing, the first check is whether the UID in the dashboard JSON matches the UID in the provisioning YAML.
2. Confirm the tags. GET /api/datasources?tag=<value>. A missing tag returns no data sources; an extra tag returns too many.
3. Check the BuiltIn marker. Inspect the tags array for the data source. Remove BuiltIn if it was added by mistake.
4. Resolve legacy naming conflicts. A capitalised Loki next to a lowercase loki is a configuration bug. Pick one and remove the other.
5. Inspect the dashboard variable filter. The variable filter is JSON inside the dashboard JSON. Compare it against the tag list in the data source YAML; a missing tag in the YAML is a silent failure.
6. Inspect Grafana’s logs. /var/log/grafana/grafana.log records provisioning reloads. A “data source with the same UID already exists” error appears as a single line with the conflicting UID.

Security implications

• Tags are not security boundaries. A tag of prod does not restrict who can query the data source. RBAC controls the access; tags filter the dashboard selector.
• UIDs are guessable. A Grafana with predictable UIDs (prom-prod-eu) lets an attacker enumerate data sources. Consider UUIDs for high-value data sources, especially when the data source URL is itself sensitive.
• editable: false prevents the team from accidentally loosening the TLS posture through the UI. The team can still query the data source; they cannot reconfigure it without a provisioning reload.
• BuiltIn is a UI affordance, not a security feature. A data source marked BuiltIn is reachable by UID regardless of its visibility in the selector.

Performance implications

• Tag-based variable queries are not cached. Every dashboard refresh re-queries the data source selector. Tags are inexpensive; the cost is dominated by the variable regex.
• UID-based proxy URLs are stable. The proxy route table is rebuilt on provisioning reload; in-between reloads, the route is a constant-time hash lookup.
• BuiltIn filtering is UI-only. It does not affect query performance; it only affects which sources the selector displays.
• Legacy naming aliases are resolved once at provisioning time. The cost is paid at reload, not at query time.

Production guidance

• Pin the UID in YAML. Renaming a data source is fine; changing the UID breaks every reference.
• Use a tag convention: env-region-role. Document it in the team’s instrumentation guide.
• Mark TestData and the preinstalled ---prefixed data sources with BuiltIn; never mark a production data source as BuiltIn.
• Pick one form for each legacy name. Either Loki or loki, not both.
• Validate UID stability in CI. A pre-commit hook that compares the dashboard JSON’s data source references against the provisioning YAML’s UIDs catches the silent regeneration shape.
• Reload Grafana provisioning through the GitOps pipeline.
• Run a periodic exercise: pick one dashboard per environment, confirm the variable filter resolves to the right UID, and confirm the proxy URL works end-to-end.

Verification

You should now be able to answer:

• What does the BuiltIn marker do, and what is the right shape for it in production data sources?
• Why does a UID change break every dashboard reference, while a name change does not?
• What does a tag-driven dashboard variable query actually filter on, and which fields of the data source does it read?
• When does the legacy capitalised name (Loki, MSSQL) still matter in Grafana 11.x?

Quiz