Loki as a Data Source

A log panel renders 200,000 lines of an outage’s worth of error log entries. The browser freezes for thirty seconds. The engineer on call has to scroll past 199,500 lines that are not the line they actually need. Loki is happy; it served the data. Grafana is not happy; it asked for too much.

A separate engineer opens the same data source in Explore, types {job="app"} | json | latency_ms > 1000, and gets no result. The line is in there. The query is fine. The data source is provisioned with maxSeries: 0 (the default), which means Loki returns the first chunk without complaint and the metric-style aggregation silently returns nothing useful.

The lesson this time is about the controls between “Loki has the data” and “Grafana can show it”: the YAML that declares the data source, the caps that protect the UI and Loki, the derived-fields framework that turns a log line into a pivot, and the timeout that decides whether a slow query is a Grafana outage or a Loki outage.

What it is

A Loki data source in Grafana is a named, configured client of the Loki HTTP API. The provisioning file declares the data source’s URL, type, credentials, TLS posture, and a small set of Loki-specific behavioural options. Once provisioned, Grafana runs every Logs panel and Explore query through a server-side proxy at /api/datasources/proxy/uid/<uid>/.... The credentials live in secureJsonData on the server.

In Grafana 11.x the canonical data source type is loki. The older form Prometheus style with type: prometheus and a Loki URL does not work; Loki has its own query language (LogQL), its own metric-aggregation semantics, and its own derived-fields framework, and the data source type is the switch that turns all of those on.

Why a sysadmin cares

Loki data source misconfigurations are the most common cause of “logs are slow” complaints that are not Loki’s fault. Four production shapes appear repeatedly:

• Unbounded maxLines. A panel with maxLines: 5000 (the UI maximum) against a noisy service renders five thousand log lines per refresh. The browser tab consumes hundreds of megabytes of memory; the engineer on call loses the ability to scroll. Loki is fine; the UX is broken.
• Unset maxSeries. A metric-style LogQL query like quantile_over_time(0.99, \{job="app"\} | json | unwrap latency [5m]) returns thousands of series. Loki happily streams them; the panel either renders a wall of indistinguishable lines or silently truncates to the first chunk. The dashboard is “kind of correct” but the answer is invisible.
• Missing or wrong derived fields. A log line contains a traceID but the data source has no derived-field rule for it. The on-call engineer copies the trace ID by hand, opens Tempo, and pastes it. The correlation pivot that Grafana advertises is not configured; the cost is paid every investigation.
• Query timeout below the data-source default. A Grafana query timeout of 30 seconds against a Loki doing a 24-hour range query with a heavy label-match returns context deadline exceeded. The on-call concludes Loki is broken; Loki is in fact serving the query correctly inside its own budget, just slower than Grafana is willing to wait.

How it works: the request path

   browser        grafana-server           loki
   -------        --------------           ----
      |                |                    |
      |--LogQL query-->|                    |
      |  /api/ds/proxy |                    |
      |  /uid/loki/    |                    |
      |  api/v1/query  |--GET /loki/api/...|
      |  ?query={...}  |  BasicAuth + TLS   |
      |                |<--stream chunks---|
      |<--panel data---|                    |

Three observations:

1. The browser never sees the credentials. Every log query from the browser is rewritten by Grafana into a server-side call to /api/datasources/proxy/uid/<uid>/loki/api/v1/.... The path is the Loki HTTP API path; Grafana adds the auth headers and the query timeout.
2. Loki returns streams, not single-line responses. A query against a 24-hour range returns a stream of chunks; Grafana reads the stream until maxLines is reached, until the upstream closes the stream, or until the query timeout fires. Whichever happens first shapes the panel.
3. Derived fields are computed after the stream lands. Grafana parses each line, applies the regex from each derived-field rule, and rewrites the line into one or more links. The parsing cost is paid per line per render; a malformed regex silently disables that field.

How to configure it

# /etc/grafana/provisioning/datasources/loki.yml
apiVersion: 1

datasources:
  - name: loki-prod-eu
    uid: loki-prod-eu
    type: loki
    access: proxy
    orgId: 1
    url: https://loki-prod-eu.internal:3100
    isDefault: false
    editable: false

    basicAuth: true
    basicAuthUser: grafana-reader
    jsonData:
      # Cap on log lines returned per query.
      # 1000 is the default; raise only when the operator
      # has accepted the browser-memory cost.
      maxLines: 1000

      # Cap on series returned by a metric-style LogQL query.
      # 100 is the conservative default for dashboards;
      # raise for ad-hoc Explore queries.
      maxSeries: 100

      # Query timeout. The default (60s) is the Grafana server
      # timeout; Loki's own --query_timeout is independent.
      # Set Loki's --query.timeout to something below this.
      timeout: 60

      tlsAuth: false
      tlsAuthWithCACert: true
      tlsSkipVerify: false

      # Derived fields turn a log line into a pivot.
      # Each rule extracts a value via regex and produces a
      # link using a template that may reference ${__value.raw}.
      derivedFields:
        - name: traceID
          matcherRegex: '"traceID":"([0-9a-f]{32})"'
          url: '$${__value.raw}'
          urlDisplayLabel: 'Open in Tempo'
          datasourceUid: tempo-prod-eu
          internalLink:
            expr: '${__value.raw}'
            datasourceUid: tempo-prod-eu
        - name: level
          matcherRegex: '"level":"(debug|info|warn|error|fatal)"'
          url: '$${__value.raw}'
          urlDisplayLabel: 'Filter by level'
          datasourceUid: loki-prod-eu
          internalLink:
            query: '{job="$${__var.job:json}"}->level="${__value.raw}"'
            datasourceUid: loki-prod-eu
    secureJsonData:
      tlsCACert: |
        -----BEGIN CERTIFICATE-----
        MIIDazCCAlOgAwIBAgIUJx...
        -----END CERTIFICATE-----
      basicAuthPassword: ${LOKI_PASSWORD}

A few production notes on the options:

• type: loki. Anything else (including prometheus with a Loki URL) produces a data source that Grafana treats as Prometheus and breaks on the first derived-fields query.
• maxLines is a UI-side cap. The Loki stream is not truncated server-side; the proxy stops reading after maxLines. Raising this to 5000 for dashboards that “need more lines” trades server memory and browser memory for very little signal.
• maxSeries is the cap on distinct time-series returned by a metric-style LogQL query. Loki is the source of truth for the number; Grafana cuts off after maxSeries to protect the panel. Setting it to 0 is the developer’s friend; setting it to a sensible number is the operator’s friend.
• derivedFields is an array, not a map. Order matters when multiple rules could match the same line; the first match wins.
• internalLink.datasourceUid is the UID of the destination data source. A misspelled UID produces a link that fails at click time, not at provisioning time.
• internalLink.query is a LogQL expression for “follow this field to other lines”. The template variables ${__value.raw} (the matched substring) and ${__var.<label>} (a stream label from the current line) are the standard hooks.

How to validate it

# READ-ONLY: the data source is provisioned and reachable.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  http://grafana.internal:3000/api/datasources/uid/loki-prod-eu
# {
#   "id": 2,
#   "uid": "loki-prod-eu",
#   "name": "loki-prod-eu",
#   "type": "loki",
#   "url": "https://loki-prod-eu.internal:3100",
#   "access": "proxy",
#   ...
# }

# READ-ONLY: the health check returns one of four states.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  http://grafana.internal:3000/api/datasources/uid/loki-prod-eu/health
# {"message":"Data source is working","status":"success"}

# READ-ONLY: a minimal LogQL query through the proxy.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  --data-urlencode 'query={job="varlogs"} |= "level=error"' \
  http://grafana.internal:3000/api/datasources/proxy/uid/loki-prod-eu/loki/api/v1/query
# {"status":"success","data":{"resultType":"streams","result":[...]}}

# READ-ONLY: a derived-field link renders correctly from the panel.
# In the UI: click the trace ID label on any log line and
# confirm the link opens a Tempo trace with the same trace ID.

# CONFIGURATION: reload Grafana provisioning.
sudo systemctl reload grafana-server

A clean validation: the UID is present, the health check is green, the proxy returns Loki streams, and a derived-field link opens in the configured destination data source. Each failure mode below maps to one of these signals failing.

How it can fail

The most expensive Loki data source failure modes from real production incidents.

1. maxLines set too high. A panel with maxLines: 5000 renders fine in development against a quiet service. In production the same query against a busy service consumes the engineer’s browser tab for thirty seconds per refresh. The symptom is intermittent UI freezes correlated with the panel’s refresh interval.
2. maxSeries: 0 against a metric-style query. A quantile_over_time(...) query returns thousands of series. Loki streams them; Grafana truncates silently. The symptom is a panel that “looks fine but is missing most of the answer” — a partial render rather than an error.
3. Derived-field regex does not match. A regex change in the log format (a trace_id instead of traceID, an extra wrapper JSON object) silently disables the field. The symptom is “the trace link is gone” with no error message in the panel.
4. Query timeout below Loki’s own timeout. Grafana’s timeout: 30 against a Loki with a 60-second budget returns context deadline exceeded for any range query over a few hours. The symptom is “logs work for the last hour and fail beyond it”.
5. Wrong type. A data source provisioned as type: prometheus with a Loki URL accepts the provisioning file without error and returns garbage from the proxy. The symptom is “every Logs panel is empty and the Explore view has no Logs toggle”.
6. derivedFields references a missing UID. The internalLink.datasourceUid points at a Tempo data source that has been removed. The provisioning reloads cleanly; the link fails at click time. The symptom is a link that 404s when an engineer clicks it during an incident.

How to troubleshoot it

The diagnostic order is “is the data source provisioned?”, “is the proxy reachable?”, “is Loki responding?”, “is the derived-fields pipeline working?”.

1. Confirm the data source exists. GET /api/datasources/uid/<uid>. If 404, the provisioning file did not reload; check grafana.log for parse errors and confirm the type is loki.
2. Confirm the health check. GET /api/datasources/uid/<uid>/health. The four states are success, error, notfound, and forbidden.
3. Reproduce the request through the proxy. curl against /api/datasources/proxy/uid/<uid>/loki/api/v1/query?query={job="varlogs"}. This is the exact request Grafana issues for the simplest log query.
4. Reproduce the request directly. curl against the upstream Loki URL with the same credentials. A working direct curl that fails through the proxy isolates the problem to Grafana’s secureJsonData or tlsConfig.
5. Validate derived fields. Open the panel in Explore, click the field label on a sample line, and confirm the link opens the destination. A regex that matches in a regex tester but not in Grafana usually has a YAML escape problem.
6. Inspect Grafana’s logs. /var/log/grafana/grafana.log records every proxy call. A 400 Bad Request from Loki appears with the query string and the upstream message; a timeout appears as context deadline exceeded.

Security implications

• The proxy holds the credentials. A Loki data source with access: direct exposes the password to every browser.
• Derived-field links can leak. A derived field that extracts a session token or a customer ID and links it to another data source turns a log line into a cross-reference pivot. Audit the destination URLs; a regex that picks up a customer_email field by mistake is a data-handling bug.
• The query timeout is a DoS knob. A Loki data source without a timeout cap can be driven into a slow-query storm by a single Explore session. Set the timeout to a value above Loki’s own --query.timeout so the upstream fails first.
• maxLines and maxSeries are cost knobs. A Grafana with no caps is a Loki customer that the rest of the team cannot budget for. Cap them.

Performance implications

• Loki streams, Grafana reads. The proxy stops reading after maxLines. Loki is unaware of the cap; the cost of a high maxLines is paid in browser memory and render time, not in Loki’s CPU.
• Metric-style LogQL is bounded by maxSeries. A query that returns ten thousand series without a cap silently truncates. The fix is to add a topk or quantile to the LogQL itself, not to raise maxSeries indefinitely.
• Derived fields are per-line regex. A regex with catastrophic backtracking on a long log line is a denial of service against the browser. Test regexes with adversarial inputs.
• The query timeout is the upper bound. A Loki with a 60-second budget against a Grafana with a 30-second timeout means the Grafana times out before Loki does, and the engineer gets a clean error rather than a slow render.

Production guidance

• Pin the uid. Renaming a data source is fine; changing the UID breaks every panel and derived-field reference.
• Use access: proxy for every Loki with credentials.
• Set maxLines to 1000 by default; raise only with a documented reason.
• Set maxSeries to a value that the dashboard actually consumes. A panel that shows 100 series with maxSeries: 0 is silently asking Loki for everything.
• Configure derived fields for the trace ID and the request ID before the first incident that needs them. Add the rules in code review, not during a Sev 1.
• Reload Grafana provisioning through the GitOps pipeline.
• Periodically exercise derived-field links. A regex that has drifted is silent until an engineer clicks it.

Verification

You should now be able to answer:

• What is the role of maxLines and maxSeries in a Loki data source, and which side of the request enforces each cap?
• How does the Grafana query timeout interact with Loki’s --query.timeout flag, and which side should be smaller?
• Why does a silent regex drift in a derived-field rule only become visible during an incident?
• What does the internalLink.datasourceUid actually do at click time, and what is the failure shape when it is wrong?

Quiz