Why Counters Aren't Plotted Directly'

A new dashboard lands in production. It plots http_requests_total directly. The first panel shows a line that climbs from 0 to 14 million requests over the last 30 days. The y-axis is dominated by the absolute count. The post-deploy drop on day 14 is invisible because it is small relative to the absolute scale. The reader cannot tell whether the day 14 deployment changed request volume at all. Two days of traffic cannot be compared: the absolute numbers differ by tens of millions.

Plotting a counter directly is the single most common dashboard mistake in a Prometheus platform. This lesson is about why, and about the three rules that replace the raw plot with something the reader can use.

What it is

A counter is a monotonically increasing numeric metric. Its absolute value at any moment is the cumulative total since the process started. A plot of a counter is a plot of a cumulative total. That shape has three properties that make it unusable for operational insight:

1. The line rises without bound. Over a long time range the y-axis is dominated by the cumulative total, not by the activity of interest. A 1% change in traffic produces a 1% change in the slope; on a 30-day plot that change is invisible.
2. Resets appear as vertical drops. A process restart resets the counter to zero. On a cumulative plot the restart looks like a sudden outage or a sudden traffic drop, depending on which direction the line is read.
3. Two time periods cannot be compared. Yesterday the counter started at 13.8 million. Today it starts at 14.2 million. The absolute values are different even if traffic is identical.

The replacement is rate(counter[window]). A rate plot shows activity per second. Two days can be compared because both are measured in per-second units. Resets are absorbed by the extrapolation the previous lesson described.

Why a sysadmin cares

Three operational questions become unanswerable on a raw counter plot:

• “Is the service getting busier?” Invisible on a 30-day counter plot. Trivially visible on a rate plot.
• “Did the deploy increase traffic?” Impossible to read off a counter plot because the post-deploy counter starts from a different baseline. Visible on a rate plot because the rate is the same units before and after.
• “What was the traffic shape during the incident?” Lost in the absolute scale of a counter plot. Recovered on a rate plot.

A wall of green panels that plot counters directly is a common production failure shape. The team that built it was satisfying the metric-emission contract. The team that investigates with it cannot answer any of these questions.

How it works

The mental model is two conversions:

  Raw counter value              Useful operational signal
  ------------------             ------------------------
  monotonic absolute count  -->  per-second rate
  scale dominated by total  -->  scale dominated by activity
  drops on restart          -->  continuous through resets
  incomparable time ranges  -->  comparable on rate axis

The conversion is rate(). The window is the rate window. The panel renders the rate at every step in the dashboard.

Three rules govern a useful rate panel:

1. Use rate(), never the raw counter. The first rule is absolute. A counter on a panel is almost always wrong.
2. Slice by labels. A counter that is not labelled is useless. sum without (instance)(rate(http_requests_total[1m])) is the canonical cluster-wide expression. The instance label is removed; the status label is kept; the panel shows one line per HTTP status class.
3. Pick a window that respects the scrape interval. The window must contain enough samples to be stable. The rule of thumb is 2x to 4x the scrape interval.

The scrape-interval rule deserves a closer look. Default Prometheus scrape interval is 15 s. A rate window of [15 s] contains at most two samples; a counter reset between them is invisible. A rate window of [1 m] contains four samples and survives a single missed scrape. A rate window of [5 m] contains twenty samples and absorbs a one-minute scrape failure.

  scrape_interval = 15s

  window      samples    stable?    use for
  --------    -------    --------   -----------------------
  [15s]       1 to 2     no         do not use
  [1m]        4          yes        ops dashboards
  [5m]        20         yes        SLO error budgets
  [15m]       60         yes        capacity trends

The “use for” column is industry convention, not PromQL law. [1 m] is responsive and noisy. [5 m] is stable and slightly slow. [15 m] is stable and slow. Each has its place.

Under the hood

How to configure it

The configuration is at the dashboard and recording-rule level. The exporter is already producing the counter. The PromQL expression is the configuration.

Grafana panel query. The panel that replaces the raw counter plot:

sum by (status) (rate(http_requests_total[1m]))

The expression:

1. Slices by the status label. One line per HTTP status class (200, 3xx, 4xx, 5xx).
2. Aggregates across instances and jobs by sum. The cluster total is preserved.
3. Wraps the counter in rate(). The y-axis is per-second requests.

Recording rule. The production-grade version of the same expression, evaluated once and shared across panels:

# /etc/prometheus/rules/http-rate.yaml
groups:
  - name: http-rate
    interval: 30s
    rules:
      - record: job:http_requests:rate1m_by_status
        expr: |
          sum by (job, status) (
            rate(http_requests_total[1m])
          )
      - record: job:http_requests:rate5m_by_status
        expr: |
          sum by (job, status) (
            rate(http_requests_total[5m])
          )

The two recording rules give the dashboards a choice: [1 m] for ops panels, [5 m] for SLO panels. Both rules avoid plotting the raw counter.

Dashboard panel settings. The Grafana panel for a 1-hour ops view:

  Panel type        : Time series
  Data source       : Prometheus
  Query             : sum by (status) (rate(http_requests_total[1m]))
  Min step          : 15s   # match the scrape interval
  Legend            : show, by status
  Unit              : reqps (requests per second)

The min step matches the scrape interval. The unit override labels the axis correctly. The legend shows one entry per status class. No raw counter appears anywhere.

How to validate it

Three commands confirm the dashboard expression returns what the reader expects.

Confirm the expression slices by status:

# READ-ONLY: list the series returned by the panel query
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=sum by (status) (rate(http_requests_total[1m]))' \
  | jq '.data.result[].metric.status'

Expected output (illustrative):

"200"
"301"
"404"
"500"

One line per status class. If the result is a single series with no status label, the by clause was dropped or the counter is not labelled.

Confirm the units are per second:

# READ-ONLY: verify the y-axis is per-second, not cumulative
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=rate(http_requests_total[1m])' \
  | jq '.data.result[0].value[1]'

Expected output (illustrative):

"412.8"

A float representing requests per second. If the value is millions, the rate() wrapper is missing.

Compare 1m and 5m windows side by side:

# READ-ONLY: rate window sensitivity check
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(http_requests_total[1m]))' \
  | jq '.data.result[0].value[1]'
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(http_requests_total[5m]))' \
  | jq '.data.result[0].value[1]'

Expected output (illustrative): two values within a few percent of each other. If the [5m] value is wildly different, the traffic shape is genuinely changing or the counter has recently reset and the [1m] window is dominated by the post-reset climb.

How it can fail

Six failure modes, each with a recognisable symptom:

1. Raw counter on a long time range. The y-axis is dominated by the absolute total. The reader cannot see incident-level detail. Symptom: a dashboard with one monotonic line climbing to the top of the panel.
2. Rate window shorter than 2x scrape interval. Too few samples; reset detection is unreliable; the panel spikes on single samples. Symptom: single-sample spikes that vanish when the window is widened to [1 m].
3. Rate window much longer than necessary. The panel smooths over real incidents. A 30 s outage is invisible. Symptom: alerts that fire after the user-visible degradation has ended.
4. Counter not sliced by labels. rate(http_requests_total[1m]) with no by or without aggregation returns one series per instance per job. A cluster of 50 instances produces 50 lines. Symptom: a panel with hundreds of lines, none of which are individually meaningful.
5. Sum across incompatible labels. sum(rate(http_requests_total[1m])) sums across all labels including the status label. The result is a single number that hides the 5xx spike inside the 200 line. Symptom: a panel that is flat while the 5xx error rate is spiking.
6. Min step larger than rate window. The graph over-samples and the panel renders the same value at every step. Symptom: a panel that looks like a stepped histogram instead of a smooth line.

How to troubleshoot it

The diagnostic order matters. Walk it from outside in.

1. Read the query. If the expression is the raw counter name, the panel is wrong. Wrap it in rate().
2. Check the by clause. If the result has too many series or too few, the by clause is wrong. The right clause is the label that gives the reader actionable slices (status, route, instance).
3. Compare the rate() value to the raw delta over the same window. If they differ by more than the extrapolation tolerance, the counter is non-monotonic.
4. Compare [1 m] and [5 m]. If the two are wildly different, the counter has a recent reset or the scrape interval is much longer than expected.
5. Confirm the min step. Min step should be the smallest interval the reader can interpret; rate window should be 2x to 4x the scrape interval.
6. Cross-check against the application logs. A 5xx spike in the application log should produce a 5xx spike in the rate() panel at the same time. If the spike is missing, the label selector is wrong.

Security implications

The relevant risks live at the dashboard query boundary:

• The rate() expression exposes per-second business volume. A user with dashboard access can reconstruct business activity from the panel. Lock Grafana behind SSO.
• A cardinality-bombing rate() expression (rate({__name__=~".+"}[5m])) is a denial-of-service vector. Apply --query.max-concurrency in Prometheus.
• Recording rules that wrap rate() run on every Prometheus reload. A typo in the recording rule produces thousands of empty series per evaluation. Validate with promtool check rules.

Performance implications

The cost is dominated by the in-memory range vector at query time. A 30-day panel with rate() over [5 m] and 15 s scrapes keeps ~172 800 samples per series in memory for the expression evaluation. The cost is paid by the Prometheus server, not by Grafana.

The levers:

• Window length. Doubling the window doubles the in-memory range vector.
• Step length. Halving the step doubles the number of evaluations but halves the samples per evaluation. Net effect on query CPU: roughly constant.
• Cardinality. A sum by (status) clause collapses per-instance series to one per status. Cardinality drops by an order of magnitude on a typical fleet.

Production guidance

• Always wrap a counter in rate() before plotting. There is no production case for a raw counter on a dashboard.
• Slice by labels. The label that gives the reader actionable slices is the right by clause.
• Use rate() window of [1 m] for ops dashboards, [5 m] for SLO error budgets, [15 m] for capacity trends.
• Match the min step to the smallest interval the reader can interpret; do not set it lower.
• Convert every raw-counter panel in the legacy dashboard to a rate() panel during the next dashboard hygiene pass.

Verification

You should now be able to answer:

• Why does a raw counter plot fail on a 30-day time range?
• What does the 2x to 4x rule recommend for the rate window?
• What is the relationship between Grafana min step and the rate window?
• How would you slice http_requests_total to expose the 5xx error rate separately from the 200 success rate?
• Why is the unit of a rate() expression always per second?

Quiz