Break/Fix: Counter Reset Causes Misleading Query

Reported symptoms

PaymentRetryRateHigh has paged the on-call rota eleven times in six weeks. Every page lands between 03:00 and 03:20. Every one resolves within five minutes without intervention. Nobody has ever found a matching error in the application logs, and no customer has ever complained about the windows.

The rota’s working theory is a nightly batch job. That theory has survived three attempts to disprove it, because the batch does run at 02:45 and it does finish around 03:05.

Three other observations exist, filed separately.

The panel is quiet at peak. An engineer building a capacity model noticed that the retry-rate panel is loudest in the small hours and calmest at 14:00, when the service handles four times the traffic. The observation went into a comment and no further, because the retry panel and the traffic panel live on different dashboard rows and nobody had put them side by side.

The totals disagree with the logs by two orders of magnitude. A finance query asked how many payment retries the platform performs per day. increase(payment_retries_total[24h]) answered about 1.1 million. Counting retry lines in Loki for the same day answered about 9,000. The gap was attributed to log sampling and left there.

The alert once looked right. During a genuine incident, the team scaled the worker pool up to clear a backlog. The alert fired during the scale-up. That was read as evidence that the alert works, and it hardened the rule against scrutiny for another month.

Finally, a second service that uses the same exporter library shows the same shape on its own retry metric. Two teams, two services, one library.

Evidence provided

The cheapest check is the one the team never ran, because it feels too basic to be worth the click: plot the counter itself.

$ curl -sG http://prometheus.internal.example.com:9090/api/v1/query_range \
--data-urlencode 'query=payment_retries_total{service="payments"}' \
--data-urlencode 'start=2026-08-14T03:00:00Z' \
--data-urlencode 'end=2026-08-14T03:04:00Z' \
--data-urlencode 'step=30s' \
| jq -r '.data.result[0].values[] | .[1]'

128
341
502
17
164
390
41
92

A monotonic counter rises and resets only when its process restarts. This one falls repeatedly, in the middle of a four-minute window. Two queries decide whether those falls are restarts.

$ curl -sG http://prometheus.internal.example.com:9090/api/v1/query \
--data-urlencode 'query=resets(payment_retries_total[5m])' \
| jq -r '.data.result[] | .metric.instance + "  " + .value[1]'

payments-01.internal.example.com:8080  6
payments-02.internal.example.com:8080  7
payments-03.internal.example.com:8080  5

$ curl -sG http://prometheus.internal.example.com:9090/api/v1/query \
--data-urlencode 'query=changes(process_start_time_seconds{service="payments"}[7d])' \
| jq -r '.data.result[] | .metric.instance + "  " + .value[1]'

payments-01.internal.example.com:8080  0
payments-02.internal.example.com:8080  0
payments-03.internal.example.com:8080  0

The exporter, meanwhile, is asserting that this is a counter.

$ curl -sf http://payments-01.internal.example.com:8080/metrics | grep -A2 '^# HELP payment_retries_total'

# HELP payment_retries_total Retries currently pending for this service.
# TYPE payment_retries_total counter
payment_retries_total{service="payments"} 341

Work the evidence before reading on

The metric is declared a counter, it is named _total, and it falls six times in five minutes on a process that has not restarted in a week.

1. resets() reports six resets in five minutes. changes(process_start_time_seconds[7d]) reports zero restarts. Both queries are correct. What does that force you to conclude about the series?
2. Read the HELP text on the exporter output. What quantity is this metric reporting, and is that quantity capable of only increasing?
3. rate() treats a fall as a reset and adds the lost increments back into the total. Given a series that rises while work queues up and falls while workers clear it, what is rate() of that series actually measuring?
4. The alert fires at 03:00 when the nightly batch stops, and stays quiet at 14:00 under four times the traffic. Reconcile that with your answer to question 3.

Before continuing: name the PromQL change that would make this panel correct. If your answer is that there isn’t one, say why not.

Root cause

1. The TYPE header is a claim, not a check

Prometheus reads # TYPE payment_retries_total counter from the exposition text and records it as metadata. Nothing verifies it. The storage layer accepts a sample smaller than the last one without complaint, and Prometheus 2.55.x issues no warning when rate() is applied to a series that is not a counter.

So the entire contract behind rate() - that the input only ever increases, except when a process restarts - is held up by the instrumentation alone. Here the instrumentation is wrong, and every layer downstream of it behaves impeccably on bad input.

2. The series is current state, published as accumulation

The exporter reads the current retry backlog from the broker on each scrape and hands it to the client library as a counter value. That value goes up while work arrives and down while workers clear it. It is a gauge in every respect except the header.

The HELP text says so, in plain English, on every scrape: “Retries currently pending for this service.” The word currently is the whole incident. It was written by the person who introduced the bug, and it has been served a few times a minute ever since.

3. rate() is measuring the drains

This is the part worth working through, because it explains all four symptoms at once.

When rate() walks the samples in its window and finds one lower than the last, it treats the fall as a reset and adds the increments it believes were lost back into the total - for each drop, the value the series held immediately before it. On a genuine counter that is exactly right. On this series it means every drain is counted as though the backlog had been retried away from scratch.

The 502 in the fourth row is not retries. It is the size of the backlog that just drained. The panel therefore reports how fast the queue is emptying, in units that read like retries per second, and every symptom follows:

• It is loudest at 03:00, when the batch stops feeding the queue and workers finally catch up. That is the biggest drain of the day, so it is the biggest number of the day.
• It is quietest at the 14:00 peak, when the queue is filling faster than it drains and there are fewer falls to count.
• increase() over a day is enormous, because the same backlog is counted again on every drain - 120 times the log count, and that ratio is roughly the number of drains per day.
• It fired during the real incident because the team scaled workers up and drained a large backlog fast, which is precisely the thing this panel measures. The alert looked correct for the one reason nobody could see.

4. Nothing was in place to catch it

The metric satisfied every convention the team enforced. The _total suffix matched the naming policy. The TYPE header matched the suffix. It appeared on a dashboard, it had an alert, it fed the SLO. There was no check anywhere that asked whether the series behaved like the thing it claimed to be, and the one piece of contradicting evidence - the 120-fold gap against the logs - was explained away as sampling.

Resolution

1. Stop the harm before fixing the cause. Silence the alert with a named owner and an expiry date, and post in the channel what the panel has actually been measuring. People have been making capacity and reliability decisions from this number for months and are entitled to know.
2. Adopt an interim signal that is independently produced. The log-derived retry count is roughly right and does not share the failure, so it can carry the alert while the instrumentation is repaired.
3. Read the exporter and confirm the mechanism rather than inferring it. A custom collector passing prometheus.CounterValue for a reading taken at scrape time is the shape to look for; the same bug in the same library is why a second service shows the same pattern.
4. Publish the backlog as a gauge under a name that describes a level, not a total. This is the metric the panel should have been plotting all along.
5. If a genuine per-second retry rate is wanted - and it probably is, since that was the question the alert was written to answer - it must be accumulated at the source: a real counter incremented once per retry attempt, which no scrape-time reading can substitute for.
6. Run old and new names in parallel for a defined window rather than redefining an existing name in place. Migrate the dashboards, recording rules and alerts, then retire the old series and record the retirement date, because historical panels end there rather than continuing.
7. Recompute the SLO for the affected window from the log-derived count and correct the error-budget record. The reported burn did not happen, and leaving it uncorrected means every future budget decision inherits the error.
8. Add the monotonicity guard below in the same change, so the next exporter written this way is caught by the platform rather than by an on-call engineer.

The rule that closes the class rather than the instance:

# /etc/prometheus/rules/metric-health.yaml
groups:
  - name: metric-health
    interval: 60s
    rules:
      # A counter may only reset when its process restarts.
      # More resets than restarts means the series is not monotonic.
      - alert: CounterNotMonotonic
        expr: |
          resets(payment_retries_total[1h])
            > on (instance, job) changes(process_start_time_seconds[1h])
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: 'Series on {{ $labels.instance }} resets more often than the process restarts'
          description: |
            rate() and increase() over this series are extrapolating across
            falls that are not restarts. Treat every panel and alert built on
            it as unreliable until the instrumentation is corrected.

Verification

1. The new counter is monotonic. resets(new_counter[1h]) is zero over any window in which changes(process_start_time_seconds[1h]) is also zero. Comparing the two is the check; either one alone proves nothing.
2. The raw plot only rises. Plot the new counter directly and confirm it climbs and never falls except at a restart you can point to.
3. The total agrees with an independent source. increase(new_counter[24h]) lands within a few per cent of the log-derived count for the same day. This is the comparison that exposed the original error, and it is the one that confirms the fix.
4. The gauge matches the broker. The new backlog gauge tracks the broker admin view of pending retries, which is the quantity it claims to report.
5. The 03:00 window is quiet. Watch a full nightly cycle: the queue drains, and nothing pages.
6. The rebuilt alert can still fire. In staging, inject a genuine retry storm and confirm it fires and routes correctly. An alert that stopped firing has not been shown to work - only to be silent.
7. The guard can fail. In staging, publish a deliberately decreasing series and confirm CounterNotMonotonic fires. A rule that has only ever been green has never been tested.

Prevention

• Alert on monotonicity fleet-wide. Comparing resets(metric[1h]) against changes(process_start_time_seconds[1h]) catches the whole class: the gauge published as a counter, the counter decremented by application logic, the counter overwritten by an external process. One recording rule, applied broadly.
• Treat the TYPE header as a claim. Nothing between the exporter and the panel verifies it, and the query engine will not warn you. It is an assertion by the least-reviewed line of code in the pipeline.
• Hold the naming line. _total belongs to a value that only accumulates. Anything read from current state at scrape time is a gauge, whatever value type the collector passes.
• Review custom collectors specifically. The typed counter API makes the mistake impossible; a custom collector makes it a one-word choice. That is where to spend review attention.
• Reconcile every new alert against an independent signal before it pages. One comparison of increase() against the log-derived count, done once when the rule was written, would have ended this before the first page.
• Take the two-orders-of-magnitude disagreement seriously. The gap was found, explained away as sampling, and left. A factor of 120 is not a sampling artefact, and treating a large unexplained discrepancy as noise is how a known symptom becomes a long incident.