Counter Monotonicity in Practice

An on-call engineer opens a Grafana panel showing rate(queue_depth_total[5m]). The panel shows a steady value of 12 items per second. The queue is empty. The application log shows the queue has been empty for the last ten minutes. The metric was registered as a counter in the instrumentation library. The library calls inc() whenever an item is added to the queue and dec() whenever an item is removed. The counter drops when the queue empties. rate() sees the drop as a reset and extrapolates a large positive increase. The panel shows 12 items per second of “phantom” activity that does not exist.

The metric is a gauge. The library registered it as a counter. rate() does exactly what its specification says: it detects the decrease as a reset and extrapolates. The output is meaningless because the input was mislabelled.

This lesson is about the contract between the instrumentation library and the query engine. The contract is called monotonicity. Breaking it produces nonsense metrics that look correct on the wire.

What monotonicity is

A counter is monotonic when its value never decreases except on a true reset (process restart, exporter restart, or a deliberate reset of the in-memory state). The Prometheus text exposition format signals this with the line # TYPE <metric> counter.

The contract has three parts:

1. Never decrement. Application logic that subtracts from the counter (a dec() call, a count -= n statement, a “current value” assignment) violates monotonicity.
2. Reset only on legitimate process boundaries. A restart of the process hosting the counter resets it to zero. This is expected; rate() handles it.
3. No external writes. A counter that is overwritten by another component (a cron job, a configuration reload, a peer) is not monotonic.

Two adjacent concepts deserve a name:

• Counter reset detection. rate() detects a value drop and treats it as a reset, extrapolating the missing increments. This is the algorithm; it is correct for legitimate resets and wrong for non-monotonic counters.
• Reset (in CR/LF sense). The text exposition format uses LF as the line separator. A label value that contains a literal LF or CR confuses the parser. This is a different failure mode (parser corruption, not monotonicity) but the word “reset” is shared.

Why a sysadmin cares

A non-monotonic counter is the most expensive silent failure mode in a Prometheus platform. The failure is silent because Prometheus 2.55.x does not warn when rate() is applied to a gauge. The metric is on the wire with the right name, the right HELP, the right TYPE header. The library reports it as a counter. rate() consumes it. The panel shows a number. The on-call cannot tell from the panel alone that the number is nonsense.

Three operational pains appear:

• Phantom traffic. rate() of a non-monotonic counter spikes whenever the underlying value decreases. The “traffic” the panel shows is the extrapolation of the reset, not real activity.
• Phantom SLO breaches. An SLO error budget calculated against a non-monotonic counter shows a constant error rate even when no errors are happening.
• Misleading alerts. Alert rules based on rate() of a non-monotonic counter fire during legitimate state transitions (queue empties, connections close) and never fire on the real problems.

The cost of detecting this in production is hours of investigation. The cost of preventing it at instrumentation time is one line of code.

How it works

The mental model is the contract between three layers:

  Application state     Instrumentation library     Prometheus scrape
  ------------------     ------------------------     -----------------
  counter += 1      -->  inc() called            -->  metric.value += 1
                                                  -->  wire: value
                                                  -->  rate() over window

  queue_length = 0  -->  dec() called            -->  metric.value -= 1
       (queue empty)                              -->  wire: value decreases
                                                  -->  rate() detects reset
                                                  -->  panel: spike

The breakage is at the first arrow. Application state that goes up and down is being modelled as a counter. The right model for state that goes up and down is a gauge. A gauge does not pretend to be monotonic; the right PromQL function is derivative() (next lesson) or a raw plot.

For the CR/LF failure mode, the mental model is the text exposition format. The format uses LF as the line separator and # as the comment character. A label value that contains \n or \r is interpreted as a new line by the parser:

  # HELP my_metric Example
  # TYPE my_metric counter
  my_metric{label="line1
  line2"} 42

The parser sees two lines: my_metric{label="line1 and line2"} 42. The label value is "line1; the metric is malformed. The label cardinality appears to be wrong; the metric series count may not match what the exporter intended.

The fix is to escape newlines in label values. Prometheus text format accepts \n and \\ as escape sequences in label values. The instrumentation library should escape them on emission; the receiver accepts the escaped form. If the exporter emits a literal LF, the parser cannot recover.

Under the hood

How to configure it

The configuration is at three layers.

Layer 1: instrumentation library. Use the right type. In Go:

import "github.com/prometheus/client_golang/prometheus"

// Counter: monotonic. Only inc() and add() exposed.
var httpRequests = prometheus.NewCounter(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total HTTP requests served.",
    },
)

// Gauge: bidirectional. inc(), dec(), set(), sub() exposed.
var queueDepth = prometheus.NewGauge(
    prometheus.GaugeOpts{
        Name: "queue_depth",
        Help: "Current items in the queue.",
    },
)

The library enforces the contract at compile time (in Go) or at type-check time (in typed languages). In Python the contract is by convention; the type is a string.

Layer 2: Prometheus scrape config. Verify the metric type on the wire:

# READ-ONLY: confirm the metric type at the exporter
curl -sf http://my-service:8080/metrics \
  | grep -E '^# (TYPE|HELP) http_requests_total'

Expected output (illustrative):

# HELP http_requests_total Total HTTP requests served.
# TYPE http_requests_total counter

The # TYPE ... counter line is the contract. If the exporter says gauge, the metric cannot be used with rate() safely. The fix is at the application layer, not in Prometheus.

Layer 3: detection rule. A recording rule that flags non-monotonic counters:

# /etc/prometheus/rules/health.yaml
groups:
  - name: metric-health
    interval: 60s
    rules:
      - record: meta:http_requests:decreases_5m
        expr: |
          sum without (instance) (
            resets(http_requests_total[5m]) > 0
          )

The expression uses resets(), which counts the number of counter resets inside the range. A counter that legitimately resets on a deploy produces a non-zero value; a counter that never resets produces zero. A counter that resets every minute produces a steady non-zero value and is the smoking gun for non-monotonic instrumentation.

How to validate it

Three commands confirm the counter is monotonic.

Validate the type on the wire:

# READ-ONLY: confirm the metric is declared as a counter
curl -sf http://my-service:8080/metrics \
  | grep -E '^# TYPE http_requests_total'

Expected output (illustrative): # TYPE http_requests_total counter. If the line says gauge, the metric cannot be used with rate().

Validate monotonicity over a window:

# READ-ONLY: count resets in the last 5 minutes
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=resets(http_requests_total[5m])' \
  | jq '.data.result[].value[1]'

Expected output (illustrative): zero or a small integer matching the deploy cadence. A value of 100 for a 5-minute window on a service that deploys once per hour means the counter is being reset by something other than the deploy.

Validate label values for CR/LF:

# READ-ONLY: check for malformed label values in the scrape output
curl -sf http://my-service:8080/metrics \
  | grep -P '\{[^}]*\r' && echo "FOUND CR in label" || echo "no CR in labels"
curl -sf http://my-service:8080/metrics \
  | grep -P '\{[^}]*$' && echo "FOUND LF in label" || echo "no LF in labels"

Expected output: no CR in labels and no LF in labels. Any positive match indicates a label value that the parser will truncate.

Validate the rate panel against a known activity:

# READ-ONLY: rate() of the counter at a known traffic level
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=rate(http_requests_total[2m])' \
  | jq '.data.result[0].value[1]'

Expected output: a per-second rate that matches the application log. If the rate is non-zero while the application log shows zero traffic, the counter is being incremented by something other than real traffic.

How it can fail

Six failure modes, each with a recognisable symptom:

1. Gauged-as-counter. The instrumentation library reports a “current value” metric as a counter. The value drops whenever the underlying state decreases. Symptom: rate() panels show spikes during normal state transitions.
2. Counter decremented by application logic. The application subtracts from the counter on a successful retry or a queue drain. Symptom: rate() panels show a baseline value even when the application is idle.
3. Counter reset on every event. The application calls counter.set(0) at the end of every request. Symptom: the counter never accumulates above 1, and rate() is dominated by resets.
4. CR or LF in label values. A label value contains a literal newline. Symptom: the parser drops the metric or produces a partial series; cardinality audit flags the metric as having fewer series than expected.
5. Counter overwritten by external process. A cron job sets the counter to a known value at midnight. Symptom: a panel that drops to a fixed value at midnight and climbs from there.
6. Counter written from multiple processes. Two application processes share a metric by writing to the same file. The last writer wins. Symptom: the counter oscillates as each process writes its own value.

How to troubleshoot it

The diagnostic order matters. Walk it from outside in.

1. Confirm the TYPE on the wire. curl /metrics | grep TYPE. If the line says gauge, the panel cannot be fixed at the PromQL layer.
2. Plot the raw counter. A monotonic counter is a line that only goes up (with drops on legitimate resets). A line that goes up and down is non-monotonic.
3. Count the resets. resets(counter[5m]) returns the number of resets inside the window. A high value relative to the deploy cadence is the smoking gun.
4. Cross-check against application logs. If the log says zero errors and rate(errors_total[5m]) is non-zero, the counter is being incremented by something else.
5. Check for CR/LF in label values. A grep over the /metrics output for \r or newlines inside {} flags the exporter. The fix is at the instrumentation layer.
6. Inspect the instrumentation code. Look for dec(), sub(), set(), or count -= n on the counter. Any of these breaks monotonicity.

Security implications

The relevant risks live at the instrumentation layer:

• A non-monotonic counter can be used to mask a real outage. An attacker with write access to the counter can reset it on every event, hiding activity. The detection rule resets(counter[5m]) > N catches the attack at the alerting layer.
• A label value with a CR or LF can be used as a parser confusion vector. A malicious exporter that emits a label value with a literal newline can suppress other metrics by truncating them at the parser. The fix is to lock down which processes can bind to the scrape port and to validate the scrape output at the Prometheus side.
• The Prometheus API surface is unchanged. Lock the API behind authentication as usual.

Performance implications

The cost of monotonicity enforcement is at the rate() evaluation step, identical to the previous lesson. CR/LF detection has no production cost; the parser scans the line once and the cost is dominated by the number of metrics, not by the string check.

The detection rule resets(counter[5m]) is itself a rate() evaluation. One series per counter per evaluation. With hundreds of counters and a 60 s evaluation interval, the cost is small.

Production guidance

• Use the right type at the instrumentation layer. Counter for monotonic accumulation, gauge for bidirectional state.
• Never expose dec(), sub(), or set() on a counter type. If the type system allows it, the contract is broken by convention.
• Validate the # TYPE line at scrape time. A metric that changes type between versions is a breaking change.
• Run resets(counter[5m]) as a recording rule and alert on unexpected resets.
• Escape CR and LF in label values at the application boundary, not at the parser.

Verification

You should now be able to answer:

• What three conditions must hold for a counter to be monotonic?
• What does rate() output when a non-monotonic counter drops in value?
• How does a literal LF in a label value affect the parser?
• Which PromQL function detects unexpected counter resets?
• Why is the failure mode of a non-monotonic counter silent in Prometheus 2.55.x?

Quiz