derivative() for Gauges

A dashboard panel shows node_memory_MemAvailable_bytes. The line trends slowly downward over the last 12 hours; the host is using more memory. The on-call needs to know whether this is a leak (memory grows unbounded) or a workload pattern (memory grows during business hours and shrinks at night). The raw gauge plot does not answer the question. The on-call needs the rate of change, not the absolute value.

The right function is derivative(). It returns the per-second slope of the gauge across the window. A positive derivative means the gauge is climbing. A negative derivative means it is falling. A derivative of zero means it is stable. The output is the same shape as rate() but without the counter-reset detection.

A different question arrives the same morning: “will the disk fill in the next four hours?”. The right function is predict_linear(). It fits a linear regression to the recent samples and extrapolates forward. The output is the predicted value at a future timestamp. It is the right tool for short-horizon capacity forecasting.

This lesson covers both functions, their limits, and their production use cases.

What derivative() and predict_linear() are

derivative(v[range_vector]) returns the per-second slope of a gauge across the window. The formula is:

derivative  =  (v[n] - v[0]) / (t[n] - t[0])

This is the average slope between the first and last samples in the range. It is the same shape as rate() but without the counter-reset detection. derivative() makes no assumption about monotonicity; a gauge that decreases is fine.

predict_linear(v[range_vector], duration, factor) fits a simple linear regression to the samples in the range vector and returns the predicted value at now + duration. The optional factor argument scales the prediction interval (default 1, which gives one standard deviation of uncertainty).

predict_linear  =  v[0] + slope * duration
                  plus or minus factor * std_error

The output is the gauge’s own units at the future timestamp. The function is not a counter function; it does not detect resets. The regression assumes the recent trend continues.

Three-way contrast:

  Function          Input           Output             Use for
  --------          -----           ------             ------
  rate()            counter         per-second rate    counters
  derivative()      gauge           per-second slope   gauges
  predict_linear()  gauge or counter  predicted value  forecasting

Why a sysadmin cares

Two operational questions become answerable once these functions are understood:

• “Is the metric growing, shrinking, or stable?” answered by derivative(). The panel shows the slope, not the value. A leak shows up as a positive derivative that does not return to zero. A workload pattern shows up as a derivative that oscillates around zero.
• “When will capacity run out?” answered by predict_linear(). The output is the predicted value at a future timestamp. The on-call can read the prediction against the threshold and decide.

Without these functions, the on-call falls back to eyeballing the gauge plot. Eyeballing works for clear trends; it fails for slow leaks and oscillating workloads.

How derivative() works

The mental model is the slope of a line. For samples (v[0], t[0]) and (v[n], t[n]):

derivative  =  (v[n] - v[0]) / (t[n] - t[0])

This is the average slope across the window. It is not the instantaneous slope at the last sample. For an instantaneous slope, use a short window (a few scrape intervals).

The consequences:

1. No reset detection. A gauge that drops and climbs produces a derivative that may be positive, negative, or zero. The function does not care about the shape inside the window; only the endpoints.
2. Same formula as rate() minus the reset handling. If you apply derivative() to a counter, you get almost the same result as rate() with one exception: derivative() does not extrapolate over resets. A counter that resets inside the derivative() window produces a noisy output.
3. Window choice dominates accuracy. A 1-minute window on a gauge that changes once per minute averages many zero-slope samples with one large slope. A 5-minute window is more stable.

How predict_linear() works

The mental model is a regression line. Prometheus fits a simple linear regression to the samples in the range vector and returns the predicted value at now + duration:

predict_linear  =  v_regression(t_now + duration)

The regression uses ordinary least squares. The factor argument scales the prediction interval:

predict_linear with factor=1 returns:
  predicted value plus or minus one standard error
predict_linear with factor=0 returns:
  predicted value only (no uncertainty)

The function is a forecaster, not a classifier. It does not know whether the metric is “good” or “bad”; it only knows the trend. The on-call interprets the prediction against a threshold.

The consequences:

1. Linear assumption. The forecast assumes the trend continues linearly. A workload pattern (memory grows by day, shrinks at night) produces a forecast that misses the cyclic component.
2. No state knowledge. predict_linear does not know the metric is “disk free” or “queue depth”. The on-call must know the semantics to interpret the prediction.
3. Uncertainty grows with duration. A 1-hour forecast on a stable metric is reliable. A 1-day forecast on a noisy metric is unreliable. The prediction interval (factor * std_error) widens with duration.

Under the hood

How to configure it

Both functions are used inline in PromQL or inside recording rules. There is no daemon-level configuration.

Recording rule for a memory leak detector. The expression detects a sustained positive slope on a memory gauge:

# /etc/prometheus/rules/host.yaml
groups:
  - name: host-memory
    interval: 60s
    rules:
      - record: host:node_memory_MemAvailable:derivative1h
        expr: |
          derivative(node_memory_MemAvailable_bytes[1h])

A positive value means memory is shrinking (the gauge is going down). A sustained positive value over several hours is the signature of a memory leak.

Recording rule for a disk-fill forecaster. The expression predicts the value of node_filesystem_files_free 4 hours from now:

# /etc/prometheus/rules/disk.yaml
groups:
  - name: disk-forecast
    interval: 5m
    rules:
      - record: host:node_filesystem_files_free:predict_4h
        expr: |
          predict_linear(node_filesystem_files_free[6h], 4 * 3600)

The window is 6 hours; the prediction horizon is 4 hours. The output is the predicted value 4 hours from now. The on-call reads the prediction against the threshold for “disk full”.

Recording rule for a queue depth trend. The expression detects a growing queue:

# /etc/prometheus/rules/queue.yaml
groups:
  - name: queue-depth
    interval: 30s
    rules:
      - record: job:queue_depth:derivative5m
        expr: |
          derivative(queue_depth[5m])

A positive value means the queue is growing. A sustained positive value over 5 to 15 minutes is the signature of upstream slowness.

How to validate it

Three commands confirm the two functions are producing the expected outputs.

Validate derivative() returns the slope:

# READ-ONLY: confirm derivative() reports a positive slope on a growing gauge
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=derivative(node_memory_MemAvailable_bytes[1h])' \
  | jq '.data.result[0].value[1]'

Expected output (illustrative): a negative float if memory is shrinking. A positive value if memory is growing. Zero if memory is stable.

Validate predict_linear() returns a forecast:

# READ-ONLY: confirm predict_linear() returns a predicted value
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=predict_linear(node_filesystem_files_free[6h], 4 * 3600)' \
  | jq '.data.result[0].value[1]'

Expected output (illustrative): a float representing the predicted bytes free 4 hours from now. If the value is wildly negative or wildly larger than the current value, the regression is fitting noise.

Validate the forecast against a known event:

# READ-ONLY: compare the forecast to the actual value 4 hours later
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=node_filesystem_files_free' \
  --data-urlencode 'time=2026-08-13T14:00:00Z' \
  | jq '.data.result[0].value[1]'
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=predict_linear(node_filesystem_files_free[6h], 4 * 3600)' \
  --data-urlencode 'time=2026-08-13T10:00:00Z' \
  | jq '.data.result[0].value[1]'

Expected output (illustrative): the two values are within 10 to 20 percent of each other if the trend is stable. A large discrepancy indicates the trend changed, the disk was cleaned up, or the workload pattern is cyclic.

How it can fail

Six failure modes, each with a recognisable symptom:

1. derivative() on a counter. A counter that resets produces a derivative that includes the drop. Symptom: a panel that spikes negative whenever a counter resets.
2. predict_linear() over a long horizon on a noisy gauge. The regression fits noise; the forecast is unreliable. Symptom: a forecast that jumps wildly between evaluations.
3. predict_linear() on a cyclic gauge. The regression fits the recent samples; the forecast misses the cyclic component. Symptom: a forecast that is wildly wrong 12 hours later when the cycle repeats.
4. derivative() on a gauge with sparse samples. A gauge that updates once per minute produces a derivative that is zero most of the time and spikes on each update. Symptom: a sawtooth pattern on the derivative panel.
5. predict_linear() on a counter with resets. The regression ignores the resets; the forecast may be wildly high or wildly low. Symptom: a forecast that does not match the actual counter value at the future time.
6. Threshold against the wrong direction. A “disk full” alert fires when predict_linear(disk_free, 4h) falls below a threshold. If the threshold is set against the wrong direction (the gauge climbs toward the threshold rather than falls away from it), the alert never fires. Symptom: an alert rule that is never triggered despite the disk approaching full.

How to troubleshoot it

The diagnostic order matters. Walk it from outside in.

1. Identify the question. “Is the metric changing?” or “where will the metric be?”. The question dictates the function.
2. Confirm the metric type. curl /metrics | grep TYPE. derivative() and predict_linear() work on gauges; rate() works on counters. Using the wrong function on the wrong type produces nonsense.
3. Plot the raw gauge. A gauge plot shows the value; a derivative plot shows the slope. The on-call needs to decide which question matters.
4. Validate the forecast against history. Compare a past prediction to the actual value at the prediction time. A large discrepancy indicates the regression is fitting noise or the trend changed.
5. Check the prediction interval. The factor argument controls the prediction interval. A factor of 0 returns the point estimate only; a factor of 1 returns the estimate plus or minus one standard error.
6. Cross-check against the underlying workload. A memory leak forecast should match the application’s known leak rate. A disk-fill forecast should match the workload’s known write rate. If they disagree, the forecast is wrong.

Security implications

The two functions are query-time. The attack surface is the Prometheus API:

• predict_linear() over a high-cardinality gauge is a denial-of-service vector. The function is cheap per series but expensive in aggregate. Apply --query.max-concurrency.
• derivative() exposes the rate of change of business metrics. A user with query access can infer activity from the slope of a queue gauge. Lock the API behind authentication.
• Recording rules with predict_linear() run on every Prometheus reload. A typo in the rule produces thousands of empty series per evaluation. Validate with promtool check rules.

Performance implications

The cost is dominated by the in-memory range vector at query time:

• Window length. Doubling the window doubles the in-memory range vector. A 6-hour window on a 15 s scrape keeps ~1 440 samples per series in memory.
• Step length. Halving the step doubles the number of evaluations. Net effect on memory: constant. Net effect on render time: doubled.
• Cardinality. A sum by (job) clause collapses per-instance series. Cardinality drops by an order of magnitude on a typical fleet.

predict_linear() is more expensive than derivative() because it computes a regression over the entire range vector. For a 6-hour window, the cost is O(1 440) per series per evaluation. With hundreds of series and a 5-minute evaluation interval, the cost is small.

Production guidance

• Use derivative() on gauges that change. Use predict_linear() for short-horizon forecasting. Use rate() on counters.
• Document the window choice in the recording rule. The window is the lever that determines the answer.
• Validate predict_linear() against history before trusting the forecast. A regression that fits the recent samples may not fit the future samples.
• Avoid predict_linear() over horizons longer than the window. A 24-hour forecast on a 1-hour window is unreliable.
• Use the prediction interval (the factor argument) to communicate uncertainty. A point estimate hides the uncertainty; a prediction interval shows it.

Verification

You should now be able to answer:

• What does derivative() return for a gauge that is decreasing over the window?
• What is the difference between derivative() and rate()?
• What assumption does predict_linear() make about the trend?
• Why is predict_linear() unsuitable for cyclic gauges?
• How would you use predict_linear() to forecast disk fill in 4 hours?

Quiz