Timestamp Issues

An alert fires: “loki_discarded_samples_total has increased by 12,400 in 5 minutes.” Theon-call engineer opens Grafana and runs the matching query. The query is empty. The investigation proceeds. The first thing the on-call engineer learns is that the discarded sample was rejected because the timestamp was outside the per-tenant window. The push succeeded. The timestamp was wrong.

A timestamp problem is invisible at the HTTP layer. The push returned 204. The distributor accepted the entry. The ingester later rejected the entry because the timestamp was older than the configured rejection_older_than. The wrong-time symptom is silent in the entry path and loud in the query path.

What it is

A log line carries two timestamps from the moment it is written to the moment it appears in Loki:

• Source timestamp. The wall-clock time the application stamped on the line, or the wall-clock time the kernel recorded when the line was written. For RFC3339 sources, this is the value in the line text. For syslog sources, this is the value the syslog parser extracts.
• Ingestion timestamp. The wall-clock time the line was accepted by the Loki distributor. This is recorded by the distributor and set on the push request as the nbf (not before) bound.

Loki stores the line against the source timestamp when the parser succeeds; otherwise it stores the line against the ingestion timestamp. The query path reads the stored timestamp. A wrong source timestamp therefore affects every downstream query: the line appears at the wrong time, in the wrong time range, and against the wrong retention boundary.

Loki 3.x accepts timestamps in RFC3339 with up to nanosecond precision. Unix epoch in seconds, milliseconds, or microseconds is also accepted. Local time without an explicit offset is the most common source of a wrong timestamp.

Why a sysadmin cares

A wrong timestamp is a wrong answer to three operational questions:

• Was the bug at 14:00 or 13:00? The investigator pivots on the chart at the wrong time. The hypothesis is wrong. The fix is delayed.
• Is the entry within retention? A timestamp one hour in the past can fall outside the per-tenant rejection window. The entry is silently dropped.
• Did the entries arrive in order? The boltdb-shipper rejects out-of-order entries. A clock that runs backwards writes a second entry that is older than the first; the second entry is rejected at the ingester.

The alert that fires on the metric is the only signal the operator has. The push is silent. The query is silent. The metric is the signal.

How it works

Loki stamps each entry with the source timestamp extracted from the line. The pipeline is:

   Source line           Agent pipeline          Loki distributor
+-----------------+   +-------------------+   +-------------------+
| 14:00:00 INFO  |   | parse -> map      |   | accept or reject  |
| 14:00:01 ERROR | -> |  -> extract ts    | -> |  -> stamp entry   |
| 14:00:02 INFO  |   |  -> rewrite       |   |  -> store         |
+-----------------+   +-------------------+   +-------------------+

Two boundaries fail at the wrong time:

• The agent → distributor boundary. The agent parses the source timestamp. If the parser fails, the agent sends the entry with no timestamp. The distributor stamps the entry with the ingestion time.
• The distributor → ingester boundary. The distributor forwards the entry to the ingester. The ingester accepts entries whose timestamp is newer than the stream’s last timestamp and within the configured rejection_older_than window. Otherwise the entry is rejected with a rejected_older_than or out_of_order reason.

How to configure it

The collector configuration owns the timestamp. The pipeline stages must extract the source timestamp and reject lines that fail to parse. The minimal River configuration for RFC3339 source logs:

// /etc/alloy/config.alloy
loki.source.file "payments" {
  targets    = local.file_match("/var/log/payments/*.log")
  forward_to = [loki.process.payments.receiver]
  labels     = { job = "payments", instance = sys.env("HOSTNAME") }
}

loki.process "payments" {
  // Reject lines that do not match the expected format. The default
  // is to forward the raw line; production should reject so the
  // failed-parse is visible in metrics.
  forward_to = [loki.write.default.receiver]

  stage.match {
    selector = "{job=\"payments\"}"
    stage.regex {
      expression = "^(?P<ts>\\S+)\\s+(?P<level>\\S+)\\s+(?P<msg>.*)$"
    }
    stage.timestamp {
      source          = "ts"
      format          = "RFC3339Nano"
      fallback_format = "RFC3339"
      action_on_error = "fwd"
    }
  }
}

loki.write "default" {
  endpoint {
    url = "http://loki-write.monitoring.svc:3100/loki/api/v1/push"
  }
}

The three settings that change the failure mode are stage.timestamp.format (the expected format), fallback_format (an alternative to try when the first fails), and action_on_error (what to do on a parse failure: fwd forwards the line as raw text, drop drops the line, no_match skips stages).

The Loki tenant-side limit controls how far in the past an entry can be accepted:

# /etc/loki/loki.yml
limits_config:
  # Reject samples with timestamps older than 1 hour.
  rejection_older_than: 1h
  # Per-stream rate limit.
  per_stream_rate_limit: 8MB

The retention window is the operational boundary. A source clock that runs behind real time by more than an hour will write entries that fall outside the window.

How to validate it

The validation reads three surfaces: the agent’s parse log, the distributor’s discarded-samples metric, and a representative query.

READ-ONLY: confirm the agent is parsing the timestamp.

sudo journalctl -u alloy -n 50 --no-pager | grep -i 'timestamp'

Expected output:

level=info msg="stage timestamp" path=/var/log/payments/payments.log
level=warn msg="failed to parse timestamp" path=/var/log/payments/payments.log

A failed to parse timestamp line means the agent saw a line whose format was not the expected RFC3339. Count the lines; a small number is normal (an old log file before the format change); a large number is the source of the wrong-time problem.

READ-ONLY: confirm the distributor is rejecting entries for timestamp reasons.

curl -s http://loki-distributor.monitoring.svc:3100/metrics \
  | grep -E 'loki_discarded_samples_total.*(older_than|out_of_order)'

Expected output:

loki_discarded_samples_total{reason="older_than",tenant="1"} 0
loki_discarded_samples_total{reason="out_of_order",tenant="1"} 0

A non-zero value here is the smoking gun. The reason label identifies the failure shape: older_than is a clock behind real time; out_of_order is a clock that ran backwards or a single delayed batch.

READ-ONLY: confirm the stored entries are within the expected time.

logcli -addr http://loki-query.monitoring.svc:3100 query \
  --since=1h '{job="payments"} | json | line_format "{{.ts}} {{.msg}}"'

Expected output:

2026-08-14T03:00:01.412Z payment failed for order 8812
2026-08-14T03:00:02.001Z payment failed for order 8813
2026-08-14T03:00:02.501Z payment failed for order 8814

The timestamps should be within the last hour. A line whose timestamp is more than an hour in the past is the wrong-time symptom; the entry should have been rejected by the agent.

READ-ONLY: confirm the ingester is accepting entries.

curl -s http://loki-ingester.monitoring.svc:3100/metrics \
  | grep -E 'loki_ingester_streams_created_total'

loki_ingester_streams_created_total should be increasing. A flat line plus a non-zero out_of_order counter means the ingester is reading the entries but rejecting them.

How it can fail

Four specific failure shapes appear in production.

1. The source clock is wrong. The host’s date is behind real time by one hour. The agent stamps the line with the host’s clock. The distributor accepts the entry; the ingester rejects it as older_than. The symptom is a non-zero loki_discarded_samples_total{reason="older_than"}. First hop: timedatectl status on the source host.

2. The format string is wrong. The line is in RFC3339 but the agent is configured to parse Unix epoch. The agent fails to parse; the agent forwards the line with the ingestion timestamp. The line is stored at the ingestion time, not the source time. First hop: the agent’s loki.process stage configuration.

3. The ingestion rate is normal, but the entry is rejected as out-of-order. The host’s clock ran backwards (NTP step, VM resume, container clock skew). The agent sends an entry with a timestamp older than the previous entry in the stream. The ingester rejects it as out_of_order. First hop: the source host’s clock; the loki_ingester_received_entries rate.

4. The timezone is implicit. The line is in local time without an offset. The agent parses the wall-clock value but cannot resolve it to UTC. The line is stored at the local-clock interpretation. A team in Europe/Berlin and a team in America/New_York see the same lines at different UTC offsets. First hop: the source log format; the format on the stage.timestamp block.

How to troubleshoot it

The diagnose-first order. Each step is read-only.

1. Confirm the symptom. Reproduce the wrong-time symptom. The metric is the signal. The query is the symptom.
2. Confirm the source clock. timedatectl status on the source host. The System clock synchronized line should be yes. The NTP service should be active.
3. Confirm the agent’s parse log. journalctl -u alloy -n 50 | grep -i timestamp. The failed to parse timestamp line is the smoking gun.
4. Confirm the distributor’s discarded samples. loki_discarded_samples_total by reason. The reason label identifies the failure shape.
5. Confirm the ingester’s out-of-order counter. loki_ingester_received_entries and the loki_ingester_streams gauge. A stream with a single flat timestamp is the symptom of one rejected entry among many.
6. Confirm the query returns the wrong-time entries. logcli query with --since=1h. The timestamps should be within the last hour. A line whose timestamp is days in the past is the wrong-time symptom.

The fix is to chase the source clock, the format string, or the time zone. The Loki side is rarely the cause. The fix is rarely in Loki.

Security implications

A wrong timestamp can hide an attack in two ways. The retention window can be set so narrow that the entries from the attack are dropped before they are stored. The clock skew can be so deliberate that the operator cannot correlate the entries with the rest of the trace. Treat a sudden, large increase in loki_discarded_samples_total{reason="older_than"} as a candidate for a security event. The metric is the only signal.

The Loki distributor accepts any push whose timestamp is within the rejection_older_than window. The window is the trust boundary. A narrow window is a small trust boundary; a wide window is a large trust boundary. The choice is operational.

Performance implications

A wrong timestamp adds no cost to the chunk fetch. The wrong timestamp adds cost to the operator’s time. The cost is in the investigation that reaches the wrong conclusion because the chart at the wrong time told the wrong story.

The other performance trap is the rejection_older_than limit. A wide window accepts more entries; the retention window later deletes them. A narrow window rejects entries at the distributor; the metric is non-zero. The choice is operational; the trade-off is between the metric and the storage.

Production guidance

• Configure the agent to parse the source timestamp. The format string must match the format of the line.
• Set action_on_error = "fwd" for production to avoid losing lines on a parser error. The cost is a non-zero “failed to parse” log; the benefit is no data loss.
• Alert on loki_discarded_samples_total growth. The reason label identifies the failure shape.
• Reject the source clock skew over widening the retention window. The window is the trust boundary; the clock is the agent’s responsibility.
• Run chronyc tracking on every source host. The drift is the cause; the metric is the symptom.

Verification

You should now be able to answer:

• What is the difference between a source timestamp and an ingestion timestamp?
• Which Loki metric is the smoking gun for a timestamp that is older than the per-tenant window?
• Which Loki metric is the smoking gun for an out-of-order timestamp?
• What is the role of the agent’s stage.timestamp block?
• Why is a wrong timestamp a worse symptom than a missing log?

Quiz