topk() and bottomk()

topk() and bottomk() is a load-bearing concept in the observability platform. This lesson covers what it is, why a sysadmin cares, how it works, how to configure it, how it can fail, how to troubleshoot it, and the production discipline around it.

What it is

A precise definition of topk() and bottomk(), scoped to production operations. This concept is a building block of the Prometheus, Grafana, Loki, and Tempo stack. It is taught in this part because it is a prerequisite for the alerting, dashboarding, and correlation parts of the course.

Why a sysadmin cares

Production framing. topk() and bottomk() exists because a real operational question needs to be answered cheaply and reliably. Without it, the operator either answers the question slowly (high investigation time) or guesses (high incident risk). The cost of topk() and bottomk() is small; the cost of not having it is large.

A common production mistake is to skip topk() and bottomk() in the initial deployment. The team discovers during the first major incident that the platform has no way to answer the question, and the investigation stalls. The lesson exists to prevent this.

How it works

The mental model. Where in the platform the concept lives. Which subsystem is responsible. Which configuration files influence it. The Prometheus data model, the Loki indexing model, and the Tempo storage model are all introduced in their respective architecture parts.

# Configuration snippet illustrating the lesson topic
example_setting: value
# explicit comments mark the production defaults

How to configure it

Real configuration examples with annotated options. Three layers:

1. Source code: the instrumentation library or exporter configuration.
2. Service config: the daemon-level configuration (e.g. prometheus.yml).
3. Platform config: the platform-wide configuration (e.g. grafana datasource provisioning).

Each layer has its own conventions. The lesson covers the most common options for topk() and bottomk().

# Validate the configuration
command-line-tool check example.yml

How to validate it

Commands the operator runs to confirm the configuration is live and correct. Outputs are real or labelled illustrative.

# Show the configuration currently in effect
promtool check config /etc/prometheus/prometheus.yml

# Show the live /metrics endpoint of an exporter
curl -sf http://localhost:9100/metrics | head -20

The outputs confirm:

• The service started with the configuration file.
• The relevant endpoints respond.
• The metrics are being scraped into Prometheus.
• The dashboards and alerts reference the expected metric.

How it can fail

The high-frequency failure modes:

1. Silent misconfiguration. The configuration loads but the option is wrong; the platform accepts the value but the behaviour is not what was intended.
2. Crash on load. An invalid option causes the daemon to exit. Promtool’s check config catches this; production does not.
3. Performance regression. A configuration value causes the service to consume excessive memory, CPU, or disk. The platform continues but the host degrades.
4. Permissions failure. A filesystem permission prevents the service from reading its configuration or writing its data. The service refuses to start or panics at runtime.
5. Schema/version drift. A new version of the daemon renames an option. The old option silently becomes a no-op or causes an error. Reading the release notes catches this.

The lesson returns to each in production depth across the alerting, gathering, and storage parts.

How to troubleshoot it

Diagnostic steps. The order matters.

1. Was it working before? Identify the most recent change.
2. What does the service’s own view say? /-/ready, /-/healthy for Prometheus; /metrics for exporters; /ui for Loki and Tempo.
3. What does the platform’s view say? The Prometheus up metric; the Loki loki_internal_* metrics; the Tempo metrics.
4. Form hypothesis, find evidence, test, validate. Apply the methodology introduced in Part I.

The discipline is to identify the boundary at which the failure is — between the producer and the platform, between the platform and the storage, between the platform and the user — and to narrow the search there.

Security implications

topk() and bottomk() has security implications wherever the relevant component exposes an HTTP endpoint, an authentication layer, or a credential. Default configurations often expose more than is appropriate:

• The Prometheus API is unprotected by default.
• The Grafana UI is anonymous-admin by default.
• The Loki API is unauthenticated by default.
• The Tempo API is unauthenticated by default.
• All exporters expose /metrics over HTTP without authentication.

The platform security part of the course returns to each.

Performance implications

Performance implications of topk() and bottomk() come from:

• The cardinality of the metrics or labels.
• The scrape / push interval.
• The size of the rule files.
• The retention of the storage.
• The query cost on a dashboard.

Each has a lever. The lesson identifies the lever for the specific topic; the performance part of the course returns to the broader discipline.

Production guidance

• Treat the configuration as code. Store it in version control.
• Validate before applying: promtool check config, logcli validate-config, tempo-cli validate-config, amtool check-config.
• Use a small set of canonical configurations (one per service class) rather than per-instance variations.
• Document the operational defaults in the team’s instrumentation guide.
• Test changes in a non-production environment; production validation is a sanity check, not a finding.

Verification

You should now be able to answer:

• What is topk() and bottomk() in production terms?
• Why does a sysadmin care about it?
• How does it fail and how do you diagnose the failure?
• What are the security and performance implications?

Quiz