Writing an Exporter

A team has a queue that nobody has heard of: 80,000 messages a minute, a single producer, three consumers, no metrics. The team looks for a community exporter, finds none that are maintained, and decides to write their own. Two weeks later the exporter is in production. Six months later it is the team’s most reliable signal during incidents.

The first question on a custom exporter is not “how do I write it”. The first question is “should I”. This lesson covers both.

What it is

Writing an exporter means implementing the Prometheus contract in a new program. In Go, the canonical implementation uses github.com/prometheus/client_golang, which provides the metric types, the registry, the /metrics handler, and the counter-reset detection that makes counters behave correctly across process restarts.

A minimal exporter has four parts:

• A registry. The registry holds the metrics. The default registry includes Go runtime metrics and process metrics for free; you register your own metrics against it.
• A set of metric variables. Counters, gauges, histograms, summaries. Declared with promauto so registration is automatic and the variable is incremented in place.
• A /metrics handler. promhttp.Handler() serves the registry in the Prometheus text format.
• An HTTP server. A standard net/http mux that exposes /metrics, /healthz, and a sane default for /.

Anything beyond this is glue: configuration parsing, logging, graceful shutdown, packaging.

Why a sysadmin cares

Writing an exporter is a code path that will live for years. Three properties follow from the choice to write one:

• Maintenance. The exporter is now part of your codebase. Upgrades to the Prometheus contract, dependency updates, CVE fixes — all of it is your problem.
• Test surface. A custom exporter must be tested like any other code: unit tests for metric emission, integration tests for the HTTP handler, end-to-end tests against a real Prometheus.
• Operational signal. A custom exporter can observe things no off-the-shelf exporter can. The trade-off is that the observation is only as good as the code.

The decision to write one should be defended: there is no existing exporter, no instrumentation library is available, and the cost of writing is justified by the operational gain.

How it works

The mental model is a registry that you put metrics into, and a handler that serialises the registry:

   metric.NewCounter()         promauto.NewCounter()
           |                              |
           v                              v
       +---------------------------------------+
       |             Registry                  |
       |  (default = promauto.DefaultRegister) |
       +---------------------------------------+
                       |
                  promhttp.Handler()
                       |
                       v
                 HTTP /metrics
                  text/plain

promauto is a thin helper that registers the metric as it is constructed. Without it, you would build the metric, then call prometheus.MustRegister(...) separately. promauto reduces the ceremony.

The default registry in client_golang already contains the Go runtime collectors (go_*) and the process collectors (process_*). You do not need to register those by hand. If you want a clean registry with only your own metrics, build a new prometheus.NewRegistry() and register your metrics against it; do not use prometheus.DefaultRegisterer in that case.

The HTTP handler does two things: it collects every metric in the registry, and it serialises the result as the Prometheus text format. The handler is also where http.CloseNotifier runs, so a slow scrape cancels cleanly.

How to configure it

The configuration of a custom exporter is the code itself and its runtime flags. There is no prometheus.yml for the exporter; the exporter reads its own configuration.

1. The minimal Go exporter. A working example:

package main

import (
  "context"
  "log/slog"
  "net/http"
  "os"
  "os/signal"
  "syscall"
  "time"

  "github.com/prometheus/client_golang/prometheus"
  "github.com/prometheus/client_golang/prometheus/promauto"
  "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
  queueDepth = promauto.NewGauge(prometheus.GaugeOpts{
    Name: "queue_depth",
    Help: "Current number of messages in the queue.",
  })

  messagesProcessed = promauto.NewCounterVec(
    prometheus.CounterOpts{
      Name: "queue_messages_processed_total",
      Help: "Total messages processed, labelled by outcome.",
    },
    []string{"outcome"}, // success | error | retry
  )

  processingDuration = promauto.NewHistogram(
    prometheus.HistogramOpts{
      Name:    "queue_processing_duration_seconds",
      Help:    "Time spent processing a single message.",
      Buckets: prometheus.ExponentialBuckets(0.001, 2, 12),
    },
  )
)

func recordMetrics() {
  // In a real exporter this would read from the queue's API.
  // The values here are placeholders for the example.
  queueDepth.Set(float64(1234))
  messagesProcessed.WithLabelValues("success").Inc()
  processingDuration.Observe(0.025)
}

func main() {
  mux := http.NewServeMux()
  mux.Handle("/metrics", promhttp.Handler())
  mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    _, _ = w.Write([]byte("ok"))
  })
  // The / handler should NEVER expose business data. A bare
  // 200 OK is the right default.
  mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    http.Error(w, "see /metrics", http.StatusNotFound)
  })

  go func() {
    ticker := time.NewTicker(5 * time.Second)
    defer ticker.Stop()
    for range ticker.C {
      recordMetrics()
    }
  }()

  srv := &http.Server{
    Addr:              "127.0.0.1:9101",
    Handler:           mux,
    ReadHeaderTimeout: 5 * time.Second,
  }

  ctx, stop := signal.NotifyContext(context.Background(),
    syscall.SIGINT, syscall.SIGTERM)
  defer stop()

  go func() {
    slog.Info("exporter starting", "addr", srv.Addr)
    if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
      slog.Error("server failed", "err", err)
      os.Exit(1)
    }
  }()

  <-ctx.Done()
  slog.Info("shutdown signal received")
  shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
  defer cancel()
  if err := srv.Shutdown(shutdownCtx); err != nil {
    slog.Error("shutdown failed", "err", err)
  }
}

The exporter binds to 127.0.0.1:9101 so it is not exposed to the network by default. The metrics are declared with HELP and TYPE via the library. Counters end in _total. Histogram buckets are exponential, sized for the expected latency.

2. Unit tests. The client_golang testutil package makes metric assertions straightforward:

package main

import (
  "net/http/httptest"
  "strings"
  "testing"

  "github.com/prometheus/client_golang/prometheus"
  "github.com/prometheus/client_golang/prometheus/promhttp"
  "github.com/prometheus/client_golang/prometheus/testutil"
)

func TestMetrics_AreRegistered(t *testing.T) {
  expected := []string{
    "queue_depth",
    "queue_messages_processed_total",
    "queue_processing_duration_seconds",
  }
  for _, name := range expected {
    if testutil.CollectAndCount(prometheus.DefaultRegisterer.(prometheus.Gatherer)) < 0 {
      t.Fatalf("registry unhealthy")
    }
    if testutil.CollectAndCompare(
      prometheus.DefaultRegisterer.(prometheus.Gatherer),
      strings.NewReader("# HELP "+name),
      name,
    ) != nil {
      t.Errorf("metric %s not registered", name)
    }
  }
}

func TestMetricsEndpoint_ServesText(t *testing.T) {
  req := httptest.NewRequest("GET", "/metrics", nil)
  w := httptest.NewRecorder()

  promhttp.Handler().ServeHTTP(w, req)

  if w.Code != 200 {
    t.Fatalf("expected 200, got %d", w.Code)
  }
  body := w.Body.String()
  for _, want := range []string{
    "# HELP queue_depth",
    "# TYPE queue_depth gauge",
    "queue_messages_processed_total",
  } {
    if !strings.Contains(body, want) {
      t.Errorf("body missing %q", want)
    }
  }
}

func TestCounterIncrements(t *testing.T) {
  before := testutil.ToFloat64(messagesProcessed.WithLabelValues("success"))
  messagesProcessed.WithLabelValues("success").Inc()
  after := testutil.ToFloat64(messagesProcessed.WithLabelValues("success"))
  if after-before != 1 {
    t.Errorf("expected counter to increment by 1, got %v", after-before)
  }
}

testutil.ToFloat64 reads the current value of a single metric. testutil.CollectAndCompare parses an expected text fixture and diffs it against the registry. The httptest round trip confirms the handler is wired correctly.

3. Container. A minimal Dockerfile:

# syntax=docker/dockerfile:1.7
FROM golang:1.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags='-s -w' -o /out/exporter ./

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/exporter /exporter
USER nonroot:nonroot
EXPOSE 9101
ENTRYPOINT ["/exporter"]

The distroless base image has no shell and no package manager, which reduces the attack surface. The build is reproducible because of -trimpath and the version-controlled go.sum.

4. What to expose at /. The default for / should be either 404 Not Found (the example above) or a small static “this is the queue-exporter” page with a link to /metrics. Do not expose queue contents, message bodies, or any business data at /. The Prometheus contract is for metrics; business data is for the application’s own API.

How to validate it

Validate that the exporter speaks the contract and that the metrics are correct. All commands are READ-ONLY.

# 1. Confirm the binary version and build info.
./exporter --version

# 2. Confirm /metrics serves the contract.
curl -sI http://127.0.0.1:9101/metrics | head -5

HTTP/1.1 200 OK
Content-Type: text/plain; version=0.0.4; charset=utf-8

# 3. Confirm the metric names and types are correct.
curl -sf http://127.0.0.1:9101/metrics | grep -E \
  '^(# HELP queue_depth|# TYPE queue_depth|queue_depth)'

# 4. Run the contract check.
curl -sf http://127.0.0.1:9101/metrics > /tmp/queue.prom
promtool check metrics /tmp/queue.prom

queue.prom: OK

# 5. Run the Go tests.
go test ./... -count=1

ok      queue-exporter   0.014s

# 6. Confirm Prometheus sees the exporter and the metric is
#    ingestable.
up{job="queue-exporter"}
queue_messages_processed_total

The outputs confirm: the binary reports its version, the endpoint serves the contract, the body parses, the unit tests pass, and Prometheus is ingesting the metrics.

How it can fail

Five specific failure modes:

1. Cardinality explosion in a label. A label is set to a user-supplied value (a request ID, a customer ID). The metric count explodes. Symptom: prometheus_tsdb_head_series for the job grows by millions per hour; *_test_util would not have caught this because the test fed fixed labels.
2. Histogram buckets sized for the wrong range. A histogram with buckets from 0.001 s to 1 s for a service whose real latency is 5-30 s puts every observation in the +Inf bucket. Symptom: histogram_quantile(0.95, ...) returns the bucket boundary; dashboards show a flat line at the upper limit.
3. Counter used where a gauge is needed. A counter that goes up and down — e.g. “current queue depth” as a counter — is a contract violation. Symptom: rate() of a gauge-like counter produces nonsense; the dashboard for “current queue depth” looks like a saw-tooth.
4. Goroutine leak under load. The exporter spawns a goroutine per scrape or per record call and never cleans up. Symptom: go_goroutines climbs steadily; eventually the exporter is OOM-killed.
5. Missing graceful shutdown. The exporter’s record loop is interrupted by SIGTERM mid-update. Symptom: the last scrape before shutdown emits a partial metric; restart re-emits from zero, which Prometheus handles correctly via counter-reset detection, but a histogram in flight is lost.

How to troubleshoot it

Diagnose in this order; it is cheapest to confirm the contract first and the values second.

1. Is the endpoint reachable? curl -I first.
2. Does the body parse? promtool check metrics.
3. Are the expected metrics present? grep for the names you registered. If a metric is missing, registration failed silently.
4. Are the values plausible? Compare the metric value to the upstream system’s own view. If the values disagree, the exporter is reading the wrong source.
5. Are the labels bounded? count by (label_name) (metric_name) in PromQL. A label with millions of values is a cardinality bug.
6. Are the goroutine and memory counters stable? go_goroutines and process_resident_memory_bytes. A steady climb is a leak.
7. Did the unit tests run in CI? A custom exporter without tests is a deployment with no safety net.

Security implications

A custom exporter inherits the security model of the system it observes. The node_exporter-class risks apply: it reads files, it holds credentials, it binds to a network interface. The minimum discipline is:

• Bind to localhost by default. Expose via a reverse proxy or service mesh when remote scraping is required.
• Run as a non-root user when possible. The distroless image above runs as nonroot.
• Do not expose business data at /. The root path is the first thing a casual visitor sees; the contract only requires /metrics to be served.
• Do not put secrets in label values. The contract permits arbitrary UTF-8 in labels; the security boundary is what you put there. A label named customer_email is a data leak in waiting.
• Restrict outbound network. A custom exporter should not phone home. If it must (for license checks, for example), the destination should be on an allow-list.

The security lesson in this module covers each of these in depth.

Performance implications

A custom exporter’s cost is dominated by three things:

• The scrape interval. Each scrape serialises the entire registry. A registry with 10,000 metric families and a 5-second scrape interval is a sustained serialisation cost.
• The number of registered metrics. Each metric adds memory, even if it has not been incremented.
• The cost of the upstream read. If the exporter reads from a database or a queue, the read cost is the dominant factor. Cache the read; do not hit the upstream on every scrape.

The trade-off: a custom exporter gives you exact metrics for a system nobody else has metrics for. The cost is the maintenance and the operational discipline to keep it honest.

Production guidance

• Pin prometheus/client_golang and prometheus/common in go.mod. Renovate or Dependabot should propose upgrades.
• Run promtool check metrics in CI against a snapshot of the /metrics output. The check is cheap and catches contract drift.
• Set GOMAXPROCS explicitly in containers; do not rely on the Go runtime default.
• Expose the exporter’s own *_build_info, go_*, and process_* metrics. They are free, and they are how you alert on the exporter itself.
• Write at least one test per metric family. The discipline is small; the payoff is large.
• Bind to localhost or a private interface. Use a sidecar proxy or service mesh for remote scraping.

Verification

You should now be able to answer:

• When does it make sense to write a new exporter, and when should you adopt one or use an instrumentation library instead?
• What does promauto do, and why is it preferred over manual MustRegister calls?
• How would you unit test a counter, a gauge, and the /metrics handler?
• What is the right bind address for a custom exporter in production, and what should the root path / return?
• What three metric types should a custom exporter always expose for self-observation?

Quiz