Prometheus exporters — textfile collector, custom scripts, and the SNMP exporter

A “router export is broken” report is not an exporter issue until the operator has the metric, the label set, and the validation evidence. Prometheus exporters are the bridge between the router’s internal state and the time-series database. The operator who cannot build, debug, and validate an exporter in 30 minutes will spend hours guessing whether the metric is missing, the label set is wrong, or the validation is incomplete.

This lesson is the production reference for Prometheus exporters on VyOS 1.5 LTS: the three exporter patterns (textfile collector, custom Python exporter, SNMP exporter), what each one gives the operator, what each one costs, and the validation discipline that prevents a custom exporter from becoming the next source of incidents.

The three exporter patterns

flowchart LR
  subgraph ROUTER["VyOS 1.5 router"]
    S["Script<br/>(shell, Python)"]
    SNMP["SNMP agent"]
  end
  subgraph EX["Exporter pattern"]
    TF["Textfile collector<br/>(node_exporter)"]
    PY["Custom Python exporter<br/>(prometheus-client)"]
    SX["SNMP exporter<br/>(snmp_exporter)"]
  end
  subgraph PROM["Prometheus"]
    P["Prometheus server"]
  end
  S --> TF
  S --> PY
  SNMP --> SX
  TF --> P
  PY --> P
  SX --> P

Textfile collector is the canonical pattern for simple derived metrics on a router. The operator writes a shell script that writes the metric in Prometheus text exposition format to a file in /var/lib/node_exporter/textfile/, and the node_exporter reads the file on every scrape. The cost: the operator must write and maintain the script; the metric is a snapshot (not a counter that increments in the exporter).

Custom Python exporter is the canonical pattern for protocol-specific state that the textfile collector cannot easily express. The operator writes a Python script that uses the prometheus-client library to expose metrics on an HTTP endpoint. The exporter’s process is long-lived; the metrics are computed on every scrape. The cost: the operator must write and maintain the script; the exporter can have a memory leak or crash loop.

SNMP exporter is the canonical pattern for bridging legacy NMS data into Prometheus. The operator configures the snmp_exporter to walk the SNMP MIBs and translate the counters into Prometheus metrics. The cost: the operator must configure the SNMP exporter’s generator.yml to map the MIBs; the exporter is a separate process.

The textfile collector pattern

The textfile collector is the canonical pattern for derived metrics on a router. The operator writes a shell script that writes the metric in Prometheus text exposition format to a file in /var/lib/node_exporter/textfile/.

#!/bin/bash
TEXTFILE=/var/lib/node_exporter/textfile/vyos-tunnel-uptime.prom
TMPFILE=$(mktemp)

echo "# HELP vyos_wireguard_uptime_seconds WireGuard tunnel uptime in seconds" > $TMPFILE
echo "# TYPE vyos_wireguard_uptime_seconds gauge" >> $TMPFILE

# Get the latest handshake timestamp for each peer and compute the uptime
wg show all latest-handshakes | while read -r peer ts; do
  if [ -n "$ts" ] && [ "$ts" != "0" ]; then
    now=$(date +%s)
    uptime=$((now - ts))
    echo "vyos_wireguard_uptime_seconds{peer=\"$peer\"} $uptime" >> $TMPFILE
  fi
done

mv $TMPFILE $TEXTFILE

The script runs on a 1-minute cron, the exporter reads the file on every scrape, and the Prometheus server sees the metric as a gauge. The discipline: use # HELP and # TYPE lines, use a stable file location, and use mktemp + mv for atomic writes (the exporter reads the file on every scrape, so a partial write would be parsed as a partial metric).

The operator validates the textfile collector:

$ /usr/local/bin/write-vyos-tunnel-uptime.sh
$ cat /var/lib/node_exporter/textfile/vyos-tunnel-uptime.prom
# HELP vyos_wireguard_uptime_seconds WireGuard tunnel uptime in seconds
# TYPE vyos_wireguard_uptime_seconds gauge
vyos_wireguard_uptime_seconds{peer="abc123"} 12345
vyos_wireguard_uptime_seconds{peer="def456"} 67890
$ curl -s http://10.0.0.1:9100/metrics | grep vyos_wireguard_uptime_seconds
vyos_wireguard_uptime_seconds{peer="abc123"} 12345
vyos_wireguard_uptime_seconds{peer="def456"} 67890

The metric is present, parseable, and exposed by the node_exporter.

The custom Python exporter pattern

The custom Python exporter is the canonical pattern for protocol-specific state that the textfile collector cannot easily express. The operator writes a Python script that uses the prometheus-client library to expose metrics on an HTTP endpoint.

#!/usr/bin/env python3

import time
from prometheus_client import start_http_server, Gauge
import subprocess
import json

# Define the metrics
vrf_route_count = Gauge(
    'vyos_vrf_route_count',
    'Number of routes per VRF',
    ['vrf', 'address_family']
)

interface_carrier = Gauge(
    'vyos_interface_carrier',
    'Interface carrier state (1 = up, 0 = down)',
    ['interface']
)

bgp_session_up = Gauge(
    'vyos_bgp_session_up',
    'BGP session state (1 = up, 0 = down)',
    ['peer', 'remote_as']
)

def collect_metrics():
    """Collect metrics from the router's operational state."""
    # Get per-VRF route count
    result = subprocess.run(
        ['vtysh', '-c', 'show ip route summary json'],
        capture_output=True, text=True
    )
    data = json.loads(result.stdout)
    for vrf, count in data.get('routes_per_vrf', {}).items():
        vrf_route_count.labels(vrf=vrf, address_family='ipv4').set(count)

    # Get interface carrier state
    result = subprocess.run(
        ['ip', '-j', 'link', 'show'],
        capture_output=True, text=True
    )
    for link in json.loads(result.stdout):
        interface_carrier.labels(interface=link['ifname']).set(
            1 if link.get('carrier', 0) == 1 else 0
        )

    # Get BGP session state
    result = subprocess.run(
        ['vtysh', '-c', 'show ip bgp summary json'],
        capture_output=True, text=True
    )
    data = json.loads(result.stdout)
    for peer, info in data.get('ipv4Unicast', {}).get('peers', {}).items():
        bgp_session_up.labels(
            peer=peer,
            remote_as=str(info.get('remoteAs', 0))
        ).set(1 if info.get('state') == 'Established' else 0)

if __name__ == '__main__':
    # Start the HTTP server on port 9103
    start_http_server(9103)
    while True:
        collect_metrics()
        time.sleep(30)

The exporter exposes the metrics on http://10.0.0.1:9103/metrics. The operator installs the exporter as a systemd service:

[Unit]
Description=VyOS Prometheus exporter
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/vyos-prometheus-exporter.py
Restart=always
RestartSec=10
User=root
MemoryMax=512M
CPUQuota=25%

[Install]
WantedBy=multi-user.target

The discipline: use Type=simple, Restart=always, RestartSec=10, User=root, MemoryMax=512M, and CPUQuota=25%. The MemoryMax and CPUQuota prevent the exporter from consuming too many resources; the Restart=always policy ensures the exporter is restarted after a crash.

The SNMP exporter pattern

The SNMP exporter is the canonical pattern for bridging legacy NMS data into Prometheus. The operator configures the snmp_exporter to walk the SNMP MIBs and translate the counters into Prometheus metrics.

# /etc/snmp_exporter/snmp.yml
modules:
  if_mib:
    walk:
      - 1.3.6.1.2.1.2.2.1.1   # ifIndex
      - 1.3.6.1.2.1.2.2.1.8   # ifOperStatus
      - 1.3.6.1.2.1.31.1.1.1.6  # ifHCInOctets
      - 1.3.6.1.2.1.31.1.1.1.10 # ifHCOutOctets
    metrics:
      - name: ifInOctets
        oid: 1.3.6.1.2.1.31.1.1.1.6
        type: counter
      - name: ifOutOctets
        oid: 1.3.6.1.2.1.31.1.1.1.10
        type: counter
      - name: ifOperStatus
        oid: 1.3.6.1.2.1.2.2.1.8
        type: gauge
    auth:
      community: public  # Or use SNMPv3

The exporter is configured as a systemd service:

[Unit]
Description=SNMP exporter
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/snmp_exporter --config.file=/etc/snmp_exporter/snmp.yml
Restart=always
RestartSec=10
User=root

[Install]
WantedBy=multi-user.target

The operator configures the Prometheus server to scrape the SNMP exporter:

scrape_configs:
  - job_name: 'snmp'
    static_configs:
      - targets: ['10.0.0.1:9116']
    metrics_path: /snmp
    params:
      module: [if_mib]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 10.0.0.1:9116

The snmp_exporter walks the SNMP MIBs and translates the counters into Prometheus metrics. The operator can now alert on the SNMP metrics in Prometheus, even if the original source is a legacy NMS.

Validation discipline

Every exporter must be validated end-to-end before production deployment. The validation discipline:

1. Confirm the exporter is running. The operator runs systemctl status <exporter> and confirms the daemon is active.
2. Confirm the metrics are present. The operator curls the exporter’s /metrics endpoint and confirms the metrics are returned.
3. Confirm the metric is changing. The operator curls the metrics twice, 5 seconds apart, and confirms the counters have incremented.
4. Confirm the label set is correct. The operator inspects the label set and confirms it matches the expected schema (e.g., peer is the peer’s IP, remote_as is the peer’s AS).
5. Confirm the rate is sensible. The operator computes the rate and compares it to the expected rate.
6. Confirm the alert fires under the expected condition. The operator tests the alert by inducing the condition and confirming the NMS alerts.

Production failure modes

The exporter failure modes the operator encounters:

• Textfile collector script fails silently. The script produces an empty output file; the exporter reports stale metrics. Fix: add error handling to the script (exit on error, log to syslog); monitor the file’s mtime.
• Custom Python exporter has a memory leak. The exporter’s memory usage grows over time; the router runs out of memory. Fix: cap the exporter’s memory usage via cgroup (MemoryMax=512M); restart the exporter periodically.
• SNMP exporter has a wrong OID. The exporter walks the wrong MIB; the metrics are missing or wrong. Fix: test the exporter with a known-good SNMP walk; use the snmp_exporter generator to produce the snmp.yml configuration.
• Exporter uses too much CPU. The exporter’s CPU usage is high; the router’s CPU is overloaded. Fix: cap the exporter’s CPU usage via cgroup (CPUQuota=25%); reduce the scrape frequency.
• Exporter is not in the Prometheus scrape configuration. The exporter is running but Prometheus is not scraping it. Fix: add the exporter to the Prometheus scrape configuration; validate the target is reachable.
• Exporter reads stale data. The exporter reads the FRR JSON output, but the FRR daemon is not refreshing the JSON. Fix: validate the FRR daemon is running; validate the JSON output is current.

Rollback

Exporter changes are typically configuration-only, but the impact can be cross-cutting. The rollback discipline:

• Textfile collector script — installed in /usr/local/bin. The rollback is to remove the script and the cron entry.
• Custom Python exporter — installed in /usr/local/bin and /etc/systemd/system/. The rollback is to remove the script and disable the systemd service.
• SNMP exporter — installed in /usr/local/bin and /etc/snmp_exporter/. The rollback is to remove the binary and the configuration.
• Prometheus scrape configuration — the Prometheus server is a separate system. The rollback is to revert the Prometheus configuration, not the exporter.

For every change, use commit-confirm:

configure
# ... make the change ...
commit-confirm 5
# If the change has unintended consequences, the auto-rollback
# fires after 5 minutes and the previous configuration is restored.

Production discipline

Cross-course references

• Part XLIX-01 (vyos-xlix-01-interface-metrics) covers the SNMP and Prometheus transports that the exporter layer builds on.
• Part XLIX-05 (vyos-xlix-05-system-metrics) covers the node_exporter and prometheus-vyos-exporter that the exporter layer extends.
• The Observability course covers the consumer side: Prometheus, Grafana, alerting on the exporter’s metrics.
• The Ansible course’s XLII-Ansible-BeyondLinux covers the automation hand-off (rolling out the exporter to a fleet via a single playbook).
• The Linux course’s XXII-Linux-NetTroubleshoot covers the underlying cgroup and systemd primitives that the exporter uses.

Quiz