SNMP for Network Devices

The new switch arrives. It has no agent, no SSH-instrumented metric daemon, and no OpenTelemetry SDK. It has an SNMP MIB and a community string. The lesson that follows is how to bring that switch into the same Prometheus shape as every host on the fleet, without dragging the rest of the platform back into SNMP.

What it is

snmp_exporter is the Prometheus exporter for SNMP. It exposes a single HTTP endpoint, conventionally /snmp, that accepts a target, module, and (optionally) auth parameter, performs one or more SNMP walks and gets against the target, and returns the result as Prometheus exposition format. Prometheus then scrapes that endpoint; the act of scraping is the poll.

The exporter translates the device’s SNMP MIB into Prometheus metrics through a configuration file, conventionally snmp.yml. The file is generated by generator against a MIB directory. The generator turns MIB definitions into a walk list and a metric-mapping section. The exporter uses the walk list to poll and the metric mapping to label the response.

The canonical alternative is a vendor-specific telemetry agent (gNMI, OpenConfig, streaming telemetry). SNMP is the right approach when the device is older, when the platform does not yet speak gNMI, or when the team is not ready to run a config management pipeline for switch telemetry. SNMPv3 with auth and priv is the right approach when the device supports it.

Why a sysadmin cares

Three production failure classes disappear the day the exporter is in place:

• Interface saturation on a switch. The link to the database host is at 92 percent. The host metrics say nothing about it because the saturation is at the switch port, not at the NIC. The metric that catches this is ifHCInOctets polled from IF-MIB, exposed as if_in_octets.
• CRC errors on a long run. A fibre run is degrading. The switch counts the errors; the host NIC does not see them because the frames are dropped at the switch port. The metric that catches this is ifInErrors from IF-MIB.
• Routing table instability. A flapping route causes a router to advertise and withdraw prefixes rapidly. The metric that catches this is a counter on routing table size or BGP state transitions.

These metrics already exist on the device. The cost of exposing them is a generated config and a scrape job. The value is that the switch shows up on the same dashboard as the host it connects.

How it works

The exporter exposes /snmp (and a few helpers). Prometheus sends a request with target, module, and (optionally) auth query parameters. The exporter looks up the module in snmp.yml, looks up the auth in auth.yml, opens an SNMP session, performs the walks and gets, and formats the response.

    prometheus.yml
         |
         v
   scrape job: snmp_switch_if
         |
         |  __param_target rewritten from __address__
         |  __param_module = if_mib
         |
         v
   snmp_exporter :9116/snmp?target=...&module=if_mib
         |
         |  walks: 1.3.6.1.2.1.2.2.1.10 (ifInOctets)
         |         1.3.6.1.2.1.2.2.1.16 (ifOutOctets)
         |         1.3.6.1.2.1.2.2.1.14 (ifInErrors)
         |         1.3.6.1.2.1.2.2.1.20 (ifOutErrors)
         |         1.3.6.1.2.1.31.1.1.1.6  (ifHCInOctets, 64-bit)
         |         1.3.6.1.2.1.31.1.1.1.10 (ifHCOutOctets, 64-bit)
         |
         v
   if_in_octets{ifIndex="49", ifDescr="GigabitEthernet0/49"} 1.82e+10
   if_in_errors{ifIndex="49", ifDescr="GigabitEthernet0/49"} 0
   if_out_octets{ifIndex="49", ifDescr="GigabitEthernet0/49"} 9.12e+09

The exporter is stateless between scrapes. There is no caching, no per-target session reuse, and no internal scheduling. Each scrape is a fresh poll. For targets with hundreds of walks, this is the dominant cost.

The if_mib walk

The most polled module is if_mib. It walks the standard interface table:

• ifInOctets, ifOutOctets (32-bit, can wrap)
• ifHCInOctets, ifHCOutOctets (64-bit, no wrap)
• ifInErrors, ifOutErrors
• ifInDiscards, ifOutDiscards
• ifInUnknownProtos
• ifSpeed, ifHighSpeed
• ifAdminStatus, ifOperStatus

The exporter maps each OID to a Prometheus metric name with labels. The generator constructs the walk list and the metric mapping from the MIB.

How to configure it

Three layers are required: the generated snmp.yml, the auth file, and the scrape job.

1. The generated snmp.yml

The generator ships as a separate binary or as a make generator target in the snmp_exporter source tree. It reads MIB definitions and emits snmp.yml:

# Clone the snmp_exporter repository and place vendor MIBs in mibs/.
git clone https://github.com/prometheus/snmp_exporter.git
cd snmp_exporter
cp /var/lib/snmp-mibs/*.mib mibs/
go run ./generator/ --fail-on-parse-errors generate
# Output: snmp.yml

The generated file is hundreds of lines. The relevant section for the interface table:

# snmp.yml (excerpt)
if_mib:
  walk:
    - 1.3.6.1.2.1.2.2          # IF-MIB::ifTable
    - 1.3.6.1.2.1.31.1.1       # IF-MIB::ifXTable (HC counters)
    - 1.3.6.1.2.1.2.2.1.1      # ifIndex
    - 1.3.6.1.2.1.2.2.1.2      # ifDescr
    - 1.3.6.1.2.1.2.2.1.5      # ifSpeed
    - 1.3.6.1.2.1.2.2.1.10     # ifInOctets
    - 1.3.6.1.2.1.2.2.1.14     # ifInErrors
    - 1.3.6.1.2.1.2.2.1.16     # ifOutOctets
    - 1.3.6.1.2.1.2.2.1.20     # ifOutErrors
    - 1.3.6.1.2.1.31.1.1.1.1   # ifName
    - 1.3.6.1.2.1.31.1.1.1.6   # ifHCInOctets
    - 1.3.6.1.2.1.31.1.1.1.10  # ifHCOutOctets
    - 1.3.6.1.2.1.31.1.1.1.15  # ifHighSpeed
    - 1.3.6.1.2.1.31.1.1.1.7   # ifHCInUcastPkts
    - 1.3.6.1.2.1.31.1.1.1.11  # ifHCOutUcastPkts
  metrics:
    - name: if_in_octets
      oid: 1.3.6.1.2.1.31.1.1.1.6
      type: counter
      indexes:
        - labelname: ifIndex
          type: gauge
      lookups:
        - labels:
            - ifIndex
          labelname: ifDescr
          oid: 1.3.6.1.2.1.2.2.1.2
          type: DisplayString
        - labels:
            - ifIndex
          labelname: ifName
          oid: 1.3.6.1.2.1.31.1.1.1.1
          type: DisplayString
    - name: if_in_errors
      oid: 1.3.6.1.2.1.2.2.1.14
      type: counter
      indexes:
        - labelname: ifIndex
          type: gauge

The walk list is a list of OIDs. The generator walks the subtree under each OID and emits one series per leaf. The ifHCInOctets OID is preferred over ifInOctets because the HC variant is 64-bit and does not wrap at 100 Mbps.

2. The auth file

auth.yml holds credentials. v2c uses community strings; v3 uses a username, auth protocol, auth password, priv protocol, and priv password.

# /etc/snmp/auth.yml (v2c example)
snmp_switch_prod:
  community: public
  version: 2
  auth_protocol: ''
  priv_protocol: ''

# /etc/snmp/auth.yml (v3 example)
snmp_router_prod:
  community: ''
  version: 3
  security_level: authPriv
  username: prometheus
  auth_protocol: SHA
  auth_password: file:/etc/snmp/auth_password
  priv_protocol: AES
  priv_password: file:/etc/snmp/priv_password
  context_name: ''

The file: prefix tells the exporter to read the password from a file rather than the YAML. This is the standard form for production: the YAML is committed, the passwords are not.

3. The scrape job

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: snmp_switch_if
    metrics_path: /snmp
    params:
      module: [if_mib]
      auth: [snmp_switch_prod]
    scrape_interval: 60s
    scrape_timeout: 30s
    static_configs:
      - targets:
          - 10.20.30.1
          - 10.20.30.2
        labels:
          device_vendor: cisco
          role: aggregation
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__address__]
        target_label: instance
      - target_label: __tmp_community
        replacement: snmp_switch_prod
      - source_labels: [__tmp_community]
        target_label: auth

The auth parameter selects the named credential from auth.yml. The __param_target relabel rule rewrites the target into the exporter’s query parameter.

How to validate it

Three layers must be confirmed: the exporter is up, the walk returned the expected metrics, and Prometheus is receiving the series.

# 1. The exporter is up.
curl -sf http://localhost:9116/metrics | grep -E '^snmp_'
# snmp_exporter_build_info{version="0.26.0"} 1
# snmp_exporter_scrape_walk_duration_seconds{module="if_mib"} 0.214

# 2. The probe works manually.
curl -sf "http://localhost:9116/snmp?target=10.20.30.1&module=if_mib&auth=snmp_switch_prod" \
  | grep -E '^if_in_octets'
# if_in_octets{ifIndex="49",ifDescr="GigabitEthernet0/49",instance="10.20.30.1"} 1.82e+10

# 3. Prometheus is receiving the series.
up{job="snmp_switch_if"}
# {instance="10.20.30.1", job="snmp_switch_if"} 1

ifHCInOctets{job="snmp_switch_if"}
# {instance="10.20.30.1", ifIndex="49"} 1.82e+10

# 4. Walk from the command line for cross-check.
snmpwalk -v2c -c public 10.20.30.1 1.3.6.1.2.1.31.1.1.1.6 \
  | head -5
# IF-MIB::ifHCInOctets.49 = Counter64: 18234567890

If snmpwalk returns values that the exporter does not, the auth file is wrong or the walk is filtered by an ACL on the device. If the exporter returns values but Prometheus is not receiving them, the scrape job is missing the relabel rules.

How it can fail

Six failure modes appear regularly. Each one is recognisable in the data.

1. ACL on the device. The exporter cannot reach the SNMP port on the device. The walk returns nothing; the scrape returns up=0 for the target. Symptom: every target on a particular subnet is up=0; targets outside the subnet work.

2. Wrong auth. The community string is wrong, the v3 password is wrong, or the auth protocol does not match what the device expects. Symptom: snmp_exporter_scrape_errors rises for the affected targets; the response body is empty.

3. MIB not loaded on the device. The walk references an OID the device does not implement. The exporter returns a partial result; the missing OIDs are simply absent. Symptom: some metrics present, others absent, no error in the exporter logs.

4. SNMP version mismatch. The exporter is configured for v3 but the device only accepts v2c, or vice versa. Symptom: snmp_exporter_scrape_errors rises with a “no such name” or “unknown user name” error in the exporter log.

5. 32-bit counter wrap on a fast interface. The walk references ifInOctets instead of ifHCInOctets. The 32-bit counter wraps every 4 GB at gigabit speeds. Symptom: the rate shows a periodic negative spike; the dashboard plots a sawtooth.

6. Walk timeout. The walk includes too many OIDs for the scrape interval to fit. Symptom: scrape_timeout fires; up{job="snmp_*"}=0 for the affected targets; the exporter log shows “context deadline exceeded”.

How to troubleshoot it

Order matters. Start at the boundary where evidence is most concrete.

1. Is the exporter alive? curl http://exporter:9116/-/ready and curl http://exporter:9116/metrics first. A missing snmp_exporter_build_info series means the exporter is not running. Look at systemd or container logs.
2. Does the device respond to SNMP at all? snmpwalk -v2c -c COMMUNITY TARGET 1.3.6.1.2.1.1.1.0 (sysDescr). If the device does not respond, the problem is at the network or ACL boundary.
3. Does the exporter’s auth match? Test with the same auth parameter the scrape job sends: curl "http://exporter:9116/snmp?target=X&module=Y&auth=Z". A 200 with empty body means the walk returned no data; a non-200 means auth or transport failed.
4. Does the walk list match the device’s MIB? Cross-check the exporter’s response against snmpwalk for the same OID. A mismatch means the generated snmp.yml is out of date relative to the device firmware.
5. Check the scrape_timeout. If the walk completes in 8 seconds but the scrape timeout is 5 seconds, Prometheus times out. Increase the timeout or split the walk into smaller modules.
6. Look at the per-target error counter. snmp_exporter_scrape_errors is the headline metric for authentication and walk failures.

Security implications

SNMPv2c traverses the network in plaintext. A community string on a switch in a co-located facility is observable by any device on the same VLAN. Treat every v2c community as a public credential.

SNMPv3 with authPriv is the right answer for production. The credentials are hashed and encrypted on the wire. The cost is a configuration change on the device; the benefit is that a packet capture no longer reveals credentials.

The auth.yml file holds plaintext passwords. The exporter must run with a restrictive filesystem policy. The file: prefix decouples the password from the YAML; the password file must have 0600 permissions and be readable only by the exporter’s user.

Do not bind the exporter’s HTTP endpoint on the public interface. The /snmp endpoint is unauthenticated; any caller can request a walk against any target the exporter can reach.

Performance implications

The dominant cost is the walk itself. A walk over the full if_mib is a few hundred SNMP GETNEXT requests. A switch with a thousand ports answers each request in milliseconds; the walk completes in a few seconds. A slow device or a high-latency link stretches this to the scrape timeout.

scrape_interval and scrape_timeout must be tuned to the device. A 60-second scrape interval with a 30-second timeout is a common starting point. A 10,000-port chassis polled every 30 seconds with 50 OIDs per walk will saturate the device’s SNMP handler. The right answer is fewer modules per target, not a shorter scrape interval.

Cardinality scales with ifIndex. A switch with 256 ports emits 256 series per metric. Twenty metrics in if_mib is 5,120 series per target. A fleet of a hundred switches is half a million series. Whitelist the ports you alert on, or use a recording rule to summarise.

Production guidance

• Use SNMPv3 with authPriv on every production device. The cost is real but the alternative is a plaintext credential on the wire.
• Whitelist the SNMP poller’s source address on the device’s ACL. A switch that accepts SNMP from any source is a switch that will be scanned.
• Pin the ifHCInOctets and ifHCOutOctets OIDs, not the 32-bit variants. The 32-bit variants wrap on gigabit links in minutes.
• Validate the generated snmp.yml in CI. The generator can fail on a parse error or a missing MIB; the change is the CI artefact.
• Document the auth file convention. file: prefix, 0600 permissions, owner is the exporter user. A committed plaintext password is an incident waiting to happen.

Verification

You should now be able to answer:

• Why use ifHCInOctets and not ifInOctets for a gigabit link?
• What is the operational difference between SNMPv2c and SNMPv3 with authPriv?
• How is the auth parameter on the scrape job wired to the auth.yml?
• What does __param_target do in the relabel_configs for an snmp_exporter scrape job?
• What four boundaries can an SNMP poll failure live at, and which metric reveals each?

Quiz