TCP Probes

The blackbox panel turns red on the row for the managed Postgres instance. The on-call opens the database team’s runbook. The runbook says: verify the connection. The on-call reaches for the TCP probe, hits it, and gets green. The database is still not accepting queries. The TCP probe, on its own, never claimed to be a database probe; the on-call conflated the two. This lesson is about what tcp_connect actually does and what it does not.

What it is

The tcp_connect module of blackbox_exporter performs a single TCP three-way handshake against the target host and port. It returns success if the handshake completed within the configured timeout and failure otherwise. There is no application-level exchange. There is no protocol-level verification. There is no authentication. The exporter does not send bytes; it does not expect bytes back. The probe is a connect() followed by a graceful close.

The headline metric is probe_success; the latency metric is probe_duration_seconds; the rest of the protocol-specific metrics (probe_http_status_code, probe_dns_lookup_time_seconds) are absent because the protocol is too thin to expose them.

Why a sysadmin cares

The TCP probe is the honest fallback when an HTTP or TLS probe does not fit. Three production questions map cleanly onto the TCP module:

• Is the port open? A database listener on 5432, a Kafka broker on 9092, a Redis on 6379, a Postgres replica on 5432. These services speak protocols that are not HTTP; the blackbox exporter speaks HTTP and TCP. The TCP probe is the closest the exporter comes to “yes, the listener is up.”
• Is a firewall dropping traffic at this boundary? A load balancer that listens on TCP/443 but rejects otherwise valid traffic. The TCP probe passes, the HTTP probe fails, and the boundary is named.
• Did the listener restart? A crash loop on a managed service. The TCP probe flaps as the listener restarts; the panel shows the shape of the failure; the alert fires with enough information to diagnose.

The probe is also the cheapest signal a service exposes at the network boundary. A TCP probe is roughly an order of magnitude cheaper than the equivalent HTTP probe, because there is no handshake payload, no TLS negotiation, and no application parsing.

How it works

The exporter invokes the Go net.DialTimeout against the configured host and port. The kernel performs the standard POSIX three-way handshake: SYN, SYN-ACK, ACK. Once the local socket reaches ESTABLISHED, the exporter closes it. The round-trip time of the handshake plus the FIN exchange is what probe_duration_seconds measures.

  exporter host                  target host
       |                              |
       | --- SYN --->                 |
       |                              |
       | <-- SYN-ACK ---              |
       |                              |
       | --- ACK --->                 |
       |                              |
       | --- FIN --->                 |
       |                              |
       | <-- FIN-ACK ---              |
       |                              |
       v                              v
   probe_success=1              listener receives
   probe_duration_seconds =     nothing
     ~ RTT + FIN

Two configuration choices govern the rest of the behaviour: preferred_ip_protocol and ip_protocol_fallback. The exporter resolves the hostname to one or more addresses, prefers the family the operator has named, and (only if ip_protocol_fallback: true) tries the other family when the chosen family fails.

The module also accepts an optional tcp.query_response configuration. The exporter can be taught to send a literal payload after the handshake, wait for a number of bytes, and expect a regex match. This is application-level probing over a TCP socket and is the only path by which the TCP probe can verify anything other than “the listener exists”.

  use case                          | approach
  ----------------------------------+--------------------------
  postgres port 5432 reachable?     | tcp_connect, no payload
  postgres accepting queries?       | tcp_connect + StartTLS + actually query
  redis reachable?                  | tcp_connect + PING\r\n, expect +PONG
  postgres or redis with strict auth| tcp_connect + actual credentials

Without a query-response block, tcp_connect answers the narrowest possible question and nothing more.

How to configure it

The TCP module has fewer options than http_2xx. Most of the configuration lives at two layers: the module configuration and the scrape job. Below is a blackbox.yml fragment with the TCP variants a typical production environment uses.

# /etc/blackbox/blackbox.yml
modules:

  # Bare reachability, no payload. Default for any port check.
  tcp_connect_pg:
    prober: tcp
    timeout: 3s
    tcp:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true

  # Reachability that prefers IPv6 first, falls back to IPv4.
  tcp_connect_dual:
    prober: tcp
    timeout: 3s
    tcp:
      preferred_ip_protocol: ip6
      ip_protocol_fallback: true

  # Reachability with a TLS handshake (not an HTTP request).
  # Useful when the application speaks its own TCP+TLS protocol
  # and the operator needs to verify the chain before continuing.
  tcp_connect_tls_pg:
    prober: tcp
    timeout: 5s
    tcp:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true
      tls: true
      tls_config:
        insecure_skip_verify: false

  # Application-level probing: send "PING\r\n" and expect
  # to receive exactly 7 bytes matching "+PONG\r\n".
  # Required when the operator must verify the application
  # is not just accepting connections but is actively replying.
  redis_ping:
    prober: tcp
    timeout: 3s
    tcp:
      query_response:
        - expect: "PONG"      # arbitrary substring match (regex)
          send: "PING\r\n"
          timeout: 1s

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: blackbox_tcp_pg
    metrics_path: /probe
    params:
      module: [tcp_connect_pg]
    scrape_interval: 30s
    scrape_timeout: 10s
    static_configs:
      - targets: ['db-write-1.example.internal:5432']
        labels:
          service: postgres
          role: writer
          env: prod
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: module
        replacement: tcp_connect_pg
      # Strip the port from instance so alerts group by host.
      - source_labels: [__param_target]
        regex: '(.*):.*'
        replacement: '${1}'
        target_label: instance

The preferred_ip_protocol choice deserves more care than it looks like. Three production shapes:

• IPv4 only. The hostname resolves to A only. Setting preferred_ip_protocol: ip6 produces a probe that fails by design before it ever starts.
• IPv6 only. The newer shape. Many cloud-managed services announce AAAA first and A second. Setting preferred_ip_protocol: ip4 works because of fallback; without fallback, the probe fails.
• Dual stack. The default-fallback combination. The exporter tries the preferred family, falls back if needed, records probe_ip_protocol on the wire (since blackbox_exporter 0.24), and the dashboards show which family succeeded.

How to validate it

# 1. Direct probe against the database port.
curl -sf "http://blackbox:9115/probe?module=tcp_connect_pg&target=db-write-1.example.internal:5432" \
  | grep -E '^probe_'
# probe_duration_seconds 0.014
# probe_ip_protocol 4
# probe_success 1

# 2. The same probe against a port that should not be open.
curl -sf "http://blackbox:9115/probe?module=tcp_connect_pg&target=db-write-1.example.internal:65000" \
  | grep -E '^probe_'
# probe_duration_seconds 3.012       # timed out as configured
# probe_success 0

# 3. Compare to a healthy vs unhealthy replica.
curl -sf "http://blackbox:9115/probe?module=tcp_connect_pg&target=db-replica-3.example.internal:5432" \
  | grep -E '^probe_success'
# probe_success 1                   # port is open
# But application-level:
redis-cli -h db-replica-3.example.internal ping
# Could not connect: Connection refused / TLS required / etc.

# 4. Confirm Prometheus has the metric.
probe_success{module="tcp_connect_pg"}
# {service="postgres",role="writer",env="prod"} 1

A green TCP probe and a red application redis-cli test is the canonical symptom of “port reachable but database not accepting.” The next step is a TLS probe plus the application log, not a rollback.

How it can fail

1. NAT rebinding. The target resolves through a load balancer that NAT’s the connection. The exporter records probe_success=1 because the handshake completed, but the backend listener has been replaced mid-flight and the application cannot make sense of subsequent traffic. Symptom: probe green, application errors in the logs.

2. TCP backlog full. A SYN arrives, the kernel replies with RST because the accept queue is exhausted. probe_success=0 but the application logs no obvious error. Symptom: probes flap between 0 and 1, every flap is short.

3. TLS presentation middleware. The probe with tls: true records probe_success=1 against an upstream proxy that terminates TLS differently from the application. The application rejects the proxy’s certificate because the CN is wrong. Symptom: TLS probe green, application rejects peer.

4. IP family mismatch with no fallback. The exporter resolves only IPv6, and the operator set preferred_ip_protocol: ip4 without enabling ip_protocol_fallback. Every probe fails by configuration. Symptom: every probe probe_success=0, internal target resolves fine, exporter host has IPv4 only.

5. Timeout shorter than handshake + retry. A managed service that takes four seconds to respond on cold cache. Timeout is one second. Probe fails. Symptom: probe_duration_seconds close to the timeout value, not close to healthy RTT.

6. Relabel bug copies port into instance. The instance label is set to the full host:port. Every host:port is a new series. The cardinality rises with the number of distinct ports probed. Symptom: TSDB pressure; the metric’s instance label is unusable.

7. query_response regex broken. The expected application greeting changes (a version bump, a rebrand). The regex no longer matches. Probe returns probe_failed_due_to_regex on a healthy application. Symptom: every probe 0, the service is up.

How to troubleshoot it

The order matters because the boundary at which the failure lives determines the remedy.

1. Confirm the port is open from the exporter host. nc -vz target 5432 or bash -c "echo > /dev/tcp/target/5432" is the one-line test. If this fails, the problem is at the network or listener, not at the exporter.
2. Run the same probe manually. curl "http://exporter:9115/probe?module=...". Look at probe_duration_seconds and whether there is a probe_ip_protocol label.
3. Cross-check with node_exporter netstat metrics. If the exporter host reports `node_netstat_Tcp_Estab{…}

   0`, the connection completes; if not, the failure is at the listener.

4. Compare TCP probe to HTTP probe, same target. TCP green, HTTP red means the listener accepts and the application is broken. TCP red means the listener, firewall, or routing is broken.
5. Compare probe latency against baseline. A drift from ten milliseconds to four hundred milliseconds on a TCP probe is a brownout; the next failure will be a SYN timeout.
6. Compare TCP probe to TLS probe. A green TCP and a red TLS points at the certificate chain, not the port.
7. Compare probe to internal up{job="<service>"}. If the application is up internally and the probe is red, the problem is outside the application’s trust zone.

Security implications

The TCP module accepts the target as a URL parameter. Without care, the exporter becomes a port-scan primitive — an attacker who can reach the exporter can ask it to probe any IP and port the exporter host can reach. Bind the exporter to a private network; do not expose it on the public internet.

When tls: true is set, tls_config.insecure_skip_verify bypasses chain verification. The probe hands back probe_success=1 against any certificate, including expired or wrong-host. Set the verifier properly; the audit will read the production config, not the documentation.

The query_response block sends the configured payload verbatim. Do not embed credentials, session tokens, or PII in the payload. The exporter logs are not a credential store, and the body is a single probe attempt; if the payload changes per scrape, you have built a noisy log channel by accident.

Performance implications

A TCP probe is roughly an order of magnitude cheaper than an HTTP probe: one SYN, one SYN-ACK, one ACK, one FIN, one FIN-ACK on the host. At scrape_interval=30s across one hundred targets, the exporter consumes approximately two hundred sockets per minute. The Go runtime handles this without ceremony.

The cost rises with two patterns. First, probes with query_response that send and wait hold the goroutine for the expected time. A one-second timeout on a hundred-target scrape is acceptable; a ten-second timeout is a budget problem. Second, probes with tls: true perform a full TLS handshake per scrape — twice as expensive as a plain TCP probe and an order of magnitude more expensive than a connect() shortcut.

Production reality: the exporter is rarely the bottleneck. The limiting factor in a healthy black-box deployment is the scrape budget Prometheus allocates, not the exporter’s throughput.

Production guidance

• Use tcp_connect for port reachable questions. Promote the probe to a TLS or HTTP variant when the question is richer than “the listener exists.”
• Pick preferred_ip_protocol deliberately. The default is IPv4 because most operators historically preferred it; many modern cloud-managed services announce IPv6 first. The wrong setting produces a probe that is green internally and red from the production path, or vice versa.
• Set ip_protocol_fallback: true unless the team has a specific reason to avoid it. The cost of fallback is one additional dial attempt on a hard failure; the benefit is surviving an IP-family outage without configuration change.
• Use tls: true when the question is “is the certificate valid on this port,” not “is the application accepting queries.” That question needs a real client.
• Treat the query_response payload as code. Review changes.
• Relabel to strip :port from instance. Group alerts by service, not by host:port.

Verification

You should now be able to answer:

• What does tcp_connect actually verify about a target?
• Why does the TCP probe alone fail to detect a database that is rejecting every session?
• When is preferred_ip_protocol a configuration choice that matters, and which symptom does the wrong setting produce?
• Why is TLS validation sometimes a separate probe from the TCP probe rather than an HTTP probe?
• Which of probe_success, probe_duration_seconds, probe_http_status_code is exposed by tcp_connect?

Quiz