SDN IPAM and DNS: NetBox, phpIPAM and PowerDNS · Proxmox VE

The previous lesson built zones, VNets and subnets. It left one question open, and it is the question that decides whether SDN is worth the machinery: who decides which address a new guest gets, and who else finds out about it?

Without IPAM, the answer is a person and a spreadsheet. Somebody picks 192.0.2.47 because the wiki page says .40 to .60 are free, writes it into the guest, updates the wiki, and — on a good day — asks somebody to add a DNS record. Three of those four steps are done by hand and the fourth is done by a different team.

With IPAM and a DNS plugin, creating a guest allocates the address and registers the name. That is a real reduction in operational surface, and it introduces a dependency that was not there before.

Three IPAM plugins

An IPAM is selected per zone, in the zone’s IPAM field, and defaults to pve.

PVE IPAM — the built-in one

The default. Allocations are stored in the Proxmox cluster filesystem alongside the rest of the SDN configuration under /etc/pve/sdn, which means they are replicated to every node and shared with them automatically.

You can “inspect the current status of the PVE IPAM Plugin via the IPAM panel in the SDN section of the datacenter configuration”, and create, update and delete IP mappings there.

This is the right starting point for almost every estate. It has no external dependency, it survives a node loss because it is in pmxcfs, and it is visible in the same GUI as everything else.

Its limitation is scope: it knows about the addresses Proxmox allocated, and nothing else. If your network team also assigns addresses in the same subnets — for switches, printers, physical servers — the PVE IPAM has no view of those, and nothing prevents a collision.

NetBox

Two configuration properties:

URL — “The NetBox REST API endpoint: http://yournetbox.domain.com/api”
Token — “An API access token”

The token is generated in NetBox itself, and the documentation points at the upstream instructions for doing so.

Configuration changeregister a NetBox IPAM— Creates the IPAM entry at datacenter level. It is not used until a zone selects it. The token is a credential with write access to your address database, so it belongs in /etc/pve, which is mode 0700 for its private subtree - not in a playbook checked into git.

pvesh create /cluster/sdn/ipams \
--ipam netbox-prod \
--type netbox \
--url https://netbox.example.com/api \
--token REPLACE_ME

pvesh get /cluster/sdn/ipams --output-format yaml

NetBox is the right choice when it is already the organisation’s source of truth for addressing. The value is not that Proxmox allocates addresses better — it is that the network team’s view and the virtualisation team’s view stop being two databases that disagree.

phpIPAM

Three configuration properties:

URL — “The REST-API endpoint: http://phpipam.domain.com/api/<appname>/”
Token — “An API access token”
Section — “An integer ID. Sections are a group of subnets in phpIPAM. Default installations use sectionid=1 for customers.”

The <appname> component of the URL is the API application you created inside phpIPAM, not a placeholder Proxmox fills in. Getting it wrong produces an authentication failure rather than a not-found error, which sends people to the token.

DNS: the PowerDNS plugin

A DNS plugin is also selected per zone, and it is what turns an allocated address into a resolvable name.

PowerDNS has to be prepared first. The documentation is specific: “You need to enable the web server and the API in your PowerDNS config” with

api=yes
api-key=arandomgeneratedstring
webserver=yes
webserver-port=8081

Then the plugin takes three options:

url — “The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost”
key — “An API access key”
ttl — “The default TTL for records”

The zone then carries the DNS settings:

Field	Level	Meaning
`DNS`	Zone	“DNS API server. Optional.”
`ReverseDNS`	Zone	“Reverse DNS API server. Optional.”
`DNSZone`	Zone	“DNS domain name. Used to register hostnames… The DNS zone must already exist on the DNS server. Optional.”
`DNS Zone Prefix`	Subnet	“Add a prefix to the domain registration”

Two details in that table repay attention.

The DNS zone must already exist. Proxmox registers records into a zone; it does not create the zone. A DNSZone naming something PowerDNS does not serve fails at registration time, which is to say at guest creation time.

Forward and reverse are separate settings. Configuring DNS and leaving ReverseDNS empty gives you A records and no PTR records — a state that looks complete until something does a reverse lookup, which in practice is a monitoring system or an SSH server with UseDNS.

DNS Zone Prefix is per subnet, which is how several subnets in one zone get distinct names: a guest in the subnet with prefix dmz becomes hostname.dmz.example.com rather than colliding with the same hostname in another subnet.

Read-only / Safewhat is configured today— Read-only. Shows the registered IPAM and DNS backends and which zones reference them. A zone with no ipam field is using the pve default rather than nothing.

pvesh get /cluster/sdn/ipams --output-format yaml
pvesh get /cluster/sdn/dns --output-format yaml
pvesh get /cluster/sdn/zones --output-format yaml

Verification that can fail

The configuration being accepted proves that Proxmox could parse it. It does not prove that the far service will answer, and the two failures look different at very different times.

Read-only / Safeprove the external services actually answer— Read-only against both services. Run it from a Proxmox node, not from your workstation - the node is what will need to reach them at guest-creation time, and a firewall between them is exactly the defect this catches.

NETBOX=https://netbox.example.com
PDNS=http://pdns.example.com:8081

curl -sS -o /dev/null -w 'netbox api: %{http_code}\n' \
-H "Authorization: Token REPLACE_ME" \
"$NETBOX/api/"

curl -sS -o /dev/null -w 'powerdns api: %{http_code}\n' \
-H "X-API-Key: REPLACE_ME" \
"$PDNS/api/v1/servers/localhost"

Then do the thing that actually matters: create a guest on a VNet in the zone, and check that the address appears in the IPAM and the name resolves.

Read-only / Safeend-to-end check after creating a test guest— Read-only. If the allocation appears in the IPAM panel but the name does not resolve, the IPAM half works and the DNS half does not - which narrows the problem to the DNS plugin or the zone that was supposed to already exist.

HOSTNAME=sdn-test-01
DOMAIN=example.com

pvesh get /cluster/sdn/ipams/pve/status --output-format yaml 2>/dev/null | head -30

dig +short "$HOSTNAME.$DOMAIN"
dig +short -x 192.0.2.47

An external IPAM makes VM provisioning depend on a service the virtualisation team does not run

This is the trade, and it is easy to make without noticing.

With the built-in PVE IPAM, creating a guest touches only the cluster. With NetBox or phpIPAM selected on a zone, creating a guest on that zone requires an HTTP call to another system to succeed. If NetBox is down, or its certificate expired overnight, or somebody rotated the token, provisioning on that zone stops.

The failure is not subtle when you are watching for it and is very confusing when you are not, because the error surfaces as a guest creation failure on a hypervisor cluster that is otherwise perfectly healthy. Nothing in pvecm status, pvesm status or the node summary points at an IPAM.

Three things make it manageable:

Put the IPAM’s availability in the same tier as the cluster’s, or accept explicitly that provisioning has a maintenance window when the IPAM does. Monitor the API endpoint from a Proxmox node rather than from a monitoring host on a different network path. And keep at least one zone on the built-in pve IPAM so there is a route to create a guest when the external one is unavailable.

Under the hood

Where the state lives, and why that determines the failure modes.

SDN configuration — zones, VNets, subnets, and the IPAM and DNS backend definitions — lives in /etc/pve/sdn, which is pmxcfs. Everything about that directory follows from the properties of pmxcfs, and those properties are the ones that surprise people.

It is replicated. A zone created on one node exists on all of them immediately. So does the IPAM token.

It is quorum-gated. When the cluster loses quorum, /etc/pve goes read-only. An SDN change cannot be applied and, with the built-in PVE IPAM, a new allocation cannot be recorded — so guest creation on an IPAM-backed zone fails during a quorum incident even though the individual node is fine.

It is in your configuration backup. Which is the good news and the subtle hazard. Restoring /etc/pve from a backup taken last month restores last month’s allocation table, and every address allocated since then is now recorded as free while still being in use. The resulting duplicate allocation appears days later as an intermittent connectivity problem on two guests that have nothing to do with each other.

An external IPAM inverts this. The allocation table is not in your backup and is not restored with it, so a /etc/pve restore cannot create phantom free addresses — but the IPAM is now a system with its own availability, its own backup, and its own change window.

The sequence at guest creation follows from the documented pieces: when a subnet has an IPAM associated with its zone, “the subnet prefix will be automatically registered in the IPAM”, and an address from it is allocated to the guest and recorded. If a DNS backend and DNSZone are configured, the hostname is registered as hostname.domain — with the subnet’s DNS Zone Prefix inserted when one is set. Each of those is a separate operation against a separate system, which is why partial success is a state you will see: an address allocated and no DNS record, or a record created for a guest whose creation then failed.

That last case is the one to have a reconciliation story for. The IPAM panel exists precisely so that a mapping created by a failed operation can be deleted by hand.

What happens if the IPAM and reality stop agreeing?

You get duplicate addresses, and they present as the least helpful symptom in networking: two hosts that work most of the time.

The drift accumulates from ordinary operations rather than from mistakes:

A guest restored from backup. The restore recreates the VM with its old address. If the IPAM freed that address when the original was deleted, it may since have handed it to someone else.

A guest whose NIC was reconfigured by hand. Somebody set a static address inside the guest that the IPAM never allocated. The IPAM believes it is free.

A failed provisioning run. The address was allocated, the guest creation failed afterwards, and the allocation was never released.

A /etc/pve restore, with the built-in IPAM. Every allocation since the backup is now recorded as free.

The symptom, when the duplicate finally matters, is ARP-level: two hosts answer for one address, the switch’s MAC table flaps between two ports, and traffic reaches whichever host answered most recently. Both hosts appear healthy. Connectivity to each is intermittent in a way that correlates with nothing. The usual diagnosis path — check the guest, check the bridge, check the firewall — finds nothing wrong, because nothing on either guest is wrong.

What actually identifies it:

# from a node on the same L2 segment
ip neigh show | sort -k1
arping -D -I vmbr0 -c 3 192.0.2.47

A duplicate-address probe that gets a reply is conclusive, and it takes seconds once you have thought of it.

Reconciliation is periodic and manual, and worth scheduling rather than doing after an incident:

Export what the IPAM believes. The IPAM panel for the built-in plugin; the API for NetBox or phpIPAM.

Collect what is actually configured. Guest network configuration is in /etc/pve/qemu-server/ and /etc/pve/lxc/, and the live view is the neighbour table on each bridge.

Diff them, and treat the discrepancies as findings rather than correcting them silently — an address in use and not in the IPAM may be a printer nobody told you about, and deleting the IPAM’s opinion is not always the right resolution.

The estates that avoid this problem are not the ones with better tools. They are the ones where a single system is authoritative and everybody knows which one it is.

Common mistakes

Treating the IPAM as the only record and never reconciling it. It drifts from restores, hand-edits and failed provisioning runs.
Pointing DNSZone at a zone PowerDNS does not serve. The zone must already exist; registration fails at guest creation.
Configuring DNS and forgetting ReverseDNS. You get A records and no PTR records, and discover it through a monitoring system.
Selecting NetBox on every zone. Provisioning now has a dependency outside the cluster on every path. Keep one zone on pve.
Getting the phpIPAM <appname> wrong in the URL. The error looks like an authentication failure.
Storing the IPAM token in a playbook rather than letting Proxmox hold it. It is a write credential for the organisation’s address database.
Assuming an external IPAM is restored with /etc/pve. It is not, which is a feature — and it means it needs its own backup.
Building production automation on the tech-preview surface without reading release notes at upgrade time.

Key takeaways

IPAM is selected per zone and defaults to the built-in pve plugin, whose state lives in /etc/pve/sdn and is replicated across the cluster.
NetBox takes a URL and a token; phpIPAM takes a URL, a token and a numeric section ID.
The PowerDNS plugin needs the API and web server enabled on the PowerDNS side, and takes url, key and ttl.
Zones carry DNS, ReverseDNS and DNSZone; subnets carry a DNS Zone Prefix. The DNS zone must already exist.
An external IPAM makes guest creation depend on a service the cluster does not control. Monitor it from a node and keep a pve-IPAM zone as a fallback.
The built-in IPAM is in your configuration backup, so a /etc/pve restore can resurrect a stale allocation table and cause duplicate addresses.
Reconcile the IPAM against guest configuration and the neighbour table on a schedule, not after an incident.

Knowledge check

Why this matters

Address management is the part of a virtualisation platform that is easiest to leave manual and hardest to leave manual for long.

At twenty guests, a spreadsheet is fine. At two hundred, the spreadsheet is wrong and everybody knows it is wrong, so people stop trusting it and start checking with ping before assigning an address — which is how you get two hosts on 192.0.2.47, because both engineers pinged it while the other one’s guest was shut down.

SDN IPAM fixes that, and the fix is worth having. What this lesson asks you to notice is that it fixes it by moving the problem rather than eliminating it. The spreadsheet was unreliable and had no dependencies. The IPAM is reliable and has one, and if you choose an external plugin that dependency sits outside the cluster, outside your change control, and outside the diagnostics anybody reaches for when a hypervisor misbehaves.

That is usually the right trade. It becomes the wrong one when nobody writes down that it was made — when a provisioning failure at 9 AM sends three people into pvecm status and journalctl -u pvedaemon for forty minutes before somebody thinks to curl NetBox.

The two habits that make this work are small. Monitor the IPAM and DNS endpoints from a Proxmox node, so the check follows the same path the real request will. And reconcile the IPAM against reality on a schedule, because drift is produced by normal operations — restores, hand-edits, failed runs — and not by anybody doing anything wrong.

An IPAM that nobody reconciles is a spreadsheet with an API.