Snowflakes and phoenixes — the topology of replaceable infrastructure

A snowflake is a node that became unique through drift. A phoenix is a node you can replace from an image. The cluster is only as healthy as the personality of its worst node, because Kubernetes treats every node as interchangeable until the operator has proven otherwise.

The three node personalities

A node in a Kubernetes cluster falls into one of three personalities:

1. Snowflake. A worker that has accumulated unique state: one-off kernel tuning, hand-installed debug tools, a leftover iptables rule, a custom CNI override. The node is the only one in the cluster with that state, and the next person who reboots it discovers the breakage.
2. Golden image. A worker that is built from a known image on every boot. The image is versioned, signed, and reviewed. A node can be replaced by booting another from the same image, and the cluster cannot tell the difference.
3. Phoenix. A worker that is rebuilt on every boot. The node is ephemeral by policy: state is on attached storage; the boot disk is wiped on every reboot. The cluster treats the node as a transient slot, not a host.

flowchart TD
    A[Node personality] --> B[Snowflake]
    A --> C[Golden image]
    A --> D[Phoenix]
    B --> B1[Drift accumulates]
    B --> B2[Hard to replace]
    B --> B3[Owner is the operator who last touched it]
    C --> C1[Image is versioned]
    C --> C2[Replace from image]
    C --> C3[Owner is the image builder]
    D --> D1[Rebuilt every boot]
    D --> D2[State on storage only]
    D --> D3[Owner is the cluster]

The order is also a maturity ladder. Most clusters start with snowflakes (because that is the default when an operator manually installs the first node), mature to golden images, and end at phoenix for stateless workloads.

Where snowflakes come from

A node becomes a snowflake through accumulation, not intention:

• A one-off sysctl to debug a memory leak that was never reverted.
• A manually installed tcpdump to capture a packet that was never removed.
• A debug kernel module loaded for a CNI test that was never unloaded.
• A node label that was never used anywhere else.
• A taint that was applied to bypass an affinity problem and never removed.

Each of these is small. The accumulation is large. After a year, the snowflake has dozens of one-offs and the operator is afraid to reboot it because they don’t know which one was load-bearing.

The golden image

The golden image is the antidote. A golden image is a reproducible artefact — built by an image pipeline (Packer, Kubernetes-Operator-driven image-builder, etc.) from a known manifest: which kernel, which packages, which sysctls, which CNI version, which kubelet flags. The artefact is signed and versioned; every node in the pool boots from the same artefact.

kubectl get nodes -o custom-columns=NAME:.metadata.name,IMAGE:.status.nodeInfo.osImage,BOOT:.status.nodeInfo.bootID

The bootID is the freshness check. If two nodes in the same pool have wildly different boot IDs, the cluster is mixing fresh and stale nodes. In a golden-image fleet, every node boot in a pool is short; the boot ID rotates every image update.

The phoenix

The phoenix is the next step. A phoenix node is rebuilt on every boot — the OS disk is wiped, the image is re-applied, and the node is re-registered. The cluster’s state lives on attached storage or in the control plane, not on the node. This is the model that Kubernetes’ Machine/MachineSet abstractions (Cluster-API, Karpenter) are designed around.

The phoenix model is mandatory for stateless workloads. It is permissible for stateful workloads only if the state is on persistent volumes that survive the reboot, and the application is designed to tolerate the node’s identity changing.

Cattle, not pets

The standard rule is “cattle, not pets.” In Kubernetes, the rule is enforced by the control plane: the scheduler may re-schedule a Pod off any node; the node controller may mark any node as NotReady and reschedule its workloads; the autoscaler may remove a node from the pool. A snowflake is a node that resists these operations because it carries state the cluster does not know about.

The cluster’s health is the inverse of snowflake density. A cluster with 100% phoenix nodes is replaceable in any order. A cluster with even one snowflake is replaceable in any order except that one.

Production discipline

A production-grade Kubernetes fleet is operated as if every node will be replaced tomorrow. The image is the source of truth, the boot is reproducible, and the state is on attached storage. Snowflakes are tolerated only when the cluster has no mature image pipeline; the goal is to eliminate them.

• Treat every node as replaceable. The cluster’s health is the inverse of snowflake density.
• Operate a node-book. Every manual touch on a node is committed to a repo with a date and a reason.
• Build a golden image. Packer or a Kubernetes-native image-builder produces a versioned, signed artefact.
• Aim for phoenix. Stateless workloads rebuild on every boot; the cluster’s state lives in the control plane and on attached volumes.
• Replace snowflakes deliberately. A node book kept up to date is the trigger for replacement; the absence of one is a warning sign.

Quiz