Lab 2: Inspect the control plane and etcd

Objective

By the end of this lab you will be able to say “the control plane is healthy” and point at the evidence for each half of that sentence separately — the API server, etcd, the scheduler, the controller manager — rather than at a green kubectl get nodes.

The lab is deliberately almost entirely read-only. It creates one ConfigMap and one snapshot file and changes nothing else, because the skill being practised is the one you need before you are allowed to change anything: establishing what is actually true about a control plane you did not build, at speed, from the cluster itself.

Two habits are the target. The first is reaching for systemctl and journalctl on a kubeadm host, where they will tell you, truthfully and uselessly, that no such units exist. The second is treating kubectl get nodes as a control-plane check. It is a kubelet check. Every node in this lab will report Ready in scenarios where the control plane is degraded.

Architecture

One control-plane host, running four control-plane containers that no service manager knows about, plus the kubelet that supervises them.

flowchart TB
    subgraph HOST["k8s-cp-1 · 192.0.2.11"]
        KUBELET["kubelet<br/>(systemd unit)"]
        DIR["/etc/kubernetes/manifests/<br/>4 YAML files"]
        API["kube-apiserver<br/>:6443"]
        ETCD["etcd<br/>:2379 client · :2380 peer"]
        SCH["kube-scheduler"]
        CM["kube-controller-manager"]
        DATA["/var/lib/etcd"]
    end
    DIR -.->|"read every fileCheckFrequency"| KUBELET
    KUBELET -->|CRI| API
    KUBELET -->|CRI| ETCD
    KUBELET -->|CRI| SCH
    KUBELET -->|CRI| CM
    API <-->|"gRPC + mTLS"| ETCD
    ETCD --> DATA
    SCH -->|"lease + watch"| API
    CM -->|"lease + watch"| API

The arrow that matters is the dotted one. The kubelet reads /etc/kubernetes/manifests/, and for every file it finds there it keeps one Pod running. Nothing else supervises the control plane. That single fact determines how you inspect it, how you stop it and how you start it.

k8s-cp-1     control plane, stacked etcd, 192.0.2.11
k8s-w-1      worker
k8s-w-2      worker

workstation
  ~/k8s-cplab/evidence/    everything this lab records

Requirements

A kubeadm cluster on Kubernetes 1.34.x built as in Lab 1, or any kubeadm cluster with stacked etcd — etcd running as a static Pod on the control-plane node. A cluster with external etcd, or a managed cluster (EKS/AKS/GKE), does not expose the files this lab reads; Tasks 2, 7 and 8 have no equivalent there and the lab is not useful on one.
kubectl 1.34.x with a cluster-admin context. Several commands read /metrics and the etcd Pod, which an ordinary namespace-scoped user cannot.
Shell access to the control-plane node with sudo, for crictl and for reading /etc/kubernetes. Roughly half the lab runs there and half from your workstation; each command block says which.
Roughly 60 MB of free disk on the control-plane host for the etcd snapshot in Task 8, plus whatever your etcd database currently occupies.
No out-of-band access requirement. Nothing here reconfigures networking, SSH or the firewall. Task 8 writes one file into /var/lib/etcd and moves it out again; that is the most invasive thing the lab does.

Scenario

You have inherited a cluster. The team that built it has moved on, the wiki page is a year old, and the current complaint is vague: “deployments seem to take ages to actually happen, and sometimes a Pod just sits there.”

kubectl get nodes shows three Ready nodes. That rules out very little: it says three kubelets are renewing their leases. A scheduler that cannot hold a leadership lease, a controller manager that is not reconciling, an etcd whose commits have gone from 20 ms to 800 ms, and an API server five days from certificate expiry are all consistent with three Ready nodes.

This lab is the sweep you run in that situation, before you touch anything. Each task ends with a file in evidence/, because the second half of this job is handing somebody a picture they can act on without your terminal.

Tasks

Task 1 — Build the workspace and record what you are looking at

Run this on your workstation.

mkdir -p "$HOME/k8s-cplab/evidence"
cd "$HOME/k8s-cplab"

CP_NODE="$(kubectl get nodes -l node-role.kubernetes.io/control-plane -o jsonpath='{.items[0].metadata.name}')"
echo "$CP_NODE" | tee evidence/00-cp-node.txt

kubectl version -o yaml            > evidence/00-versions.yaml
kubectl get nodes -o wide          > evidence/00-nodes.txt
kubectl get nodes -o custom-columns='NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion,UNSCHEDULABLE:.spec.unschedulable' \
                                   > evidence/00-nodes-stable.txt
kubectl -n kube-system get pods -o wide > evidence/00-kube-system.txt
kubectl cluster-info               > evidence/00-cluster-info.txt

cat evidence/00-nodes.txt

CP_NODE is derived, not typed. Every subsequent command that names a control-plane Pod builds the name from it, because the static Pod names are kube-apiserver- plus the node name and getting that wrong by hand is the most common way this lab stalls.

00-nodes.txt is for you to read; 00-nodes-stable.txt is the one Cleanup diffs against. The -o wide output carries an AGE column that changes every minute, so a diff against it always reports a difference and therefore proves nothing. The custom-columns projection contains only fields that should be identical before and after a read-only lab.

Task 2 — Find the control plane, and fail to find it with systemd

On the control-plane node, ask systemd about the API server. The answer is the point of the task.

Read-only / Safek8s-cp-1— The expected answer is a failure. Record it.

$ systemctl status kube-apiserver etcd kube-scheduler kube-controller-manager

Unit kube-apiserver.service could not be found.
Unit etcd.service could not be found.
Unit kube-scheduler.service could not be found.
Unit kube-controller-manager.service could not be found.

Illustrative output

Now ask systemd about the one unit that does exist, and then look where the components actually are:

systemctl status kubelet --no-pager | head -12
ls -l /etc/kubernetes/manifests/
sudo crictl ps

Four files, four containers, one systemd unit. The kubelet reads that directory on an interval — fileCheckFrequency, 20 seconds by default — and keeps exactly one Pod running per file. There is no unit to restart, no journalctl -u to read, and no dependency ordering that systemd is enforcing.

Capture the manifests and the API server’s view of the same four Pods:

CP_NODE="$(hostname -s)"
mkdir -p "$HOME/cplab-node"
cd "$HOME/cplab-node"

sudo cp -a /etc/kubernetes/manifests/ ./manifests-copy/
sudo chown -R "$(id -u):$(id -g)" ./manifests-copy/
ls -1 ./manifests-copy/

kubectl -n kube-system get pod \
  "kube-apiserver-$CP_NODE" "etcd-$CP_NODE" \
  "kube-scheduler-$CP_NODE" "kube-controller-manager-$CP_NODE" \
  -o wide

Those four Pods exist in the API but nothing created them through the API. They are mirror Pods: read-only representations the kubelet publishes so that the cluster can see what it is running. Prove it from the object:

CP_NODE="$(hostname -s)"

kubectl -n kube-system get pod "kube-apiserver-$CP_NODE" \
  -o jsonpath='{.metadata.annotations}' | tr ',' '\n'

Look for kubernetes.io/config.source. On an ordinary Pod that annotation is absent; here it says file, and it is the difference between an object you can edit and an object that will be silently rewritten from disk.

Task 3 — Read the API server’s health, at three levels of detail

Back on your workstation. There are three health endpoints and they answer different questions.

cd "$HOME/k8s-cplab"

kubectl get --raw '/livez'   ; echo
kubectl get --raw '/readyz'  ; echo
kubectl get --raw '/healthz' ; echo

Three words, all ok, and almost no information. livez asks whether the process should be restarted. readyz asks whether it should receive traffic — it includes a shutdown check that livez does not, so during a graceful termination readyz fails first and livez stays healthy. healthz is the older combined endpoint, kept for compatibility.

The useful form is the verbose one, which names each check:

Read-only / Safeworkstation— Each line is a separate subsystem check, and one of them can fail alone

$ kubectl get --raw '/readyz?verbose'

[+]ping ok
[+]log ok
[+]etcd ok
[+]etcd-readiness ok
[+]informer-sync ok
[+]poststarthook/start-apiserver-admission-initializer ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]shutdown ok
readyz check passed

Illustrative output

The exact list depends on which feature gates and admission plugins your API server runs, so record yours rather than trusting the one above:

kubectl get --raw '/readyz?verbose' > evidence/03-readyz-verbose.txt
kubectl get --raw '/livez?verbose'  > evidence/03-livez-verbose.txt

grep -c '^\[+\]' evidence/03-readyz-verbose.txt
grep '^\[-\]' evidence/03-readyz-verbose.txt || echo "no failing checks"

A [-] line is the whole diagnosis. [-]etcd failed on an API server that still answers livez means the process is alive and its storage is not — which is a completely different incident from an API server that is down, and one where restarting the API server makes things worse.

Now read the flags the API server is actually running with. They are in the manifest you copied in Task 2, not in a config file the API server re-reads — so this block runs on the control-plane node, not the workstation:

grep -E '  - --(etcd-servers|authorization-mode|max-requests-inflight|max-mutating-requests-inflight|service-cluster-ip-range|audit-log-path|enable-admission-plugins)' \
  "$HOME/cplab-node/manifests-copy/kube-apiserver.yaml" || echo "flag not set; the API server default applies"

Every flag absent from that output is running at its default. --audit-log-path unset means this cluster keeps no audit log, which is worth knowing before an incident rather than during one.

Finally, the certificates. kubeadm-issued client and serving certificates are one-year by default, and an expired API server certificate presents as a total cluster outage with a misleading error.

# On the control-plane node
sudo kubeadm certs check-expiration | tee "$HOME/cplab-node/03-certs.txt"
grep -E 'apiserver|etcd-server' "$HOME/cplab-node/03-certs.txt"

The rows that matter are apiserver, apiserver-kubelet-client and etcd-server: those are one-year certificates that a cluster nobody has upgraded in eleven months is about to be surprised by. The CERTIFICATE AUTHORITY rows below them are ten-year and are not the ones that catch people out.

Task 4 — Find the leaders, and time a lease renewal

The scheduler and the controller manager are leader-elected even when there is one replica. Each holds a Lease object, and the Lease is the authoritative answer to “which process is currently doing this work”.

cd "$HOME/k8s-cplab"

kubectl -n kube-system get lease
kubectl -n kube-system get lease kube-scheduler kube-controller-manager \
  -o custom-columns='NAME:.metadata.name,HOLDER:.spec.holderIdentity,DURATION:.spec.leaseDurationSeconds,RENEWED:.spec.renewTime,TRANSITIONS:.spec.leaseTransitions' \
  | tee evidence/04-leases.txt

Read leaseTransitions first. It counts how many times leadership has changed hands. On a single-replica control plane that has been up since Lab 1 it should be very small; a number in the dozens on a cluster nobody has restarted means the component has been losing its lease, and a component that keeps losing its lease is not reconciling for leaseDurationSeconds at a time, repeatedly.

Measure the renewal interval rather than quoting a default:

for i in 1 2 3 4 5 6; do
  kubectl -n kube-system get lease kube-controller-manager \
    -o jsonpath='{.spec.renewTime}{"\n"}'
  sleep 3
done | tee -a evidence/04-renewals.txt

The gaps between distinct values are the renewal period, and the object’s own leaseDurationSeconds is how long a successor waits before deciding the holder is gone. Those two numbers together are your failover budget: the time between “the leader stopped” and “a replacement is reconciling”.

Node leases live in a different namespace and answer a different question:

kubectl -n kube-node-lease get lease
kubectl -n kube-node-lease get lease -o custom-columns='NODE:.metadata.name,DURATION:.spec.leaseDurationSeconds,RENEWED:.spec.renewTime' \
  | tee -a evidence/04-leases.txt

This is the mechanism behind Ready and NotReady. A kubelet renews its node lease on a short interval; when the node controller sees a lease go stale for longer than its grace period, it marks the node NotReady and starts evicting. kubectl get nodes is a rendering of these objects, which is exactly why it is a kubelet check rather than a control-plane check.

Read the components’ own logs through the API, since there is no journal to read:

CP_NODE="$(cat evidence/00-cp-node.txt)"

kubectl -n kube-system logs "kube-scheduler-$CP_NODE" --tail=40 \
  > evidence/04-scheduler.log
kubectl -n kube-system logs "kube-controller-manager-$CP_NODE" --tail=40 \
  > evidence/04-controller-manager.log

grep -iE 'leaderelection|lease' evidence/04-controller-manager.log | tail -5

Task 5 — Watch a request traverse the pipeline

The API server’s request pipeline is authenticate, authorise, mutate, validate, persist. You can observe the authorisation stage directly, without changing anything, by asking on behalf of another identity.

cd "$HOME/k8s-cplab"

kubectl auth whoami 2>/dev/null || kubectl config view --minify -o jsonpath='{.users[0].name}{"\n"}'

kubectl auth can-i --list > evidence/05-my-permissions.txt
head -12 evidence/05-my-permissions.txt

kubectl auth can-i get pods --as=system:serviceaccount:default:default
kubectl auth can-i list secrets -A --as=system:serviceaccount:default:default

Both impersonated answers should be no. The default ServiceAccount in the default namespace is bound to nothing, and that is a property of the cluster worth confirming rather than assuming — a yes here means somebody has bound a role to system:serviceaccounts and every Pod in the cluster inherited it.

Now watch the persistence stage from the other end. Turn up kubectl’s verbosity so it prints the HTTP exchange:

kubectl -v=6 get namespace kube-system 2>&1 | grep -E 'GET|round trip' | head -5

The line shows the URL, the status code and the round-trip time. That last number is the floor for everything else: kubectl apply cannot be faster than the API server’s response time, and the API server cannot be faster than the etcd commit underneath it.

Task 6 — Create one object and find it in etcd

This is the task that makes the architecture concrete. Create a ConfigMap through the API, then find the key it produced in the database.

cd "$HOME/k8s-cplab"

kubectl create namespace cplab
kubectl -n cplab create configmap lab-marker \
  --from-literal=purpose='control-plane inspection lab' \
  --from-literal=created="$(date -Is)"

kubectl -n cplab get configmap lab-marker \
  -o jsonpath='{.metadata.uid}{"\t"}{.metadata.resourceVersion}{"\n"}' \
  | tee evidence/06-configmap-identity.txt

resourceVersion is the etcd revision at which that object was written. It is not a version number you chose; it is a position in the database’s history, and it is what makes optimistic concurrency work — a write that carries a stale resourceVersion is rejected with a conflict rather than silently overwriting somebody else.

Now go to the control-plane node and ask etcd directly. etcdctl is not installed on the host; it ships inside the etcd image, so run it in the Pod:

CP_NODE="$(hostname -s)"

kubectl -n kube-system exec "etcd-$CP_NODE" -- \
  etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get --prefix --keys-only /registry/configmaps/cplab/

The key is /registry/configmaps/cplab/lab-marker. Every API object follows the same shape: /registry/ plus the resource, plus the namespace for namespaced kinds, plus the name. Count what else is in there:

CP_NODE="$(hostname -s)"

kubectl -n kube-system exec "etcd-$CP_NODE" -- \
  etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get --prefix --keys-only /registry/ | grep -c .

That number is the object count the etcd sizing guidance is about. On a fresh three-node cluster it is in the hundreds; the interesting version of this command is the one you run on a cluster where somebody has a controller creating objects it never deletes.

Try to read the value, and notice that you cannot:

CP_NODE="$(hostname -s)"

kubectl -n kube-system exec "etcd-$CP_NODE" -- \
  etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/configmaps/cplab/lab-marker | head -3

The value is not YAML or JSON. The API server stores objects protobuf-encoded by default, so what comes back begins with a k8s magic prefix and is binary after that. Decoding it needs a tool that understands the Kubernetes protobuf envelope; there is not one in the etcd image, and there does not need to be, because the readable copy is one kubectl get -o yaml away. Record what the first bytes look like and move on — the lesson is that etcd is a byte store that knows nothing about Kubernetes, and every piece of Kubernetes semantics lives in the API server above it.

Task 7 — Establish etcd’s health, capacity and quorum

Still on the control-plane node. Define the flags once so the rest of the task is readable:

CP_NODE="$(hostname -s)"
ETCD_POD="etcd-$CP_NODE"
E_FLAGS="--endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key"

kubectl -n kube-system exec "$ETCD_POD" -- sh -c "etcdctl $E_FLAGS member list -w table"
kubectl -n kube-system exec "$ETCD_POD" -- sh -c "etcdctl $E_FLAGS endpoint health -w table"
kubectl -n kube-system exec "$ETCD_POD" -- sh -c "etcdctl $E_FLAGS endpoint status -w table"

Write the quorum arithmetic for your topology into the evidence directory, in words, before reading on. For the single-member cluster Lab 1 builds: quorum is 1, failure tolerance is 0, and the loss of that member is not a degraded cluster but a stopped one. For a three-member cluster: quorum 2, tolerance 1. For five: quorum 3, tolerance 2 — not 3, which is the arithmetic error that turns a survivable incident into an unsurvivable one.

Now the two numbers that predict a future incident:

CP_NODE="$(hostname -s)"
ETCD_POD="etcd-$CP_NODE"
E_FLAGS="--endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key"

kubectl -n kube-system exec "$ETCD_POD" -- sh -c "etcdctl $E_FLAGS endpoint status -w json"

The JSON carries dbSize and dbSizeInUse. dbSize is the file on disk; dbSizeInUse is the part of it holding live data. They diverge because deleting a key does not shrink the file — the space is freed for reuse inside the database and never returned to the filesystem until a defragmentation rewrites it.

A large gap is the signal that a defrag is due. It is not the signal to run one now: etcdctl defrag rewrites the database file and blocks that member for the duration, so on a single-member cluster it is a write outage and on a multi-member cluster it is done one member at a time in a maintenance window. Record the two numbers and the ratio; the decision belongs to a change window, not to a lab.

Then check for alarms, which is the fastest way to find a cluster that has already hit a limit:

CP_NODE="$(hostname -s)"
ETCD_POD="etcd-$CP_NODE"
E_FLAGS="--endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key"

kubectl -n kube-system exec "$ETCD_POD" -- sh -c "etcdctl $E_FLAGS alarm list"

Empty output is the healthy answer. A NOSPACE alarm means etcd has passed its quota and has put itself into a maintenance mode where it refuses writes — and because every Kubernetes write goes through etcd, that presents as a cluster that can be read and not changed. Clearing the alarm without first reclaiming space puts you straight back into it.

If your etcd manifest exposes a metrics listener, its own metrics are the richest source available:

grep -E 'listen-metrics-urls' "$HOME/cplab-node/manifests-copy/etcd.yaml" \
  || echo "no metrics listener configured on this etcd"

If a URL is printed, curl it from the node and look for etcd_server_has_leader and etcd_disk_wal_fsync_duration_seconds. The first must be 1. The second bounds every write in the cluster: etcd cannot acknowledge a commit faster than it can fsync its write-ahead log, so a slow disk under etcd is a slow kubectl apply for everyone.

Task 8 — Take a snapshot and verify it is real

A control-plane inspection that does not end with a verified backup has skipped the part that matters at 03:00.

Configuration changek8s-cp-1— Reads the live database and writes one file. Does not interrupt writes.

$ kubectl -n kube-system exec etcd-k8s-cp-1 -- etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /var/lib/etcd/cplab-snapshot.db

The snapshot has to be written somewhere the Pod can see; /var/lib/etcd is mounted from the host, so the file lands on the host filesystem. Move it out of the data directory immediately — a file inside the directory you are protecting is not a backup of it — and verify it:

CP_NODE="$(hostname -s)"

sudo mv /var/lib/etcd/cplab-snapshot.db "$HOME/cplab-node/cplab-snapshot.db"
sudo chown "$(id -u):$(id -g)" "$HOME/cplab-node/cplab-snapshot.db"
ls -l "$HOME/cplab-node/cplab-snapshot.db"

Verification is not “the file exists”. Ask etcd what is in it:

CP_NODE="$(hostname -s)"

sudo cp "$HOME/cplab-node/cplab-snapshot.db" /var/lib/etcd/verify.db
kubectl -n kube-system exec "etcd-$CP_NODE" -- \
  etcdutl snapshot status /var/lib/etcd/verify.db -w table
sudo rm -f /var/lib/etcd/verify.db

The table reports a hash, a revision and a key count. The revision should be at or above the resourceVersion you recorded in Task 6, because the ConfigMap was created before the snapshot — that is your proof the snapshot contains the state you think it does, rather than a stale file with a plausible timestamp.

Task 9 — Write the triage note

The deliverable that outlasts the lab. In evidence/triage.md, put the checks in the order you would run them on a cluster that is misbehaving, and for each one write what a clean result rules out — not what it proves.

Five lines is enough. The ordering argument is the content: readyz?verbose is first because it is one request and it distinguishes “the API server is down” from “the API server is up and its storage is not”, which are the two branches everything else hangs off. The lease table is second because it is the only check in the set that sees a control plane that is running, healthy, and not doing any work.

Validation

The lab succeeded when all of the following hold. Run the checks; do not assume them.

cd "$HOME/k8s-cplab"

ls -1 evidence/
grep -c '^\[+\]' evidence/03-readyz-verbose.txt
grep -E 'kube-scheduler|kube-controller-manager' evidence/04-leases.txt
awk 'NF' evidence/04-renewals.txt | sort -u | wc -l
cat evidence/06-configmap-identity.txt

evidence/03-readyz-verbose.txt lists more than one check, and every one is [+]. If any line starts with [-], the lab has found a real problem and that is the thing to chase, not a lab failure.
evidence/04-leases.txt names a holder for both kube-scheduler and kube-controller-manager, and the two holders are the same host.
evidence/04-renewals.txt contains at least two distinct renew times, so you observed a renewal rather than a static field.
You can state, from evidence/04-leases.txt, how long this cluster would go without reconciliation if the controller manager died right now.
evidence/06-configmap-identity.txt has a UID and a resourceVersion, and the key /registry/configmaps/cplab/lab-marker was returned by etcd.
The etcd snapshot’s revision, from etcdutl snapshot status, is greater than or equal to that resourceVersion.
You can state this cluster’s quorum and failure tolerance without looking either up.
evidence/triage.md orders the checks and says what each one rules out.

Expected Outcome

workstation  ~/k8s-cplab/
└── evidence/
    ├── 00-cp-node.txt, 00-versions.yaml, 00-nodes.txt, 00-nodes-stable.txt
    ├── 00-kube-system.txt, 00-cluster-info.txt
    ├── 03-readyz-verbose.txt, 03-livez-verbose.txt
    ├── 04-leases.txt, 04-renewals.txt
    ├── 04-scheduler.log, 04-controller-manager.log
    ├── 05-my-permissions.txt
    ├── 06-configmap-identity.txt
    └── triage.md

k8s-cp-1     ~/cplab-node/
             ├── manifests-copy/{etcd,kube-apiserver,kube-controller-manager,kube-scheduler}.yaml
             ├── 03-certs.txt
             └── cplab-snapshot.db

The cluster itself is exactly as you found it apart from one namespace containing one ConfigMap, which Cleanup removes. No control-plane file was edited, no component was restarted, and no node changed state.

Troubleshooting

kubectl get --raw '/readyz?verbose' returns 403. The health endpoints are readable by system:masters and by the system:monitoring group; an ordinary user is not authorised. Use the admin kubeconfig, or bind the system:monitoring role to your identity deliberately.

kubectl -n kube-system exec etcd-... says “cannot exec into a container in a completed pod” or the Pod name is not found. Rebuild the name from the node: kubectl -n kube-system get pods | grep etcd. The suffix is the node name as the kubelet registered it, which is not always what hostname prints.

etcdctl says “context deadline exceeded”. Either the certificate paths are wrong for your distribution — check the actual paths in manifests-copy/etcd.yaml under --cert-file and --key-file — or the endpoint is not 127.0.0.1:2379. Both are read from the manifest rather than memorised.

etcdutl: not found inside the etcd Pod. Older etcd images ship only etcdctl, where snapshot status is an etcdctl subcommand. Check the image tag in manifests-copy/etcd.yaml and use whichever binary that release ships; the output is the same table.

kubeadm certs check-expiration reports !MISSING! for several entries. Expected on a cluster using external CA mode or an external etcd, where kubeadm does not own those keys. It is a finding to record, not an error.

The lease renewTime never changes across six samples. Either the sampling window was shorter than the renewal period — extend the loop — or the holder has stopped renewing, which is precisely the failure this task exists to detect. Check leaseTransitions and the component’s log.

crictl ps prints a warning about the runtime endpoint. crictl probes a list of sockets when none is configured. Harmless here; setting --runtime-endpoint unix:///run/containerd/containerd.sock silences it.

Cleanup

This lab created one namespace, one ConfigMap, one snapshot file and two working directories. Everything else it did was a read.

Step 1. Confirm what is about to go:

kubectl -n cplab get all,configmap

Step 2. Remove the namespace:

Destructiveworkstation— Removes the lab namespace and the ConfigMap inside it

$ kubectl delete namespace cplab --wait=true

Step 3. Prove the control plane is untouched — this is the real cleanup check, because the risk in this lab was never the ConfigMap:

cd "$HOME/k8s-cplab"

kubectl get namespace cplab || echo "namespace gone, as expected"
diff <(kubectl get nodes -o custom-columns='NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion,UNSCHEDULABLE:.spec.unschedulable') \
     evidence/00-nodes-stable.txt \
  && echo "node inventory unchanged"
kubectl get --raw '/readyz' ; echo

On the control-plane node, confirm the manifest directory still holds exactly four files and that nothing was left behind in the etcd data directory:

ls -1 /etc/kubernetes/manifests/
sudo ls -l /var/lib/etcd/ | grep -E 'cplab-snapshot|verify' \
  || echo "no lab files left in the etcd data directory"

Step 4. Keep the evidence, then remove the working directories.

ls -la "$HOME/k8s-cplab" "$HOME/cplab-node"

mkdir -p "$HOME/k8s-lab-deliverables/control-plane"
cp -a "$HOME/k8s-cplab/evidence" "$HOME/k8s-lab-deliverables/control-plane/"

rm -rf "$HOME/k8s-cplab"

On the control-plane node, once you have decided what to do with the snapshot:

shred -u "$HOME/cplab-node/cplab-snapshot.db" 2>/dev/null \
  || rm -f "$HOME/cplab-node/cplab-snapshot.db"
rm -rf "$HOME/cplab-node"

Production notes

In a real estate this lab is not a change; it is the read-only sweep that precedes one, and it maps onto a change window in three ways.

Before the window. The evidence pack is the “current state” section of the change record. readyz?verbose, the lease table and the etcd status table are the three artefacts that let somebody else judge whether the cluster was healthy when you started — which is the question every post-incident review asks first.

As the go/no-go gate. A control-plane change should not begin on a cluster with a [-] line in readyz, a non-empty etcd alarm list, a certificate inside 30 days, or a leaseTransitions count that has moved since the last sweep. Each of those is a pre-existing fault that will be blamed on your change, and fixing it first is both faster and fairer.

As the rehearsal for the snapshot. Task 8 is the first half of the restore procedure. Running it in calm conditions is how you find out that the data directory is nearly full, or that the snapshot takes four minutes rather than four seconds, at a time when finding out is free.

The honest limit of this lab: a single-member etcd on one control-plane node cannot demonstrate quorum loss, leader change or a defrag under load, because there is no second member and no spare capacity. The arithmetic in Task 7 is the substitute, and it is worth writing out for the topology you actually run rather than the one in front of you.

What You Learned

The control plane has no service manager. Four files in one directory and a kubelet that reads them. systemctl and journalctl -u return nothing, and kubectl delete pod on a mirror Pod restarts nothing.
ok is not a health check. readyz?verbose names the subsystem, and [-]etcd on a live API server is a different incident with a different fix from an API server that is down.
kubectl get nodes is a kubelet check. It reads node leases. Every node can be Ready while the scheduler holds no lease and nothing is being scheduled.
A lease is the only evidence that a component is working rather than running. holderIdentity, leaseDurationSeconds and leaseTransitions answer questions no probe does.
Every object is a key. /registry/configmaps/cplab/lab-marker is where the ConfigMap went, the value is protobuf, and everything that makes it a Kubernetes object lives in the API server rather than in the store.
dbSize and dbSizeInUse diverge on purpose, and the gap is a maintenance-window decision rather than a command to run when you notice it.
A snapshot is verified by its revision, not its existence. The number from etcdutl snapshot status is what tells you the file contains the state you were trying to protect.

Objectives

Prerequisites