Finalizers — blocking deletion until cleanup completes

Finalizers are how Kubernetes tells controllers “this object is being deleted, but don’t actually remove it until I tell you the cleanup is done”. This lesson covers the mechanics, common finalizer patterns, and how to debug stuck deletions.

What finalizers do

When the operator runs kubectl delete, the API server does not immediately delete the object. Instead:

1. The API server sets metadata.deletionTimestamp on the object.
2. The object persists in etcd with deletionTimestamp set.
3. Controllers see the deletionTimestamp and run their cleanup logic (each controller for its own finalizer).
4. Each controller, when its cleanup is done, removes its finalizer from metadata.finalizers.
5. When metadata.finalizers is empty, the API server actually deletes the object.

sequenceDiagram
    autonumber
    participant U as Operator
    participant API as API server
    participant C1 as Controller A
    participant C2 as Controller B
    participant ETCD as etcd

    U->>API: DELETE /pod/web-abc
    API->>API: set metadata.deletionTimestamp
    API->>API: ensure metadata.finalizers preserved
    API->>ETCD: persist (deletionTimestamp set, finalizers kept)
    API-->>U: 200 OK (object still exists, marked for deletion)
    Note over C1,C2: controllers observe deletionTimestamp
    C1->>API: cleanup complete, remove my finalizer
    C2->>API: cleanup complete, remove my finalizer
    Note over API: finalizers array now empty
    API->>ETCD: actually delete the object

The object exists in Terminating state (visible via kubectl get) until all finalizers are removed.

Common built-in finalizers

kubernetes.io/pv-protection

Present on PersistentVolumes. Prevents the PV from being deleted while it is bound to a PVC. Removed by the PV controller when the PVC is deleted.

kubectl get pv pvc-001 -o jsonpath='{.metadata.finalizers}'
# ["kubernetes.io/pv-protection"]

kubernetes.io/pvc-protection

Present on PersistentVolumeClaims. Prevents the PVC from being deleted while in use by a Pod. Removed by the PVC controller when no Pod is using it.

finalizers.kubernetes.io/finalizer.kube-system

Used by kube-system controllers (e.g., the namespace controller) to manage cleanup of dependent resources.

custom finalizers

Application-specific finalizers set by custom controllers (Operators). The convention:

metadata:
  finalizers:
  - example.com/cleanup-database

When the object is deleted, the custom controller sees the finalizer, runs its cleanup logic, and removes the finalizer.

Finalizers vs owner references

These are related but distinct:

Owner references cascade delete across objects. Finalizers prevent deletion of a single object until cleanup completes.

A controller might use both:

1. The controller owns children (via ownerReferences).
2. The controller sets a finalizer on the parent.
3. When the parent is deleted, the controller sees the finalizer, runs cleanup, removes the finalizer.
4. Once removed, the parent is deleted.
5. The GC then cascades to the children.

Custom finalizers in Operators

A typical Operator pattern:

const databaseFinalizer = "example.com/cleanup-database"

func (r *ReconcileDatabase) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    obj := &Database{}
    if err := r.Get(ctx, req.NamespacedName, obj); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Add the finalizer if not present
    if !containsString(obj.Finalizers, databaseFinalizer) {
        obj.Finalizers = append(obj.Finalizers, databaseFinalizer)
        return ctrl.Result{}, r.Update(ctx, obj)
    }

    // Handle deletion
    if obj.DeletionTimestamp != nil {
        if containsString(obj.Finalizers, databaseFinalizer) {
            // Run cleanup
            if err := r.cleanupDatabase(obj); err != nil {
                return ctrl.Result{}, err
            }
            // Remove the finalizer
            obj.Finalizers = removeString(obj.Finalizers, databaseFinalizer)
            return ctrl.Result{}, r.Update(ctx, obj)
        }
        return ctrl.Result{}, nil
    }

    // Reconcile normally
    return r.reconcileNormal(ctx, obj)
}

The pattern:

1. On creation, add the finalizer if not present.
2. On deletion (deletionTimestamp set), run cleanup, then remove the finalizer.
3. On normal reconcile, do the controller’s work.

Production Operators follow this pattern for any external resource cleanup (database drop, DNS record removal, cloud resource deletion).

How to debug stuck deletions

A common production problem: an object is stuck in Terminating. The cause is almost always an unremoved finalizer.

$ kubectl get pod web-abc
NAME        READY   STATUS        RESTARTS   AGE
web-abc     1/1     Terminating   0          30m

$ kubectl get pod web-abc -o jsonpath='{.metadata.finalizers}' | jq
[
  "example.com/cleanup-database"
]

The example.com/cleanup-database finalizer has not been removed. The custom controller that should remove it is not running, or its cleanup failed.

Diagnostic steps:

# 1. List the finalizers
kubectl get pod web-abc -o jsonpath='{.metadata.finalizers}'

# 2. Check if the responsible controller is running
kubectl get pods -n operator-system

# The controller Pod named by step 2 above:
CONTROLLER_POD=operator-controller-manager-6d8f7c9b54-r7k4d

# 3. Check the controller's logs
kubectl logs -n operator-system "$CONTROLLER_POD" --tail=100

# 4. Check if the cleanup action failed
kubectl describe pod web-abc | grep -A 10 "Finalizers:"

How to remove a finalizer manually

If the controller is permanently gone and you need to force the deletion, you can remove the finalizer manually:

# Patch the finalizers array to empty
kubectl patch pod web-abc -p '{"metadata":{"finalizers":[]}}' --type=merge

# Or, more targeted:
kubectl patch pod web-abc -p '{"metadata":{"finalizers":["only-the-ones-you-want"]}}' --type=merge

After removing the finalizer, the API server actually deletes the object.

Common production scenarios

Scenario 1: namespace stuck in Terminating

$ kubectl get namespace team-a-prod
NAME           STATUS        AGE
team-a-prod    Terminating   1h

$ kubectl get namespace team-a-prod -o jsonpath='{.spec.finalizers}'
[
  "kubernetes"
]

The kubernetes finalizer is part of the namespace controller’s cleanup. If the controller cannot remove all objects in the namespace (e.g., a stuck PVC), the namespace remains in Terminating.

Fix: investigate what’s blocking cleanup.

kubectl get all -n team-a-prod
kubectl get pvc -n team-a-prod

Scenario 2: PVC stuck in Terminating

$ kubectl get pvc data-db-0
NAME          STATUS        AGE
data-db-0    Terminating   30m

$ kubectl get pvc data-db-0 -o jsonpath='{.metadata.finalizers}'
[
  "kubernetes.io/pvc-protection"
]

The PVC controller is waiting for the Pod using the PVC to be deleted. Check:

kubectl get pods -o jsonpath='{.items[?(@.spec.volumes[?(@.persistentVolumeClaim.claimName=="data-db-0")])].metadata.name}'

If a Pod is still using the PVC, delete it (or wait for the controller to evict it).

Scenario 3: Custom CRD object stuck

$ kubectl get postgrescluster prod
NAME    STATUS        AGE
prod    Terminating   45m

$ kubectl get postgrescluster prod -o jsonpath='{.metadata.finalizers}'
[
  "postgres.example.com/cleanup"
]

The custom controller has not cleaned up. Check the controller’s logs and cluster status.

Finalizers and dry-run

kubectl delete --dry-run=server runs the full API server pipeline (authn, authz, admission) without persisting. It shows what would happen, including finalizer behaviour:

kubectl delete postgrescluster prod --dry-run=server -o yaml

Production: use this to verify a deletion will behave as expected before committing.

Cross-course references

• The Linux course part XXIX-Linux-Hardening covers least-privilege principles that map onto finalizer discipline.
• The Observability course part CIX-Observability-InvestigationWorkflows covers investigation methodology for finalizer-stuck deletions.
• The Linux course part XXIV-Linux-Time covers chrony — timestamps in deletion depend on clock accuracy.
• The Docker course part XXXVIII-Docker-Secrets covers secret handling; finalizers are not for secrets.

Quiz

Production discipline

• Treat finalizers as the controller’s contract: every finalizer must be removed when its cleanup is complete.
• Audit objects in Terminating state regularly. A stuck finalizer is a sign of a broken controller or external dependency.
• Document external resources held by each finalizer: if the controller is gone, what cleanup must happen manually?
• Test deletion paths in staging, including finalizer cleanup. A stuck deletion in production is often discovered only when the operator runs kubectl delete.
• Remove finalizers manually only as a last resort and with full awareness that controller-side cleanup is bypassed.