Idempotency and the safe retry — why retry only works if the operation is idempotent

The Kubernetes API uses a verb called replace for an update that should succeed whether the resource currently matches the desired state or not. The verb update is more strict - it expects the resource to match the previous version. The difference is idempotency. replace is idempotent because applying it twice produces the same state as applying it once; update is not idempotent because the second application will fail the version check. The same discipline applies to every operation the pipeline retries: idempotency is what makes a retry safe.

The mathematical definition, in operational terms

Mathematically, an operation is idempotent when f(f(x)) = f(x). In the pipeline, the operational translation is:

Re-running the operation against an already-completed state leaves the state unchanged.

A deploy that creates a resource by kubectl apply -f is idempotent: the second apply against an existing resource matches the desired state and leaves it unchanged. A deploy that creates a resource by kubectl create -f is not idempotent: the second create fails with already exists. The difference is the verb.

flowchart LR
    A["Operation f"] --> B{"Is f idempotent?"}
    B -- "Yes: f(f(x)) = f(x)" --> C["Safe to retry freely"]
    B -- "No: f(f(x)) != f(x)" --> D["Retry is unsafe"]
    D --> E["Use checkpoint OR manual retry"]

Three categories of operations are usually non-idempotent, and recognising them is the first step in making the pipeline safely retryable:

• Append. An operation that adds to a list, queue, or log. Re-running the operation appends again.
• Create-if-absent. An operation that creates a resource only when no resource with the same name exists. Re-running when the resource exists produces an error rather than leaving the state unchanged.
• Double-send. An operation that sends a message, request, or notification. Re-running sends the message twice.

Each non-idempotent operation has an idempotent equivalent that the pipeline should use instead.

The safe-retry pattern

A safe retry requires three things:

1. The operation is idempotent. f(f(x)) = f(x) holds for the operation as the pipeline calls it.
2. The retry has a bound. The pipeline does not retry forever; the retries stop at a per-job maximum.
3. The retry has a backoff. The retries are spaced using exponential backoff with jitter (LXI-02).

A retry with all three is a safe retry. A retry missing any one of the three is an unsafe retry, even if the other two are present. The pattern:

jobs:
  apply-idempotent:
    retries: 3
    steps:
      - run: ./apply.sh  # idempotent by design
  apply-non-idempotent:
    retries: 0
    steps:
      - run: ./apply.sh  # uses checkpoint for resumability

The first job is safe to retry because the operation is idempotent. The second job is not safe to retry at the platform level because the operation is not idempotent; the resumability comes from the checkpoint pattern of LXI-04.

Recognising a non-idempotent operation

The categories of non-idempotent operation have recognisable shapes in pipeline code:

Each non-idempotent operation has an idempotent equivalent that costs the same to run, performs the same action, and is safe to retry. The pipeline that uses the idempotent equivalent at every step is a pipeline that can be safely retried by the platform’s automatic retry policy.

Non-idempotency in the common deploy tools

The deploy tools in this course have their own non-idempotency patterns:

• Terraform. terraform apply is idempotent because the state file records what was applied. terraform import is not idempotent because importing an existing resource twice produces a duplicate-state error. The discipline is to use import once and rely on apply after that.
• Ansible. Ansible playbooks are nominally idempotent because tasks run with changed_when checks. A task that uses command: instead of a module is not idempotent because the task runs every time and the operator cannot tell whether anything changed. The discipline is to prefer Ansible modules over raw command:.
• Helm. helm install is not idempotent because the second install fails with already exists. helm upgrade --install is idempotent because the upgrade applies to the existing release. The discipline is to use --install so the chart can be re-applied.
• kubectl. kubectl apply is idempotent because the server-side apply matches the desired state. kubectl create is not. The discipline is to use apply for every manifest the pipeline re-runs.

Each tool has its own idempotent verb. The pipeline that uses the idempotent verb is the pipeline that can be safely retried by the platform.

Idempotency keys for external systems

External systems - payment APIs, notification services, third-party APIs - often have their own idempotency support. Stripe’s API accepts an Idempotency-Key header; the API deduplicates retries that use the same key. AWS SQS supports message deduplication IDs on FIFO queues. The pattern across these systems is the same:

IDEMPOTENCY_KEY="$(uuidgen)"
curl -X POST "$API_URL/payments" \
    -H "Idempotency-Key: $IDEMPOTENCY_KEY" \
    -d @payment.json

The key is a UUID generated once per logical operation and reused on every retry. The receiver deduplicates the retries that carry the same key. The pipeline that generates the key once and uses it across retries is a pipeline that treats idempotency as a contract with the external system, not just as a property of the local operation.

Production discipline

1. Idempotency first, retry second. Enable automatic retry only after every retried operation is idempotent by construction.
2. Use the idempotent verb in every deploy tool. apply for kubectl, --install for Helm, modules for Ansible, apply for Terraform.
3. Generate idempotency keys once per logical operation. The key is a UUID; the receiver deduplicates by key.
4. Distinguish safe retries from unsafe retries in the workflow file. The same retry policy does not apply to both.
5. Treat non-idempotency as a code smell in the pipeline. A non-idempotent operation in a retried job is a state-leak waiting for the next failure.

Cross-course references

• This course, Part LXI-04 (PartialDeployment) covers how idempotency enables the resumable retry pattern.
• This course, Part LXI-02 (Retries) covers the backoff and budget that bound the retry.
• Linux for Production Sysadmins - Part XXXIII (ServiceReliability) covers the platform-level idempotency patterns.

Quiz