Recovery time objectives — what RTO is acceptable for a Git hosting failure

Recovery Time Objective is a design constraint, not an incident decision. The team that decides what RTO is acceptable for a Git hosting failure has decided, before the incident, which fallback tier (mirror, multi-remote failover, self-hosted fallback, local bare) is worth the operational cost. The RTO is the cost ceiling; the tier is the recovery path that fits within the cost.

flowchart LR
    A["team RTO"] --> B["workflow\npush / fetch / review / deploy"]
    B --> C["tier choice"]
    C --> D["mirror-only\nminutes"]
    C --> E["multi-remote failover\ntens of minutes"]
    C --> F["self-hosted fallback\nhours"]
    C --> G["local bare\nhours-to-days"]
    D --> H["operational cost"]
    E --> H
    F --> H
    G --> H
    H --> I["rehearsal cadence"]

The two numbers: RTO and RPO

The two numbers every DR plan names are the Recovery Time Objective and the Recovery Point Objective.

• RTO (Recovery Time Objective): how long the team accepts the system being down. For a Git hosting failure, this is the elapsed time from the outage to the moment the team can push, fetch, review, and deploy.
• RPO (Recovery Point Objective): how much data the team accepts losing. For a Git hosting failure, this is the gap between the last commit on the fallback and the disaster time.

A team that has decided on an RTO of fifteen minutes has decided that fifteen minutes is the maximum the team can operate without pushing, fetching, reviewing, and deploying through the canonical remote. A team with an RTO of four hours has decided the canonical is not on the critical path for four hours; an outage of the canonical for the first four hours is absorbed by the existing buffer.

# The RTO and RPO are written in the runbook before the incident
cat runbook/git-hosting-failure.md
# RTO: 30 minutes (push, fetch, review, deploy through the fallback)
# RPO: 5 minutes (push-mirror period from hosted to self-hosted)
# Tier: self-hosted fallback (Gitea in second region)

The four workflows that drive the RTO

A team uses the canonical remote for four workflows; each has a different RTO tolerance. Push has the highest tolerance - teams absorb a four-hour outage by batching commits. Fetch has the lowest - CI and GitOps controllers cannot build or reconcile without a fetch. Review has a high tolerance - delays merges but does not block deploys. Deploy has the lowest tolerance when GitOps is in use.

The workflow with the lowest RTO tolerance drives the team’s overall RTO. A team that runs GitOps controllers against the canonical has an RTO dictated by the controller’s reconciliation period (typically three minutes), not the engineer’s tolerance.

The four tiers and their RTOs

The four tiers map to RTOs from minutes to hours. The mirror-only tier (Part XCVIII-03) serves the read path in minutes; the write path is unavailable until the canonical recovers. The multi-remote failover tier (Part XCVIII-02) serves the write path in tens of minutes: the failover flip runs when the trigger is exceeded. The self-hosted fallback tier (Part XCVIII-05) serves the full forge in hours: push, fetch, review, and CI integration are restored. The local bare tier (Part XCVIII-04) serves the write path in hours-to-days: engineers push to a bare repo with no web UI, no pull-request workflow, no CI integration.

The cost of each tier

The cost of each tier is operational, not financial. The mirror-only tier costs a few hours of setup plus a cron job and quarterly rehearsal. The multi-remote failover adds the script’s maintenance and the quarterly drill. The self-hosted fallback adds a full forge, migration, patching, and CI integration - a few days of bootstrap plus ongoing maintenance. The local bare costs only a directory and SSH.

The cost scales with the RTO: a team that wants a fifteen-minute RTO for the full workflow accepts the cost of the self-hosted fallback; a team that accepts a four-hour RTO for the write path uses the multi-remote failover.

Rehearsal cadence

The cadence of rehearsal scales with the tier. The mirror-only and multi-remote failover tiers run quarterly: verify the mirror is current, run the failover flip in a non-production environment, measure the elapsed time. The self-hosted fallback runs quarterly with a full migration drill: provision a non-production fallback, migrate a sample repository, run the failover flip, measure the timeline, run the reverse push-back. The local bare runs annually: bootstrap from a clone, push, verify SSH access, decommission.

A drill that has not been run is a hypothesis. The cadence is the minimum that keeps the runbook fresh.

Production discipline

1. The RTO and RPO are written in the runbook before the incident. A runbook that names the numbers is a runbook that the on-call engineer can execute.
2. The workflow with the lowest RTO tolerance drives the team’s overall RTO. A team that runs GitOps against the canonical has an RTO dictated by the controller, not the engineer.
3. The tier matches the RTO. Mirror-only for read-path RTOs, multi-remote failover for write-path RTOs of tens of minutes, self-hosted fallback for full-workflow RTOs of hours, local bare for CLI-only RTOs of hours-to-days.
4. The cost of each tier is documented. A team that does not know the operational cost of a tier is a team that cannot choose between tiers.
5. The rehearsal cadence matches the tier. Mirror-only and multi-remote failover are quarterly; self-hosted fallback is quarterly with a full migration drill; local bare is annually.

Cross-course references

• Git, CI/CD & GitOps — Parts XCVIII-02 through XCVIII-05 cover the four tiers of fallback this lesson compares.
• Git, CI/CD & GitOps — Part XCVII-06 (The Disaster Recovery Drill) covers the rehearsal cadence this lesson assumes.
• Terraform for Production Sysadmins — Part XXI (State Backup) covers the analogous RTO/RPO discipline for Terraform state.

Quiz