Self-hosted forge and CI control-plane lifecycle

A self-hosted Git or CI platform becomes the path through which every other production change travels. Calling it “internal tooling” does not reduce its availability, integrity, confidentiality, or recovery requirements. If it is unavailable, teams cannot review or ship. If it loses integrity, attackers can change source, workflows, credentials, or artifacts. If it cannot be restored, the organization has lost part of its operational memory.

Model the complete state graph

The visible web process is only one node:

The inventory must name the owner, version, location, backup method, restore order, monitoring, and dependency for every row. A VM snapshot of the web node is not a platform backup.

Choose a platform from operating requirements

GitLab integrates repository hosting, pipelines, registry, packages, and many supporting services, which increases both capability and recovery scope. Jenkins is primarily an automation controller whose plugins and controller state are part of its supply chain. Forgejo and Gitea provide a smaller forge surface, while CI commonly comes from Actions-compatible runners or another controller. The correct choice depends on:

• required source, review, package, registry, and CI features;
• user and repository scale, concurrency, artifact volume, and growth;
• database/object-storage operations already supported by the organization;
• SSO, MFA, audit, retention, and compliance needs;
• supported backup, restore, upgrade, and HA procedures;
• plugin/action ecosystem risk and update cadence; and
• the number of operators capable of restoring it at 03:00.

A product with more features but no practiced restore owner is less available than a smaller platform the team can rebuild from evidence.

Backup consistency and restore order

A recoverable set must capture repository storage, database, object storage, configuration, and secrets at a mutually consistent recovery point. Product backup utilities coordinate some of these boundaries; storage snapshots alone may not. Record the platform version and configuration beside every backup.

The restore plan normally proceeds:

1. Establish clean hosts, DNS isolation, time, storage, and certificates.
2. Install the exact compatible platform version without exposing it to users.
3. Restore database and repository/object state according to vendor order.
4. Restore secrets and encryption keys through an independent recovery path.
5. Validate repository fsck/sample clones, permissions, protected branches, approvals, artifacts, package/registry objects, audit events, and webhooks.
6. Reconnect SSO, SMTP, runners, integrations, mirrors, and public ingress.
7. Re-enable writes only after reconciliation and acceptance tests pass.

Upgrade and rollback boundaries

Before an upgrade, read the supported upgrade path and intermediate-version requirements; verify OS, database, object-store, runner, plugin, and API compatibility; restore the current backup into isolation; drain writes and background jobs as required; record configuration; and establish abort criteria.

Database migrations create the rollback boundary. After an irreversible migration, rolling application binaries back while keeping the new database may be unsupported. The safe rollback may be a full restore of database, repositories, object storage, configuration, and secrets to the pre-upgrade recovery point. State this before starting, including the RPO cost of writes accepted after the migration.

Plugins, actions, and extensions are executable dependencies. Pin them, review their permissions and maintainers, stage their upgrades, inventory their configuration, and keep the controller able to start with non-essential extensions disabled.

HA is dependency-specific

Two web replicas behind a load balancer are not an HA platform when they share one database, filesystem, Redis node, object store, identity provider, or failure domain. For each stateful dependency define:

• failure unit and quorum/consistency model;
• placement across power, host, rack, zone, or site boundaries;
• client failover and connection behavior;
• backup independent of replication;
• maintenance sequence and capacity with one member unavailable; and
• the alert that proves redundancy has already been consumed.

Capacity plans track repository count/size, LFS and artifact growth, database size and latency, object count, concurrent jobs, queue depth, clone/push rate, webhook throughput, runner wait time, backup duration, restore duration, and headroom during node loss or maintenance.

Observability and operating cadence

Monitor user journeys, not just processes: authenticate, list a repository, clone/fetch, create a test ref, enqueue a pipeline, allocate a runner, upload and retrieve an artifact, deliver a webhook, and send mail. Alert separately on availability, integrity, capacity, security, and recoverability.

The operating cadence includes certificate and secret expiry, failed backups, restore age, database/storage growth, queue backlog, runner saturation, pending upgrades, vulnerable plugins/actions, audit-log export, emergency-account use, and dependency redundancy. A quarterly isolated restore is the evidence gate; “backup job succeeded” is not.

Phase-two execution contract

Phase two chooses one platform and proves install, TLS, SSO/emergency access, SMTP, repository and artifact operation, backup, isolated restore, supported upgrade, rollback boundary, monitoring, and teardown. Until that evidence is captured, this lesson describes the operating model but does not claim a tested platform lifecycle.

Knowledge check