Backup strategy — what to protect

The Kubernetes backup strategy is the discipline of protecting the cluster’s data. The 3-2-1 backup rule, the what-to-protect, the recovery time are the components. This lesson walks the backup strategy, the components, the recovery time, and the production patterns.

What to protect

The Kubernetes cluster has many components:

flowchart LR
    A[Cluster] --> B[etcd]
    A --> C[PV data]
    A --> D[Secrets]
    A --> E[Certificates]
    A --> F[Manifests in Git]
    A --> G[External dependencies]

The components are the backup targets.

The etcd backup

The etcd backup:

sudo ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  --endpoints=https://127.0.0.1:2379 \
  snapshot save /var/backups/etcd-$(date +%Y%m%d-%H%M%S).db

The etcd backup is the cluster state.

The persistent data backup

The persistent data backup:

# Velero backup
velero backup create my-backup --include-namespaces=default

# Or use the cloud provider's snapshot
# AWS EBS snapshot
aws ec2 create-snapshot --volume-id vol-12345

# GCP persistent disk snapshot
gcloud compute disks snapshot my-pd

# Azure disk snapshot
az snapshot create --resource-group my-rg --name my-snapshot

The persistent data is the application data.

The certificates backup

The certificates backup:

# Backup the PKI directory
sudo tar -czf /var/backups/pki-$(date +%Y%m%d).tar.gz /etc/kubernetes/pki/

# Or use Velero
velero backup create my-pki --include-resources=secrets

The certificates are the cluster’s identity.

The secrets backup

The secrets backup:

# Backup the secrets
kubectl get secrets -A -o yaml > /var/backups/secrets-$(date +%Y%m%d).yaml

# Or use Velero
velero backup create my-secrets --include-resources=secrets

The secrets are the cluster’s credentials.

The external dependencies

The external dependencies:

- External databases (PostgreSQL, MySQL)
- External object storage (S3, GCS)
- External message queues (Kafka, RabbitMQ)
- External DNS (Route53, Cloud DNS)
- External certificate authorities

The external dependencies are the cluster’s integrations.

The 3-2-1 backup rule

The 3-2-1 backup rule:

- 3 copies of the data
- 2 different storage media
- 1 offsite (different physical location)

The 3-2-1 rule is the production standard.

The RTO and RPO

The RTO and RPO:

- RTO (Recovery Time Objective): how long can the cluster be down?
- RPO (Recovery Point Objective): how much data can be lost?

The RTO and RPO are the recovery targets.

The backup schedule

The backup schedule:

# Daily etcd backup
0 2 * * * /usr/local/bin/etcd-backup.sh

# Weekly persistent data backup
0 3 * * 0 /usr/local/bin/velero-backup.sh

# Monthly certificates backup
0 4 1 * * /usr/local/bin/cert-backup.sh

The backup schedule is the operational discipline.

The production patterns

The production patterns:

flowchart LR
    A[etcd] --> B[Daily backup]
    C[PV data] --> D[Velero backup]
    E[Secrets] --> F[Velero backup]
    G[Certs] --> H[Backup]
    B --> I[3-2-1 rule]
    D --> I
    F --> I
    H --> I
    I --> J[DR site]

The pattern is the production discipline.

The cross-course references

• The etcd course covers the etcd backup.
• The Velero course covers the persistent data backup.
• The DR course covers the recovery strategy.

Quiz

Production discipline

• Identify what to protect. The etcd, the PV data, the secrets, the certificates.
• Apply the 3-2-1 rule. 3 copies, 2 media, 1 offsite.
• Define the RTO/RPO. The recovery targets.
• Schedule the backups. Daily etcd, weekly PV.
• Encrypt the backups. KMS for the secrets.
• Document the backup strategy. The components, the schedule.

The backup strategy is the cluster’s data protection. Operating it well is the what-to-protect, the 3-2-1 rule, the RTO/RPO, and the production patterns.