Credential boundaries — where the secrets live in each model

Every deployment model is a credential arrangement. The push model and the pull model both need secrets; they put them in different places. This lesson is the inventory - a list of every credential each model requires, where it lives, how long it lives, and what its compromise enables. Two systems running the same model are distinguished in production by the discipline applied to these credentials.

The credential inventory

Five kinds of credentials appear in a Kubernetes-shaped deployment, and each model places them differently:

flowchart LR
    subgraph Push["Push model - credentials"]
        P1["CI runner holds: kubeconfig, cloud creds, registry creds, Git R/W"]
        P2["Cluster holds: ServiceAccount for kubelet, no controller"]
        P3["Git holds: deploy key for CI only"]
    end
    subgraph Pull["Pull model - credentials"]
        Q1["CI runner holds: Git R/W"]
        Q2["Cluster holds: ServiceAccount for controller, Git R/O deploy key"]
        Q3["Git holds: deploy keys for both CI and controller"]
    end

The push model concentrates the secrets at the runner. The pull model concentrates the cluster-side secrets at the controller and the Git-side secrets at the runner. The trade is between number of secrets and blast radius if a holder is compromised.

Cluster credentials

Push model. A kubeconfig with write access to the cluster API lives in the CI runner’s environment. The kubeconfig is typically a ServiceAccount token (or its IRSA/Workload-Identity equivalent) issued by the cluster. Its lifetime is short (OIDC-federated tokens are typically one hour) but its holder is outside the cluster.

Pull model. A ServiceAccount token with write access to the cluster API lives in the controller pod’s filesystem (or is mounted as a projected token). Its holder is inside the cluster. Its scope is a tight Role or ClusterRole - cluster-admin is a placeholder, never a production setting.

Git credentials

Push model. The CI runner holds a Git token (PAT, deploy key, or GitHub App installation token) with read-write scope to the manifests repository. The runner needs read-write because it may push status checks, comments, and (in some pipelines) tag updates back to the repo.

Pull model. Two Git tokens exist:

• The CI runner holds a token with read-write to the repository, same as the push model. The runner still needs write to push commits and status checks.
• The controller pod holds a token with read-only scope: a deploy key, a fine-grained PAT, or a credential from a Git proxy. The controller only reads.

The asymmetry is deliberate. The CI runner is a less-trusted holder than the controller, so its credential is the more constrained of the two in the push direction.

Cloud-provider credentials

Both models sometimes need credentials against the cloud provider’s control plane (AWS IAM, GCP IAM, Azure AD) - typically because manifests reference IAM roles, KMS keys, or service meshes that live outside Kubernetes.

Push model. These credentials live on the CI runner. They are usually short-lived (IRSA-style federation from the runner to AWS, Workload Identity Federation to GCP) but they are push credentials: the holder can issue API calls against the cloud from outside the cluster.

Pull model. The controller may need cloud credentials too, especially if it manages cloud-side resources via a CloudControllerManager or a CRD-driven provider. These credentials live in the controller pod, scoped to the resources the controller manages. IRSA on EKS, Workload Identity on GKE: the cloud knows the credential belongs to a specific ServiceAccount in a specific namespace.

flowchart LR
    SA["ServiceAccount in cluster"] -->|"federated"| IRSA["IAM role"]
    IRSA -->|"STS token"| AWS["AWS APIs"]

The federation step is the operational difference. A credential issued this way cannot be reused from outside the cluster because the STS token is bound to the pod’s identity.

Registry credentials

Push model. The CI runner pulls base images during build. It needs a registry token with read access to the base image registry and write access to the target registry. The token is typically stored as a Secret in CI.

Pull model. The controller does not pull images; kubelet does, and kubelet’s credential story is independent of the GitOps model. Registry credentials are not part of the GitOps control plane in the pull model - they are part of the kubelet’s runtime credentials.

Production discipline

1. Inventory the credentials. Every pipeline should have a written list of which credentials exist, where they live, who can read them, and how they rotate. A credential that is not in the inventory is a credential that does not get rotated.
2. Scope every credential to what its holder actually needs. Role not ClusterRole; read-only deploy keys where the controller only reads; fine-grained tokens scoped to a single repository where the runner only touches one.
3. Rotate on a schedule and on every incident. OIDC-federated tokens rotate by design; long-lived tokens must rotate on a schedule, and any credential involved in an incident must rotate immediately, regardless of schedule.

Cross-course references

• Kubernetes for Production Sysadmins - Parts on RBAC, ServiceAccounts, and IRSA cover the cluster-credential side of both models.
• Terraform for Production Sysadmins - Parts on Terraform Cloud and the operator pattern cover credential models for infrastructure provisioning.
• Linux for Production Sysadmins - Parts on secrets management cover the operational mechanics of credential rotation.

Quiz