Runner authentication failure (runner record or credentials invalid)

Reported symptoms

●GitHub Actions tab shows jobs queued indefinitely; "in_progress" is empty; nothing is running
●The runner, registered the previous day, shows "Offline" in the repo/organisation settings
●The runner disappeared from repository settings after an automated cleanup, or flipped to "Offline" when its local configuration volume was replaced
●The runner service is up; `systemctl status actions.runner.*` shows "active (running)"
●The runner logs at `~/actions-runner/_diag/` show authentication or session failures after startup
●New jobs that should match this runner's labels are stuck in "queued" with no matching runner
●`Actions Runner Controller` (if used) reports `RunnerPod` resources in `CrashLoopBackOff`

Evidence

· `sudo journalctl -u actions.runner.<org>-<repo> --since "24 hours ago"` shows when the runner stopped creating sessions and whether its configuration volume changed at the same time
· `stat ~/actions-runner/.runner ~/actions-runner/.credentials` records whether local configuration was recreated, truncated, or lost; do not print credential contents into an incident log
· `gh api repos/<org>/<repo>/actions/runners --jq '.runners[] | {name,status,busy,labels: [.labels[].name]}'` returns the runner with `status: offline` and `busy: false`; `sudo ./svc.sh status` on the host and the `_diag/Runner_*.log` entries show when its connection attempts started failing
· Repository or organisation audit logs show whether an operator or cleanup job removed the runner record
· If using ARC runner scale sets: inspect the controller, listener, and runner pod events for GitHub App/PAT authentication, registration, or network errors
· No firewall or DNS change occurred; `curl https://api.github.com/zen` from the runner host succeeds

Diagnosis and resolutionclick to reveal

Root cause

The server-side runner record was deleted by cleanup automation, or the runner's supported local configuration was lost when its persistent volume was replaced. The registration token is not a session credential: GitHub issues it for the `config.sh` registration step, and it expires after one hour whether or not the configured runner remains online. Its expiry does not stop an already configured runner from polling. Correlating an offline runner with the age of the original registration token is therefore a false diagnosis. The structural weakness is treating a persistent runner and its undocumented local files as durable state instead of provisioning runners from a supported, repeatable registration flow.

Remediation

Preserve logs and determine whether the server-side record still exists. If the record was removed or local configuration was lost, stop the service, quarantine the stale configuration directory, generate a new one-hour repository registration token, and run the supported unattended command: `./config.sh --url https://github.com/<org>/<repo> --token <registration-token> --name <unique-name> --labels <labels> --unattended`. Reinstall/start the service only after configuration succeeds. Do not hand edit `.runner` or `.credentials`. If the record exists and the local files are intact, investigate TLS interception, DNS, proxy, clock, egress, GitHub availability, or repository/organisation policy before re-registering. For ARC runner scale sets, validate the controller's GitHub App or PAT secret and let the controller create a replacement ephemeral runner; do not patch a runner pod's internal files.

Verification

The runner shows "Online" in GitHub UI and `gh api repos/<org>/<repo>/actions/runners --jq '.runners[] | {name,status}'` reports its documented `status` field as `online`. The runner picks up a test job dispatched with its labels, and the job completes. The runner's `_diag/` log no longer contains 401 entries. For ARC: `kubectl get runners` shows `Phase: Ready` and a healthy listener. The change that re-enabled the runner (supported re-registration, identity repair, or network/policy fix) is captured in the runbook, not in someone's shell history.

Prevention

Do not run long-lived self-hosted runners in production. Use ephemeral runners that register fresh on every job — Actions Runner Controller for Kubernetes, ARC runners on EKS/GKE/AKS, or HashiCorp Nomad / Terraform Cloud agents for non-Kubernetes environments. Each ephemeral runner uses a short-lived registration authorization and a clean runtime, so stale local identity is not treated as permanent infrastructure. If long-lived runners are unavoidable (an air-gapped network, a hardware dependency that cannot be containerised), rebuild and re-register them through a documented supported procedure; do not rotate registration tokens or patch runner config on a timer. Alert on offline status or missed job pickup as signals that the runner needs diagnosis.

A registration token is not a runner session credential. It is a one-time proof of authority for config.sh and expires after one hour without taking an already configured runner offline. Diagnose the runner record, supported local configuration, network path, and organisation policy; never schedule registration-token rotation as an availability control.