Time sync, DNS, certificates — foundations and observability stack

Time sync, DNS, certificates, and observability are the operational foundations of every Kubernetes cluster. Each is a discipline in itself; this lesson walks the configuration of each and the production discipline of keeping them healthy.

Time sync — chrony / NTP

Every node must have accurate time. The reasons:

• Certificate validity. TLS cert validation requires accurate time; skew > minutes breaks.
• etcd leadership. Etcd leader election is time-bounded; skew can trigger spurious elections.
• Kubelet registration. Cert time skew causes CSR failures.
• Logging. Logs are timestamped; skew makes logs incomparable.

The chrony install

sudo dnf install -y chrony
sudo systemctl enable --now chronyd

Configuration at /etc/chrony.conf:

# Use the cluster's internal NTP server
server ntp1.internal.example iburst
server ntp2.internal.example iburst
server ntp3.internal.example iburst

# Allow large time corrections at startup
makestep 1.0 3

# Synchronise with NTP sources that have low jitter
rtcsync
logdir /var/log/chrony

# Verify
sudo chronyc tracking

Reference ID    : C0A80101 (ntp1.internal.example)
System time     : 0.000000234 seconds fast of NTP time
Last offset     : -0.000012345 seconds
RMS offset      : 0.000023456 seconds
Frequency       : 12.345 ppm slow
Residual freq   : 0.001 ppm
Skew            : 0.123 ppm
Root delay      : 0.001234 seconds
Root dispersion : 0.002345 seconds
Update interval : 64.2 seconds
Leap status     : Normal

The clock-skew check

# On every node, check the time drift
date; sudo chronyc tracking

Drift between nodes must be < 1 second; ideally sub-second.

DNS — CoreDNS and upstream

CoreDNS serves the cluster’s DNS:

sequenceDiagram
    participant Pod
    participant CDNS as CoreDNS
    participant UD as Upstream DNS
    Pod->>CDNS: query billing.prod-app.svc.cluster.local
    CDNS->>CDNS: lookup service / endpoint
    CDNS-->>Pod: 10.96.0.10
    Pod->>CDNS: query api.example.com
    CDNS->>UD: forward
    UD-->>CDNS: response
    CDNS-->>Pod: response

CoreDNS is a Deployment that registers with the kubelet as the cluster DNS (via --cluster-dns flag).

The CoreDNS configuration

The Corefile (ConfigMap in kube-system/coredns):

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

The configuration:

• errors — log errors.
• kubernetes plugin — watch Services and Pods.
• forward plugin — forward non-cluster queries upstream.
• cache plugin — cache responses for 30 seconds.
• loop plugin — detect forwarding loops.
• reload plugin — pick up Corefile changes.

$ kubectl -n kube-system get cm coredns -o yaml | head -30

...

The nodelocal DNS cache

For high-DNS-rate clusters:

# Install nodelocal DNS cache
kubectl apply -f https://github.com/kubernetes/kubernetes/raw/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml

Each node runs a DNS cache as a DaemonSet; Pods query the local cache (less latency than central CoreDNS).

flowchart LR
    Pod --> NL[nodelocaldns]
    NL -->|cache miss| CoreDNS
    NL -->|cache hit| Pod

Latency drops ~10x; load on CoreDNS drops.

Certificates — cert-manager

cert-manager automates certificate management:

flowchart LR
    Cert[cert-manager] -->|watch| CM[Certificate CRDs]
    CM -->|ACME / Cert request| ACME[Let's Encrypt]
    ACME -->|issue| CM
    CM -->|apply to Secret| API[API server]

# Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.x/cert-manager.yaml

The cert-manager issuers and certificates

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
    - http01:
        ingress:
          class: nginx

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-example-com
  namespace: my-namespace
spec:
  secretName: api-example-com-tls
  dnsNames:
  - api.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

cert-manager handles:

• ACME flow (Let’s Encrypt).
• DNS-01 / HTTP-01 challenges.
• Renewal (30 days before expiry).
• Storage in Kubernetes Secrets.

The observability stack

Production clusters typically run:

The kube-prometheus-stack (Helm):

# Install
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace

This installs Prometheus, Alertmanager, Grafana, and the kube-state-metrics exporter.

The kube-state-metrics

kube-state-metrics exposes object-state metrics:

• Deployment replicas (vs available).
• StatefulSet replicas.
• Pod conditions (Ready, Initialized).
• Node conditions (Ready, MemoryPressure, DiskPressure).
• Lease objects.

These are different from host / resource metrics; they describe K8s object state.

The metrics to alert on

Production clusters alert on (from kube-prometheus-stack defaults):

• NodeNotReady. Node Ready=False for >5 min.
• EtcdNoLeader. No etcd leader.
• KubeAPIDown. API server unreachable.
• KubeControllerManagerDown. Controller manager unreachable.
• KubeSchedulerDown. Scheduler unreachable.
• KubeDeploymentReplicasMismatch. Deployment has fewer available than desired replicas.
• KubeNodePressure. MemoryPressure / DiskPressure / PIDPressure.

These are the high-signal alerts that bind to the cluster’s heartbeat.

The summary checklist

FOUNDATION CHECKLIST
====================

Time sync:
  [ ] chrony / NTP installed on every node
  [ ] Stratum-correct (NTP source not too far)
  [ ] Monitoring enabled

DNS:
  [ ] CoreDNS Deployment in kube-system
  [ ] Pod's resolv.conf points at CoreDNS (default)
  [ ] nodelocal DNS for high-rate workloads
  [ ] Upstream DNS forwarder configured

Certificates:
  [ ] cert-manager installed
  [ ] ClusterIssuers for staging / production
  [ ] Certificates issued via cert-manager
  [ ] Renewal handled automatically

Observability:
  [ ] Prometheus + Alertmanager
  [ ] Grafana dashboards
  [ ] kube-state-metrics
  [ ] Loki / Tempo as needed
  [ ] Custom alerts for SLOs

The discipline

• Time sync with monitoring. chrony with monitoring; alert on drift.
• DNS redundancy. CoreDNS HA via multiple replicas; nodelocal for cache.
• Cert-manager for application TLS. No manual certs.
• Observability stack from day one. Production clusters need alerting.
• Document the foundations. Runbook entries for each; team understanding matters.

Quiz

Production discipline

• Foundations before features. Time sync first; DNS next; certs and observability on top.
• Monitor the foundations. chrony’s drift, CoreDNS’s error rate, cert-manager’s reconciliation, Prometheus’s health.
• Document every choice. Each foundation has alternatives; the chosen one must be recorded.
• Rehearse the recovery. What happens if chrony fails? What if CoreDNS is down? The answers should be in the runbook.
• Prefer standard tools. kube-prometheus-stack, cert-manager, CoreDNS — battle-tested defaults.

The foundations are the cluster’s basement. Operating them well is the floor under everything else.