Observability without employee ranking — the ethical boundary and what metrics actually measure

A CI dashboard is supposed to answer the question “how is the system doing?”. The dashboard is not supposed to answer the question “how is each engineer doing?”. The two questions look similar; they produce different metrics; they produce different operational decisions. A team that measures the system improves the system. A team that measures the individuals harms the individuals and, in the process, degrades the system.

What the metrics measure

The DORA metrics - deployment frequency, lead time, change failure rate, MTTR - measure the pipeline as a system. The metrics describe the team’s collective output and the pipeline’s collective behaviour. The metrics do not attribute behaviour to individuals:

• Deployment frequency is a team-level count. It does not name the engineers whose deploys are counted.
• Lead time for changes is a team-level distribution. It does not name the engineers whose commits contribute to the distribution.
• Change failure rate is a team-level ratio. It does not name the engineers whose changes caused the failures.
• MTTR is a team-level median. It does not name the engineers who responded to the incident.

The team-level framing is intentional. DORA’s research found that high-performing teams have the practices that produce the metrics; the metrics are the result of the practices, not the driver of the practices. A team that treats the metrics as targets (deploy faster, ship more) without investing in the practices is a team that optimises the metric and degrades the system.

flowchart LR
    A["Practices\nsmall changes, automated tests, fast feedback"] --> B["System metrics\nDORA four keys"]
    B --> C["Healthy team\nhigh throughput, high stability"]
    A -. "do not use for ranking" .-> D["Per-engineer metrics\ncommits, builds, deploys"]
    D --> E["Degraded team\ngaming, hiding, burnout"]

The misuse pattern

The misuse pattern is straightforward and common. A team that has a dashboard of team-level metrics decides to “drill down” into per-engineer metrics. The drill-down looks like:

• Commits per engineer per week.
• Pull requests opened per engineer per week.
• Deploys triggered per engineer per week.
• Build failures attributed to the engineer whose commit triggered the build.
• Review response time per engineer.

Each of these metrics, surfaced to a manager and used for performance evaluation, produces a predictable failure mode. The engineer who fixes the flaky test is the engineer whose build failures are high; the engineer who reviews carefully is the engineer whose review response time is slow; the engineer who automates the toil is the engineer whose commits are low. The metrics punish the practices that produce healthy teams.

The structural patterns that produce healthy teams

The DORA research found that high-performing teams share four structural patterns:

1. Small changes. The team deploys in small increments; the review, test, and deploy cost of each change is low.
2. Fast feedback. The team gets feedback on each change quickly; the lead time from commit to deploy is short.
3. Automated tests. The team has automated tests that catch regressions before production; the change failure rate is low.
4. Rehearsed recovery. The team has runbooks and automated rollback; the MTTR is short.

None of the four patterns is an individual metric. None requires ranking engineers to achieve. The patterns are system properties that emerge from the team’s collective investment in tooling, processes, and culture.

flowchart TB
    A["Small changes"] --> D["High deployment frequency"]
    B["Fast feedback"] --> E["Low lead time"]
    C["Automated tests"] --> F["Low change failure rate"]
    G["Rehearsed recovery"] --> H["Low MTTR"]
    D --> I["High-performing team"]
    E --> I
    F --> I
    G --> I

The patterns are the investment. The metrics are the result. A team that invests in the patterns gets the metrics as a side effect. A team that targets the metrics without investing in the patterns is a team that will game the metrics and degrade the system.

Designing dashboards that improve the system

A team-level dashboard that improves the system without ranking the individuals follows three principles:

1. Show the team-level metric, not the individual breakdown. The dashboard shows “deployment frequency: 3 per day”; it does not show “engineer A: 5 deploys, engineer B: 1 deploy”.
2. Show the system property, not the activity count. The dashboard shows “test coverage: 85%”; it does not show “commits per engineer”. The first metric measures a system property that predicts stability; the second measures an activity that does not.
3. Show the trend, not the rank. The dashboard shows “median lead time: trending down 20% over 6 months”; it does not show “engineer ranking by lead time”. The first is a diagnostic; the second is a ranking.

The dashboard’s purpose is to surface the friction the team can address. The friction is usually structural (review backlog, test queue, deploy serialisation) and is rarely individual. A dashboard that shows the friction points the team at the structural fix; a dashboard that shows the individual breakdowns points the team at the individual - and the individual cannot fix the structural problem alone.

Production discipline

1. Never surface per-engineer metrics for performance evaluation. The metric is for the system; the individual is not the unit of measurement.
2. Surface team-level metrics as trends, not rankings. The trend is the diagnostic; the ranking is the misuse.
3. Surface the friction points, not the activity counts. Review backlog, test queue, and deploy queue are friction points; commits and builds are activity counts.
4. Invest in the practices (small changes, fast feedback, automated tests, rehearsed recovery). The metrics are the result; the practices are the investment.
5. Audit the dashboard for misuse patterns. A dashboard that shows per-engineer breakdowns is a dashboard that will be misused; remove the breakdowns before the misuse happens.

Cross-course references

• Observability course - Part I (Foundations) and Part XII (OrgHealth) cover the ethical boundary of observability: metrics measure systems, not people.
• This course, Part XXXI (CodeOwners) covers the code-ownership discipline that affects review response time; the discipline is about the system, not the individual.
• This course, Part LVII (Approvals) covers the approval discipline that affects lead time; the discipline is about the system, not the individual.

Quiz