The idea behind it
Goodhart's law holds that a measure used as a target stops being a good measure. The economist Charles Goodhart stated it in 1975 about monetary policy, and the anthropologist Marilyn Strathern gave it its popular wording. The social scientist Donald Campbell made a similar observation: the more a quantitative indicator drives decisions, the more it is open to corruption. Steven Kerr's paper "On the Folly of Rewarding A, While Hoping for B" showed that organizations routinely pay for one behavior while asking for another. In IT the pattern shows up as ticket counts, velocity, uptime targets and cost KPIs that teams learn to hit without improving the work behind them.
A pre-mortem is a planning exercise developed by the psychologist Gary Klein. The team imagines the plan has already failed and works backward to explain why. Treating the failure as a given makes people more willing to name problems they would otherwise keep to themselves. Applied to an incentive, the pre-mortem assumes the metric is in place and being gamed, then asks how, by whom and at whose expense. It also looks past the first effect to second-order consequences: rivalry between teams, harm to customers, and short-term results bought with long-term health. It ends with safeguards, such as a paired metric and a review date, and a decision to ship, revise or drop the metric.
What's inside
- A header for the metric under review: owner, intended behavior, measurement method, reporting frequency, the consequence of hitting or missing it, and the workshop date and facilitator.
- Section 1, gaming prediction: four prompts, each with an example of how a real metric gets gamed.
- Section 2, winners, losers and power: four questions on who benefits, who is penalized, who can game it and who has no voice.
- Section 3, a second-order effects grid across cross-team, customer, short-term versus long-term, and measurement cost.
- Section 4, a checklist of five safeguards, such as a complementary metric and a sunset date.
- A workshop outcome: ship, revise or abandon, with a 6-month review date and a metric owner.
- The Excel version holds every section, with dropdowns for the safeguards and the decision, and fills in the review date 6 months after the workshop.
Format: PDF (3 pages) and Excel workbook.
A look inside
Pages 1 and 2 of 3 of the PDF. Select an image to enlarge it.
Who uses it
- CIOs and IT leadership teams, to test a new KPI or scorecard before it goes into performance reviews.
- HR and reward leads, to find the behaviors a bonus scheme will reward by accident.
- Engineering and service managers, to see how a delivery or incident metric will be gamed by the teams it measures.
- Transformation office and PMO leads, to review OKRs or program metrics before funding is tied to them.
When to use it
- A new KPI is about to be added to team or individual performance reviews.
- A bonus scheme for IT staff, or incentive payments for a vendor or outsourcer, is being drafted.
- A service contract is being negotiated with SLA credits tied to metrics.
- Leadership asks for a single headline number, such as velocity or tickets closed, to compare teams.
- An existing metric shows every team green while users report problems.
How to use it
Run it as a 90-minute workshop with the people the metric will affect, before the metric or bonus scheme is announced.
- Before the workshop, fill in the header: the metric's name, who proposed it, the behavior it should drive, how it is measured, how often it is reported, and what happens when it is hit or missed.
- In Section 1, spend 5 to 10 minutes on each of the four gaming prompts and list as many specific scenarios as possible. The sheet's examples, such as teams splitting stories to raise velocity, show the level of detail.
- In Section 2, answer who benefits, who is disadvantaged, who controls the inputs and who had no voice in the design.
- In Section 3, work through the second-order effects grid: cross-team effects, customer effects, the short-term versus long-term trade-off and the cost of measuring.
- In Section 4, mark each of the five safeguards Yes, No or Partial and name an owner for each.
- Close with the Workshop Outcome: ship as designed, revise or abandon, the changes required, a 6-month review date and the metric owner.
You leave with: A list of the ways the metric is likely to be gamed and its side effects, the safeguards agreed against them, and a recorded decision with an owner and a review date.
Where it goes wrong
- Inviting only the people who designed the metric. The teams it measures know the gaming routes best, and Section 2 asks whose voice is missing.
- Stopping at the obvious gaming scenarios. Section 1 asks for quantity because the obvious ones are already known.
- Shipping the metric with no complementary measure, so the most likely gaming route goes unchecked.
- Setting no end date. The safeguards checklist asks for a sunset review after 6 months.
Get the facilitation worksheet
It is free. Enter your work email and we send the download link.
Questions
What is Goodhart's law?
Goodhart's law says that when a measure becomes a target, it ceases to be a good measure. People optimize the number, and the number drifts away from what it was meant to track. Charles Goodhart, an economist at the Bank of England, first stated a version of it in 1975.
What is a pre-mortem?
A pre-mortem is a session held before a plan launches in which the team assumes the plan has failed and lists the reasons. Gary Klein developed it to surface risks people are reluctant to raise in a normal review. For a metric or bonus scheme, the imagined failure is that people hit the number without delivering the value.
How do you stop a KPI from being gamed?
Pair it with a complementary metric that catches the most likely gaming route, such as a quality measure next to a throughput measure. Publish the definition, name an owner who watches for gaming, and set a date to review it. Allow a named decision-maker to override the number with a documented reason.