The scoreboard problem
Somewhere along the way, AI usage became the scoreboard. Tokens consumed, lines generated, suggestions accepted, seats deployed — the numbers go up and to the right, and everyone feels productive. Engineers compare token counts; vendors publish them; leaders put adoption charts in board decks. The activity has become the achievement. Call it tokenmaxxing.
It is a textbook case of Goodhart's law: when a measure becomes a target, it stops being a good measure. Usage is trivially easy to count, and outcomes are hard to attribute, so the easy number wins — and quietly stops meaning anything.
Why usage looks like progress
Generation is immediate and visible. Value is lagging and distributed. When an AI tool emits two hundred lines, you see two hundred lines; you do not see the hour spent reviewing them, the half that gets thrown away, or the maintenance burden the survivors add for the next two years. The work that converts generation into value is invisible on a usage dashboard — which is exactly why usage can hit records while delivery flatlines.
There is nothing wrong with high usage. It simply is not evidence of anything except spend. Tokens are an input and a cost. Treating them as a result is like a logistics company bragging about how much diesel it burned.
Four substitutions
For every vanity metric, there is an outcome metric that measures what you actually wanted. Swap them:
| Stop measuring (vanity) | Start measuring (outcome) |
|---|---|
| Tokens consumed / API spend | Value shipped per dollar of AI spend |
| Lines of AI code generated | Change-failure rate / defect escape rate |
| Suggestions accepted / acceptance rate | Cycle time: idea to production |
| Seats deployed / % adoption | Share of shipped value traceable to AI |
The right-hand column is harder to produce. That difficulty is the point — it is the difference between reporting that you are busy and reporting that you are effective. Several of these (delivery throughput, lead time, change-failure rate) are the long-standing DORA delivery metrics, and they apply to AI-assisted work exactly as they do to any other.
Net the claim
The honest way to value AI is to start with the claimed productivity gain and subtract the realities that erode it:
- Not all generated output ships — some is thrown away.
- What ships costs time to review and rework.
- What survives adds a maintenance tax for as long as it lives.
- A fraction of unreviewed output causes incidents, which have a cost.
- And the tooling itself has a price.
Run a claimed thirty-percent gain through those discounts and the realized figure is often a fraction of the headline — still positive, frequently worth it, but nowhere near the brag. The gap between claimed and realized is not pessimism; it is the part of the story tokenmaxxing leaves out. Knowing its size is what lets you decide whether the next tranche of seats is an investment or a subsidy for throwaway code.
The board slide
Replace the adoption chart with four numbers a board can actually use:
- Delivery throughput / cycle time — are we shipping faster?
- Change-failure rate — are we shipping safely?
- Net realized value vs. spend — is the ROI real after rework and maintenance?
- Maintenance load of AI-authored code — what are we borrowing against the future?
None of these can be gamed by burning more tokens. That is precisely why they belong on the slide.
Try the math
The AI ROI Reality Check turns this into a model: enter your spend and the gain someone is claiming, apply the reality discounts, and see the net realized value, the gap, and the metric swap — a defensible number to bring to the conversation instead of a token count.