The idea behind it
Most risk registers rate each risk High, Medium or Low. Those labels hide disagreement, since one person's Medium can mean a one-in-ten chance and another's an even bet. They also cannot be added up or compared with the cost of a control. A calibrated register replaces the labels with two numbers: the probability the risk occurs in the period, as a percentage, and the cost if it does, in dollars. Multiplying them gives expected value, which ranks risks on one scale and shows whether a mitigation costs more than the loss it avoids.
The numbers are only useful if the people estimating them are calibrated. A calibrated estimator who says 70% is right about 70% of the time. Douglas Hubbard, in How to Measure Anything, showed that calibration is a skill people can train with practice questions and feedback. Philip Tetlock's forecasting research reached a similar conclusion. Several techniques help. Reference classes, an approach associated with Daniel Kahneman and Bent Flyvbjerg, start from what happened to similar projects rather than from the specifics of this one. Estimating before hearing others avoids anchoring. Ranges are more honest than single points. Logging estimates against outcomes shows each estimator whether they are overconfident.
What's inside
- A register sheet: risk ID, category, description, probability in percent, impact in dollars, owner, review date, mitigation, residual probability, residual impact and status.
- Expected value calculates as probability times impact, and risk rank orders the risks by it. Residual expected value and the reduction each mitigation buys calculate from the residual estimates.
- A sample register with ten IT risks sorted by expected value.
- A calibration primer with five true-or-false practice questions, a confidence estimate for each and reference answers with their sources.
- Five tips for better probability estimates, such as using reference classes and logging estimates against outcomes.
Format: Excel workbook.
Who uses it
- CIOs and CISOs, to rank technology risks by expected dollar loss instead of by color.
- Enterprise risk managers, to move an IT register from qualitative ratings to figures that compare with other business risks.
- Program and project directors, to set contingency and decide which mitigations are worth funding.
- Finance partners, to test whether a proposed control costs more than the loss it is expected to prevent.
When to use it
- A risk register has too many High ratings to prioritize.
- The board or audit committee asks how large the top technology risks are in dollars.
- A mitigation budget request has to be justified against the loss it avoids.
- A risk workshop is planned and participants have never estimated probabilities.
- The organization is moving toward cyber risk quantification.
How to use it
Replace the High/Medium/Low ratings in an existing register, then rank the risks by expected value.
- Run the Calibration Primer first: have each participant enter an answer and a confidence for the five practice questions before reading the reference answers.
- In the Risk Register sheet, enter each risk with an ID, category, description, owner and review date.
- Enter Probability as a number from 0 to 100 and Impact in dollars; Expected Value calculates in column F and Risk rank in column G.
- Sort the register by Expected Value, largest first, to get the ranked list, as in the Sample sheet.
- Record the mitigation or control, then estimate residual probability and residual impact after it takes effect; residual expected value and the reduction calculate.
- Set Status to Open, Mitigated, Accepted or Closed, and log estimates against what actually happened at each review.
You leave with: A register ranked by expected dollar loss, with residual estimates that show what each mitigation is expected to buy.
Where it goes wrong
- Entering probability as a decimal such as 0.3 instead of 30; the formula divides by 100.
- Using 50% to mean 'I don't know' rather than 'equally likely either way'.
- Anchoring on the first estimate spoken in the workshop instead of collecting estimates independently.
- Never comparing estimates with outcomes, so overconfidence goes uncorrected.
Get the register
It is free. Enter your work email and we send the download link.
Questions
What is a calibrated risk register?
It is a risk register that records each risk as a probability and a dollar impact instead of High, Medium or Low. Expected value, probability times impact, ranks the risks on one scale. The estimates come from people trained to give probabilities that match outcomes.
What does calibrated mean in probability estimation?
An estimator is calibrated when their stated probabilities match how often things happen. Events they call 70% likely occur about 70% of the time. Douglas Hubbard showed calibration can be trained with practice questions and feedback.
How do you calculate expected value for a risk?
Multiply the probability of the risk occurring, as a fraction, by its impact in dollars. The result is the average loss the risk represents over the period. Ranking risks by expected value puts likely, costly risks at the top.