Does the risk matrix actually work?

Open any risk register and you will find it: the matrix of likelihood times impact, usually five by five, with green, yellow and red cells. The risk matrix is the most widely used instrument in risk management, from safety engineering to accountancy and from healthcare to information security. Kantyra, too, has one at the heart of its risk module. That is precisely why I have to be honest about something few vendors say out loud: the scientific literature is remarkably critical of this instrument. In this article I place that criticism alongside daily practice and show how, despite its limitations, you can work with a matrix responsibly.

This article builds on the scientific basis of the model, in which I substantiate the four phases of Detect, Assess, Resolve and Demonstrate. The risk matrix belongs to the Assess phase, and it deserves the same honest treatment.

What does the science say is wrong with the matrix?

The most fundamental criticism comes from Tony Cox, an American risk scientist who devoted two articles to it, one with colleagues in 2005 and one on his own in 2008, that have since become classics. His analysis in Risk Analysis, the leading journal of the field, is mathematical in nature and can be summarised in three findings.

First, a matrix has coarse discriminating power. Because likelihood and impact are pressed into a handful of categories, risks that differ strongly in numerical terms receive the same colour, and a matrix can only say correctly which of two randomly chosen risks is larger in a limited share of cases. Second, a matrix makes demonstrable ranking errors: there are situations in which a risk that is numerically smaller still lands in a higher cell than a larger risk. Third, and this is the harshest conclusion, for risks where likelihood and impact are negatively correlated, that is, where the rare events are precisely the severe ones, a matrix performs in the worst case no better than random guessing. Cox uses the now famous phrase that the matrix is then "worse than useless".

Researchers from the oil and gas sector later sharpened this. Thomas, Bratvold and Bickel showed in 2014 that the design choices of the matrix itself, such as the number of categories, the division of the scales and the scores attached to the cells, also change the final ranking of risks profoundly. Two organisations that assess exactly the same risks but use a different matrix layout arrive at a different priority list, without anyone having made a mistake. The ranking that comes out is thereby partly a consequence of the instrument rather than of reality.

Why do two assessors land so far apart on the same risk?

The second line of criticism is not about the mathematics but about the human filling in the cells. David Ball and John Watt investigated in 2013 how different assessors place the same hazard on a matrix. The scatter turned out to be large, and did not disappear after reflection and explanation either. Moreover, the differences turned out not to stem from carelessness, but from diverging worldviews and convictions that rarely reach the table during a risk session.

On top of that comes a language problem. Terms such as "likely", "rare" and "considerable" mean something different to everyone. David Budescu and colleagues had test subjects translate probability expressions from the reports of the IPCC, the climate panel of the United Nations, into numbers. Even with the official definitions alongside, the interpretations diverged strongly. Saying "likely" may evoke thirty percent in one person and ninety in another. Every risk matrix that works with such words inherits this problem.

And beneath all of that lies the oldest insight of this field. Amos Tversky and Daniel Kahneman described as early as 1974 how people estimate probabilities using rules of thumb that produce systematic errors of thought: we overestimate what is recent or vivid, we anchor on the first number that comes along and we are structurally too sure of our own judgement. A risk session in which experts pick cells without further support is susceptible to all of those effects.

Does that write off the matrix?

No, and that is not my judgement but that of the same literature. Nijs Jan Duijm, affiliated with the Technical University of Denmark, published a balanced review of the weaknesses in 2015, together with concrete recommendations for design and use. His core message: the matrix is usable as an instrument for prioritising and for structuring the conversation about risks, as long as you do not treat it as a measuring instrument and you define the categories carefully and anchor them in numbers.

More recent experimental research from the Winton Centre in Cambridge points in the same direction. Holly Sutherland and colleagues tested in randomised experiments with almost two thousand seven hundred participants how people understand matrices. A matrix turned out not to be self-evidently better than plain text for conveying risk information, and the design mattered greatly: for scales that do not increase linearly, labelling that makes this explicit helped considerably. A follow-up study by Rossa Proto and colleagues from 2023 examined the colours themselves and found that coloured bands distort judgement: participants attached more value to risk reductions that crossed a colour boundary than to equally large reductions within the same colour. The border between yellow and orange then starts to weigh in as if it were a property of the risk, while it is a property of the picture.

The sum of it: the matrix is not a measuring instrument and must not be used as one, but as a structured form of conversation and prioritisation it has value, provided it is well designed and its users know its limitations.

What is the alternative?

The scientific critics almost all advocate the same alternative: calculate with numbers instead of with cells. Douglas Hubbard and Richard Seiersen worked this out furthest for information security. They show that even with little data you can express probabilities and losses in ranges, that you can train experts to make calibrated estimates, and that even a simple quantitative model removes most of the distortion of the matrix.

For an organisation with its own risk team and sufficient data, that is the better road. But here the SME research from my earlier article returns: for most organisations without a full-time security department, a fully quantitative approach is not feasible at this moment, and a poorly executed quantification creates more false certainty than a well-understood matrix. So it is not a choice between pure and sinful. The realistic route is a well-designed matrix for the breadth of the register, with quantitative deepening for the few risks on which the biggest decisions depend.

How do you work with it responsibly in practice?

Five habits follow from the literature that make the difference between a matrix that misleads and a matrix that helps.

1. Anchor every category in numbers. "Likely" then becomes, for example, "more often than once a year", and "severe" gets a range in euros or recovery time. That largely removes Budescu's language problem and forces assessors towards the same frame of reference.
2. Never assess alone. The scatter that Ball and Watt found does not disappear, but it becomes manageable as soon as differences in judgement are discussed explicitly. Two assessors who land on different cells have something to explain to each other, and precisely that conversation yields the insights.
3. Record the reasoning, not just the score. A cell without a rationale is worth nothing a year from now. The assumptions behind likelihood and impact are what an auditor, a successor or your own future self needs in order to test the judgement.
4. Use the matrix to prioritise, never to prove that a risk is small. Cox's ranking errors sit mainly in the comparison of individual cells. For the question of which ten risks deserve attention first, the matrix is usable; for the conclusion that one specific risk is negligible, it is too coarse.
5. Reassess on signals rather than on the calendar. An incident, a failed exercise or a worrying assessment says more about the shelf life of a judgement than the passing of twelve months. This is the cycle from the model: what you detect feeds the next assessment.

For completeness: this is also how Kantyra is set up. The scales of the matrix can be defined per organisation, so that you can anchor them in numbers. Every risk asks for a rationale alongside the score, serious incidents that touch a risk automatically queue up a reassessment, and the trail of earlier judgements is preserved. The matrix itself thereby remains what the literature allows it to be: an instrument for prioritising and for conversation, embedded in a working method that absorbs its weaknesses.

What does this mean for your organisation?

If you take one thing from this article, let it be this: do not distrust the matrix, distrust a bare cell. A score without a numerical definition, without a second assessor, without a rationale and without reassessment when new signals arrive is exactly the symbolic risk management the literature warns against. The same matrix, embedded in those four habits, is a defensible and workable instrument, even in front of a critical auditor who has read Cox.

So ask yourself three questions. Are the categories of your matrix defined numerically anywhere, or does "likely" mean something different to everyone in your organisation? Can you retrieve the reasoning behind the judgement for your five biggest risks? And when was that judgement last tested against what has actually happened since? Whoever can answer these three questions uses the matrix the way the science permits. Whoever cannot does not have a matrix problem but an assessment problem, and that, fortunately, can be solved.

Accountability and sources

This article is a substantiated synthesis of existing scientific research and not independent, peer-reviewed research of its own. Where I interpret or explain, that interpretation is my own responsibility.

1. Cox, L.A. (2008). What's Wrong with Risk Matrices? Risk Analysis, 28(2). https://doi.org/10.1111/j.1539-6924.2008.01030.x
2. Cox, L.A., Babayev, D. & Huber, W. (2005). Some Limitations of Qualitative Risk Rating Systems. Risk Analysis, 25(3). https://doi.org/10.1111/j.1539-6924.2005.00615.x
3. Thomas, P., Bratvold, R.B. & Bickel, J.E. (2014). The Risk of Using Risk Matrices. SPE Economics & Management, 6(2). https://doi.org/10.2118/166269-PA
4. Ball, D.J. & Watt, J. (2013). Further Thoughts on the Utility of Risk Matrices. Risk Analysis, 33(11). https://doi.org/10.1111/risa.12057
5. Duijm, N.J. (2015). Recommendations on the use and design of risk matrices. Safety Science, 76. https://doi.org/10.1016/j.ssci.2015.02.014
6. Sutherland, H., Recchia, G., Dryhurst, S. & Freeman, A.L.J. (2022). How People Understand Risk Matrices, and How Matrix Design Can Improve their Use: Findings from Randomized Controlled Studies. Risk Analysis, 42(5). https://doi.org/10.1111/risa.13822
7. Proto, R. and others (2023). Do colored cells in risk matrices affect decision-making and risk perception? Insights from randomized controlled studies. Risk Analysis. https://doi.org/10.1111/risa.14091
8. Budescu, D.V., Broomell, S.B. & Por, H.-H. (2009). Improving Communication of Uncertainty in the Reports of the Intergovernmental Panel on Climate Change. Psychological Science, 20(3). https://doi.org/10.1111/j.1467-9280.2009.02284.x
9. Tversky, A. & Kahneman, D. (1974). Judgment under Uncertainty: Heuristics and Biases. Science, 185(4157). https://doi.org/10.1126/science.185.4157.1124
10. Hubbard, D.W. & Seiersen, R. (2023). How to Measure Anything in Cybersecurity Risk (second edition). Wiley. https://doi.org/10.1002/9781119892335