Kantyra
Kantyra
Alain Rees · 10-07-2026 · 6 min leestijd
Every security report is full of them: the number of patched systems, the number of completed trainings, the number of handled alerts. All counts of activity, and all of them answer a question nobody asked. The question the board, the regulator and the auditor do ask, namely whether the organisation is becoming more secure, remains unanswered in most dashboards. This article is about the difference between counting and knowing, and about the small set of indicators you can start with.
This article belongs to the Demonstrate phase of the model behind Kantyra. Whoever wants to demonstrate that controls work needs indicators that actually say so.
The scientific literature on security metrics begins with an uncomfortable observation. Vilhelm Verendel combed through all quantitative security research from 1981 to 2008 and concluded that the assumption that security can be correctly captured in numbers is largely unproven: most proposed indicators have never been empirically tested. An indicator is, in other words, a hypothesis, not a fact, until someone has shown that it correlates with what you want to know.
Shari Lawrence Pfleeger and Robert Cunningham explained why this field is so unruly. Security is a negative property: you measure the absence of events, and the adversary adapts to whatever you measure. Whoever reports the number of blocked attacks mainly measures the attacks that never stood a chance anyway. Research from Delft University of Technology, from Michel van Eeten's group that measures security empirically using real abuse data, adds a practical lesson: raw counts mislead as long as you do not correct for size and exposure. In their analysis of web compromises at hosting providers, only after statistical normalisation did it become clear who was really performing well or poorly. For your dashboard this means: ten incidents across a thousand systems is something different from ten across a hundred, and a rising count can mean deterioration as well as better visibility.
There is a second pitfall, and it is behavioural. The anthropologist Marilyn Strathern coined the now famous formulation of Goodhart's law: as soon as a measure becomes a target, it ceases to be a good measure. Whoever judges teams on "ninety-five percent of findings closed within thirty days" gets closed findings, but not necessarily solved problems. Ross Anderson and Tyler Moore showed in their influential overview of security economics that this is not an accident but a structure: security failure stems more often from misaligned incentives than from technology. An indicator that ignores the incentives measures the wrong thing and, worse, steers in the wrong direction.
One criterion keeps surfacing in the literature: an indicator is good if a decision is attached to it. Rainer Böhme connected metrics to investment decisions and showed that many popular figures cannot answer a single investment question. The test is simple: if the number suddenly doubles or halves next month, who does anything differently? If the answer is nobody, it is not an indicator but wallpaper.
The American standards institute NIST completely revised its guideline for security measures, publication 800-55, at the end of 2024, and the direction of that revision confirms this picture. The new version emphasises choosing and prioritising a small set of measures tied to the organisation's goals, explicitly recognises qualitative assessment as a full equal of quantitative measurement, and devotes an entire second volume to the measurement programme around it: owners, decision moments and data management. The ISO world already asks for this too: ISO 27001 obliges organisations in clause 9.1 to evaluate the effectiveness of the system, and the accompanying guideline ISO 27004 explicitly distinguishes measuring performance from measuring effectiveness. Whoever only counts activity does not even meet the standard on their own wall.
The temptation is to solve this by measuring more. The research points the other way. The systematic review by Max van Haastrecht and colleagues (Utrecht University and Leiden), which I also cite in the base article, shows that the supply of indicators is mainly technical and activity-oriented, while organisations without their own measurement team need few, comprehensible and context-adaptable indicators. Bilge Yigit Ozkan and Marco Spruit developed this into an adaptable maturity assessment for SMEs: start light, measure what fits your organisation and grow along.
Translated into practice, this is the starter set I recommend to every organisation, three questions per key control: does it exist, does it work, and has someone recently reviewed that? It sounds almost embarrassingly simple, but these are exactly the questions most organisations cannot answer, and they are qualitative judgements that the revised NIST guideline treats as fully valid. On top of that fit a handful of outcome indicators that touch real decisions: the time between signal and treatment, the share of controls with a passed effectiveness test, the number of open exceptions past their end date, and the reporting speed for phishing that I wrote about in the awareness article. Kantyra is deliberately set up this way: the dashboard does not count activity, but shows the status, test outcomes and review dates of controls, and the Statement of Applicability turns that into a report per standard.
If you take one thing from this article, let it be this: remove indicators sooner than you add them. Ask yourself three questions. Can you say for every indicator on your current report which decision rests on it? Do you know for your five most important controls whether they exist, work and have recently been reviewed? And is there at least one number on your dashboard you would rather not show the board? That last one is not a trick question: a report on which everything is green almost certainly measures the wrong things. The value of an indicator lies not in reassuring, but in alarming on time.
This article is a substantiated synthesis of existing scientific research and not independent, peer-reviewed research of its own. Where I interpret or explain, that interpretation is my own responsibility.
In een demo van 30 minuten lopen we het model door aan de hand van jouw situatie.
Plan een demo