Maturity Level Assessments of Information Security Controls: An Empirical Analysis of Practitioners' Assessment Capabilities

Maturity models are a widely used concept for measuring information security. The idea is to systematically evaluate the maturity of security-relevant processes in an organisation. This enables decision makers to get an overview of the implementation status of relevant processes to identify neuralgic points. Maturity models thus play a central role in the conception of information security management systems (ISMS). Some industries, for instance, the German automotive industry, have even established security maturity levels as the de facto standard for measuring information security. However, the quality of security maturity level assessments has not been sufficiently investigated yet. We have analysed to what extent security managers can accurately assess the maturity levels of security controls. To verify the quality of maturity level assessments a case study was conducted where security experts assessed a subset of the ISO/IEC 27002 security controls for a hypothetical scenario using the COBIT maturity levels. Additionally, ex-post interviews have been conducted with several study participants to verify some of the hypotheses developed during the previous analyses. Our results show that many security experts struggled with the task and did not perform well. However, we discovered professional characteristics that have a strong significant effect on the assessment capabilities. We also identified various types of additional support that can help practitioners to make more reliable assessments in practice. Moreover, the experts' self-perception was overly optimistic when asked to assess their performance. We even found a weak inverted correlation for more experienced experts, also known as Dunning-Kruger effect. Our results have a strong impact on practise since they indicate that practitioners need support to carry out high-quality assessments and they also show what kind of support addresses the identified challenges.

@Article{SSHP21cose,
  author    = {Christopher Schmitz and Michael Schmid and David Harborth and Sebastian Pape},
  title     = {Maturity Level Assessments of Information Security Controls: An Empirical Analysis of Practitioners' Assessment Capabilities},
  journal   = {Computers \& Security},
  year      = {2021},
  volume    = {108},
  month     = {09},
  doi       = {10.1016/j.cose.2021.102306},
  keywords  = {CS4E, security, human factors, select},
  url       = {https://www.sciencedirect.com/science/article/pii/S0167404821001309},
}