Analyses within risk strata overestimate gain in discrimination: the example of coronary artery calcium scores

Abstract

Risk prediction models are potentially useful tools for health practitioners and policy makers. When new predictors are proposed to add to existing models, the improvement of discrimination is one of the main measures to assess any increment in performance. In assessing such predictors, we observed two paradoxes: 1) the discriminative ability within all individual risk strata was worse than for the overall population; 2) incremental discrimination after including a new predictor was greater within each individual risk strata than for the whole population. We show two examples of the paradoxes and analyse the possible causes. The key cause of bias is use of the same prediction model as for both stratifying the population, and as the base model to which the new predictor is added