External validation, recalibration and updating of the OxSATS risk model for suicide after self-harm in England.

Abstract

External validations of existing risk models are efficient steps towards potential implementation, avoiding the need to develop new models. However, validation in new clinical settings poses several challenges. To externally validate the OxSATS tool using data from the Oxford Monitoring System for Self-harm in England. OxSATS is a validated tool to predict suicide after self-harm developed using Swedish population registers. We selected episodes of self-harm (International Classification of Diseases, Tenth Revision codes X60-X84; Y10-Y34) by individuals who presented to a large regional hospital between 1 January 2000 and 31 December 2018, and were followed up until 31 December 2019. We applied the OxSATS tool to estimate each individual's suicide risk within 12 months after their index self-harm. We assessed model performance using discrimination (Harrell's concordance index (c-index)) and calibration measures (calibration plot and the observed-to-expected events ratio (O:E)). We assessed the effects of missing predictors on calibration and subsequently recalibrated the model. We identified 16 130 individuals who presented to hospital with self-harm, of whom 106 (0.7%) died by suicide in the 12-month follow-up period. The OxSATS model showed good discrimination in external validation (c-index 0.73, 95% CI 0.68 to 0.79). Recalibration was required because initial calibration reflected a lower outcome rate in the new data. After recalibration, calibration performance was excellent (O:E 1.00, 95% CI 0.77 to 1.25). Despite differences in clinical services and outcome ascertainment, suicide risk models can maintain good predictive performance in new settings. However, recalibration should be considered when applying prediction models in new settings, and the impact of missing predictors should be assessed using sensitivity analyses. OxSATS may help clinicians assess suicide risk after self-harm by providing structured and transparent estimates as part of a comprehensive clinical assessment. Structured risk assessment tools may support identification of individuals who could benefit from additional risk management procedures, such as closer monitoring and follow-up or referral to specialist services, particularly in settings with limited access to such services.