GRADE concept paper 2: Concepts for judging certainty on the calibration of prognostic models in a body of validation studies

Prognostic Factors and Models for Predicting Work Absence in Adults with Musculoskeletal Conditions Consulting a Healthcare Practitioner: A Systematic Review

PURPOSE

It is difficult to predict which employees, in particular those with musculoskeletal pain, will return to work quickly without additional vocational advice and support, which employees will require this support and what levels of support are most appropriate. Consequently, there is no way of ensuring the right individuals are directed towards the right services to support their occupational health needs. The aim of this review will be to identify prognostic factors for duration of work absence in those already absent and examine the utility of prognostic models for work absence.

METHODS

Eight databases were search using a combination of subject headings and key words focusing on work absence, musculoskeletal pain and prognosis. Two authors independently assessed the eligibility of studies, extracted data from all eligible studies and assessed risk of bias using the QUIPS or PROBAST tools, an adapted GRADE was used to assess the strength of the evidence. To make sense of the data prognostic variables were grouped according to categories from the Disability Prevention Framework and the SWiM framework was utilised to synthesise findings.

RESULTS

A total of 23 studies were included in the review, including 13 prognostic models and a total of 110 individual prognostic factors. Overall, the evidence for all prognostic factors was weak, although there was some evidence that older age and better recovery expectations were protective of future absence and that previous absence was likely to predict future absences. There was weak evidence for any of the prognostic models in determining future sickness absence.

CONCLUSION

Analysis was difficult due to the wide range of measures of both prognostic factors and outcome and the differing timescales for follow-up. Future research should ensure that consistent measures are employed and where possible these should be in-line with those suggested by Ravinskaya et al. (2023).

Risk predictors of severe adverse maternal outcomes in pre-eclampsia: a systematic review and meta-analysis protocol

INTRODUCTION

Pre-eclampsia (PE) remains a major contributor to maternal morbidity and mortality globally. Early identification of risk factors and evaluation of prognostic models for severe adverse maternal outcomes are essential for improving management and reducing complications. While numerous studies have explored potential risk markers, there is still no consensus on the most reliable factors and models to use in clinical practice. This systematic review aims to consolidate research on both individual predictors and prognostic models of severe adverse maternal outcomes in PE, providing a comprehensive overview to support better clinical decision-making and patient care.

METHODS AND ANALYSIS

This review follows the Meta-analyses Of Observational Studies in Epidemiology (MOOSE) guidelines and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Protocol 2015 checklist. A systematic search will be performed using a detailed strategy across Medline, Embase, Cochrane, ProQuest dissertations, and grey literature from inception to 2 April 2024. Eligible studies will include those investigating clinical, laboratory-based, and sociodemographic predictors of severe adverse maternal outcomes in PE. Two reviewers will independently assess titles, abstracts, full texts, and extract data and assess study quality using the Quality In Prognostic Studies (QUIPS) tool for studies on risk predictors and the Prediction model Risk of Bias Assessment Tool (PROBAST) for prognostic models. The inclusion criteria will encompass cohort, case-control, and cross-sectional studies published in English and French involving women diagnosed with PE and reporting on the risk prediction for adverse maternal outcomes. The main outcomes of interest will include severe maternal morbidity and mortality during pregnancy, delivery, or within the postpartum period. Analyses will include both narrative synthesis and, where appropriate, meta-analysis using random-effects models. Pooled estimates will be calculated, with publication bias assessed through funnel plots and statistical tests (eg, Begg's and Egger's). Heterogeneity will be primarily assessed through visual inspection of forest plots, supported by statistical measures, such as the I² test, with further exploration through sensitivity, subgroup, and meta-regression analyses.

ETHICS AND DISSEMINATION

This systematic review will be based on published data and will not require ethics approval. Results will be disseminated through peer-reviewed publications and presentations at academic conferences.

PROSPERO REGISTRATION NUMBER

CRD42024517097.

Prediction models for diagnosis and prognosis of the colonization or infection of multidrug-resistant organisms in adults: a systematic review, critical appraisal, and meta-analysis

BACKGROUND

Prediction models help to target patients at risk of multidrug-resistant organism (MDRO) colonization or infection and could serve as tools informing clinical practices to prevent MDRO transmission and inappropriate empiric antibiotic therapy. However, there is limited evidence to identify which among the available models are of low risk of bias and suitable for clinical application.

OBJECTIVES

To identify, describe, appraise, and summarise the performance of all prognostic and diagnostic models developed or validated for predicting MDRO colonization or infection.

DATA SOURCES

Six electronic literature databases and clinical registration databases were searched until April 2022.

STUDY ELIGIBILITY CRITERIA

Development and validation studies of any multivariable prognostic and diagnostic models to predict MDRO colonization or infection in adults.

PARTICIPANTS

Adults (≥ 18 years old) without MDRO colonization or infection (in prognostic models) or with unknown or suspected MDRO colonization or infection (in diagnostic models).

ASSESSMENT OF RISK OF BIAS

The Prediction Model Risk of Bias Assessment Tool was used to assess the risk of bias. Evidence certainty was assessed using the Grading of Recommendations Assessment, Development, and Evaluation approach.

METHODS OF DATA SYNTHESIS

Meta-analyses were conducted to summarize the discrimination and calibration of the models' external validations conducted in at least two non-overlapping datasets.

RESULTS

We included 162 models (108 studies) developed for diagnosing (n = 135) and predicting (n = 27) MDRO colonization or infection. Models exhibited a high-risk of bias, especially in statistical analysis. High-frequency predictors were age, recent invasive procedures, antibiotic usage, and prior hospitalization. Less than 25% of the models underwent external validations, with only seven by independent teams. Meta-analyses for one diagnostic and two prognostic models only produced very low to low certainty of evidence.

CONCLUSIONS

The review comprehensively described the models for identifying patients at risk of MDRO colonization or infection. We cannot recommend which models are ready for application because of the high-risk of bias, limited validations, and low certainty of evidence from meta-analyses, indicating a clear need to improve the conducting and reporting of model development and external validation studies to facilitate clinical application.

Prognostic prediction models for treatment experienced people living with HIV: a protocol for systematic review and meta-analysis

INTRODUCTION

Despite the favourable efficacy of antiretroviral therapy (ART), HIV/AIDS continues to impose significant disease burdens worldwide. This study aims to systematically review published prognostic prediction models for survival outcomes of treatment experienced people living with HIV (TE-PLHIV), to describe their characteristics, compare their performance and assess the risk of bias and real-world clinical utility.

METHODS AND ANALYSIS

Studies will be identified through a comprehensive search in PubMed, EMBASE, Scopus, the Cochrane Library, and OpenGrey databases. Two reviewers will independently conduct a selection of eligible studies, data extraction and critical appraisal. Included studies will be systematically summarised using appropriate tools designed for prognostic prediction modelling studies. Where applicable, evidence will be summarised with meta-analyses.

ETHICS AND DISSEMINATION

Ethical approval is not required because only available published data will be analysed. The results of this work will be published in a peer-reviewed journal.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO registration number CRD42023412118.

GRADE concept paper 8: judging the certainty of discrimination performance estimates of prognostic models in a body of validation studies

BACKGROUND

Prognostic models incorporate multiple prognostic factors to estimate the likelihood of future events for individual patients based on their prognostic factor values. Evaluating these models crucially involves conducting studies to assess their predictive performance, like discrimination. Systematic reviews and meta-analyses of these validation studies play an essential role in selecting models for clinical practice.

METHODS

In this paper, we outline 3 thresholds to determine the target for certainty rating in the discrimination of prognostic models, as observed across a body of validation studies.

RESULTS AND CONCLUSION

We propose 3 thresholds when rating the certainty of evidence about a prognostic model's discrimination. The first threshold amounts to rating certainty in the model's ability to classify better than random chance. The other 2 approaches involve setting thresholds informed by other mechanisms for classification: clinician intuition or an alternative prognostic model developed for the same disease area and outcome. The choice of threshold will vary based on the context. Instead of relying on arbitrary discrimination cut-offs, our approach positions the observed discrimination within an informed spectrum, potentially aiding decisions about a prognostic model's practical utility.

A brief step-by-step guide on conducting a systematic review and meta-analysis of prognostic model studies

Prognostic models provide an avenue to predict the risk of individual patients and support shared-decision making. Many prognostic models are published annually, and systematic reviews provide an avenue to collate the existing evidence behind prognostic models to determine whether a model demonstrates adequate predictive performance and is ready for real-world use. This article provides a brief step-by-step guide on how to conduct a systematic review and meta-analysis of prognostic model studies and how these reviews differ from systematic reviews of therapy and diagnosis.

Multi-domain prognostic models used in middle-aged adults without known cognitive impairment for predicting subsequent dementia

BACKGROUND

Dementia, a global health priority, has no current cure. Around 50 million people worldwide currently live with dementia, and this number is expected to treble by 2050. Some health conditions and lifestyle behaviours can increase or decrease the risk of dementia and are known as 'predictors'. Prognostic models combine such predictors to measure the risk of future dementia. Models that can accurately predict future dementia would help clinicians select high-risk adults in middle age and implement targeted risk reduction.

OBJECTIVES

Our primary objective was to identify multi-domain prognostic models used in middle-aged adults (aged 45 to 65 years) for predicting dementia or cognitive impairment. Eligible multi-domain prognostic models involved two or more of the modifiable dementia predictors identified in a 2020 Lancet Commission report and a 2019 World Health Organization (WHO) report (less education, hearing loss, traumatic brain injury, hypertension, excessive alcohol intake, obesity, smoking, depression, social isolation, physical inactivity, diabetes mellitus, air pollution, poor diet, and cognitive inactivity). Our secondary objectives were to summarise the prognostic models, to appraise their predictive accuracy (discrimination and calibration) as reported in the development and validation studies, and to identify the implications of using dementia prognostic models for the management of people at a higher risk for future dementia.

SEARCH METHODS

We searched MEDLINE, Embase, PsycINFO, CINAHL, and ISI Web of Science Core Collection from inception until 6 June 2022. We performed forwards and backwards citation tracking of included studies using the Web of Science platform.  SELECTION CRITERIA: We included development and validation studies of multi-domain prognostic models. The minimum eligible follow-up was five years. Our primary outcome was an incident clinical diagnosis of dementia based on validated diagnostic criteria, and our secondary outcome was dementia or cognitive impairment determined by any other method.

DATA COLLECTION AND ANALYSIS

Two review authors independently screened the references, extracted data using a template based on the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS), and assessed risk of bias and applicability of included studies using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). We synthesised the C-statistics of models that had been externally validated in at least three comparable studies.  MAIN RESULTS: We identified 20 eligible studies; eight were development studies and 12 were validation studies. There were 14 unique prognostic models: seven models with validation studies and seven models with development-only studies. The models included a median of nine predictors (range 6 to 34); the median number of modifiable predictors was five (range 2 to 11). The most common modifiable predictors in externally validated models were diabetes, hypertension, smoking, physical activity, and obesity. In development-only models, the most common modifiable predictors were obesity, diabetes, hypertension, and smoking. No models included hearing loss or air pollution as predictors. Nineteen studies had a high risk of bias according to the PROBAST assessment, mainly because of inappropriate analysis methods, particularly lack of reported calibration measures. Applicability concerns were low for 12 studies, as their population, predictors, and outcomes were consistent with those of interest for this review. Applicability concerns were high for nine studies, as they lacked baseline cognitive screening or excluded an age group within the range of 45 to 65 years. Only one model, Cardiovascular Risk Factors, Ageing, and Dementia (CAIDE), had been externally validated in multiple studies, allowing for meta-analysis. The CAIDE model included eight predictors (four modifiable predictors): age, education, sex, systolic blood pressure, body mass index (BMI), total cholesterol, physical activity and APOEƐ4 status. Overall, our confidence in the prediction accuracy of CAIDE was very low; our main reasons for downgrading the certainty of the evidence were high risk of bias across all the studies, high concern of applicability, non-overlapping confidence intervals (CIs), and a high degree of heterogeneity. The summary C-statistic was 0.71 (95% CI 0.66 to 0.76; 3 studies; very low-certainty evidence) for the incident clinical diagnosis of dementia, and 0.67 (95% CI 0.61 to 0.73; 3 studies; very low-certainty evidence) for dementia or cognitive impairment based on cognitive scores. Meta-analysis of calibration measures was not possible, as few studies provided these data.

AUTHORS' CONCLUSIONS

We identified 14 unique multi-domain prognostic models used in middle-aged adults for predicting subsequent dementia. Diabetes, hypertension, obesity, and smoking were the most common modifiable risk factors used as predictors in the models. We performed meta-analyses of C-statistics for one model (CAIDE), but the summary values were unreliable. Owing to lack of data, we were unable to meta-analyse the calibration measures of CAIDE. This review highlights the need for further robust external validations of multi-domain prognostic models for predicting future risk of dementia in middle-aged adults.

Clinical First-Trimester Prediction Models for Gestational Diabetes Mellitus: A Systematic Review and Meta-Analysis

BACKGROUND

Gestational diabetes mellitus (GDM) is a common pregnancy complication that negatively impacts the health of both the mother and child. Early prediction of the risk of GDM may permit prompt and effective interventions. This systematic review and meta-analysis aimed to summarize the study characteristics, methodological quality, and model performance of first-trimester prediction model studies for GDM.

METHODS

Five electronic databases, one clinical trial register, and gray literature were searched from the inception date to March 19, 2022. Studies developing or validating a first-trimester prediction model for GDM were included. Two reviewers independently extracted data according to an established checklist and assessed the risk of bias by the Prediction Model Risk of Bias Assessment Tool (PROBAST). We used a random-effects model to perform a quantitative meta-analysis of the predictive power of models that were externally validated at least three times.

RESULTS

We identified 43 model development studies, six model development and external validation studies, and five external validation-only studies. Body mass index, maternal age, and fasting plasma glucose were the most commonly included predictors across all models. Multiple estimates of performance measures were available for eight of the models. Summary estimates range from 0.68 to 0.78 (I² ranged from 0% to 97%).

CONCLUSION

Most studies were assessed as having a high overall risk of bias. Only eight prediction models for GDM have been externally validated at least three times. Future research needs to focus on updating and externally validating existing models.