SPIN-PM: A consensus framework to evaluate the presence of spin in studies on prediction models

OBJECTIVES

To develop a framework to identify and evaluate spin practices and its facilitators in studies on clinical prediction model, regardless of the modelling technique.

STUDY DESIGN

We followed a three-phase consensus process: (1) pre-meeting literature review to generate items to be included; (2) a series of structured meetings to provide comments, discussed and exchanged viewpoints on items to be included with a panel of experienced researchers; and (3) post-meeting review on final list of items and examples to be included. Through this iterative consensus process, a framework was derived after all panel's researchers agreed.

RESULTS

This consensus process involved a panel of eight researchers and resulted in SPIN-PM which consists of two categories of spin (misleading interpretation and misleading transportability), and within these categories, two forms of spin (spin practices and facilitators of spin). We provide criteria and examples.

CONCLUSION

We proposed this guidance aiming to facilitate not only the accurate reporting but also an accurate interpretation and extrapolation of clinical prediction models which will likely improve the reporting quality of subsequent research, as well as reduce research waste.

GRADE concept paper 8: judging the certainty of discrimination performance estimates of prognostic models in a body of validation studies

BACKGROUND

Prognostic models incorporate multiple prognostic factors to estimate the likelihood of future events for individual patients based on their prognostic factor values. Evaluating these models crucially involves conducting studies to assess their predictive performance, like discrimination. Systematic reviews and meta-analyses of these validation studies play an essential role in selecting models for clinical practice.

METHODS

In this paper, we outline 3 thresholds to determine the target for certainty rating in the discrimination of prognostic models, as observed across a body of validation studies.

RESULTS AND CONCLUSION

We propose 3 thresholds when rating the certainty of evidence about a prognostic model's discrimination. The first threshold amounts to rating certainty in the model's ability to classify better than random chance. The other 2 approaches involve setting thresholds informed by other mechanisms for classification: clinician intuition or an alternative prognostic model developed for the same disease area and outcome. The choice of threshold will vary based on the context. Instead of relying on arbitrary discrimination cut-offs, our approach positions the observed discrimination within an informed spectrum, potentially aiding decisions about a prognostic model's practical utility.

Global disease burden of and risk factors for acute lower respiratory infections caused by respiratory syncytial virus in preterm infants and young children in 2019: a systematic review and meta-analysis of aggregated and individual participant data

BACKGROUND

Infants and young children born prematurely are at high risk of severe acute lower respiratory infection (ALRI) caused by respiratory syncytial virus (RSV). In this study, we aimed to assess the global disease burden of and risk factors for RSV-associated ALRI in infants and young children born before 37 weeks of gestation.

METHODS

We conducted a systematic review and meta-analysis of aggregated data from studies published between Jan 1, 1995, and Dec 31, 2021, identified from MEDLINE, Embase, and Global Health, and individual participant data shared by the Respiratory Virus Global Epidemiology Network on respiratory infectious diseases. We estimated RSV-associated ALRI incidence in community, hospital admission, in-hospital mortality, and overall mortality among children younger than 2 years born prematurely. We conducted two-stage random-effects meta-regression analyses accounting for chronological age groups, gestational age bands (early preterm, <32 weeks gestational age [wGA], and late preterm, 32 to <37 wGA), and changes over 5-year intervals from 2000 to 2019. Using individual participant data, we assessed perinatal, sociodemographic, and household factors, and underlying medical conditions for RSV-associated ALRI incidence, hospital admission, and three severity outcome groups (longer hospital stay [>4 days], use of supplemental oxygen and mechanical ventilation, or intensive care unit admission) by estimating pooled odds ratios (ORs) through a two-stage meta-analysis (multivariate logistic regression and random-effects meta-analysis). This study is registered with PROSPERO, CRD42021269742.

FINDINGS

We included 47 studies from the literature and 17 studies with individual participant-level data contributed by the participating investigators. We estimated that, in 2019, 1 650 000 (95% uncertainty range [UR] 1 350 000-1 990 000) RSV-associated ALRI episodes, 533 000 (385 000-730 000) RSV-associated hospital admissions, 3050 (1080-8620) RSV-associated in-hospital deaths, and 26 760 (11 190-46 240) RSV-attributable deaths occurred in preterm infants worldwide. Among early preterm infants, the RSV-associated ALRI incidence rate and hospitalisation rate were significantly higher (rate ratio [RR] ranging from 1·69 to 3·87 across different age groups and outcomes) than for all infants born at any gestational age. In the second year of life, early preterm infants and young children had a similar incidence rate but still a significantly higher hospitalisation rate (RR 2·26 [95% UR 1·27-3·98]) compared with all infants and young children. Although late preterm infants had RSV-associated ALRI incidence rates similar to that of all infants younger than 1 year, they had higher RSV-associated ALRI hospitalisation rate in the first 6 months (RR 1·93 [1·11-3·26]). Overall, preterm infants accounted for 25% (95% UR 16-37) of RSV-associated ALRI hospitalisations in all infants of any gestational age. RSV-associated ALRI in-hospital case fatality ratio in preterm infants was similar to all infants. The factors identified to be associated with RSV-associated ALRI incidence were mainly perinatal and sociodemographic characteristics, and factors associated with severe outcomes from infection were mainly underlying medical conditions including congenital heart disease, tracheostomy, bronchopulmonary dysplasia, chronic lung disease, or Down syndrome (with ORs ranging from 1·40 to 4·23).

INTERPRETATION

Preterm infants face a disproportionately high burden of RSV-associated disease, accounting for 25% of RSV hospitalisation burden. Early preterm infants have a substantial RSV hospitalisation burden persisting into the second year of life. Preventive products for RSV can have a substantial public health impact by preventing RSV-associated ALRI and severe outcomes from infection in preterm infants.

FUNDING

EU Innovative Medicines Initiative Respiratory Syncytial Virus Consortium in Europe.

Community based complex interventions to sustain independence in older people: systematic review and network meta-analysis

OBJECTIVE

To synthesise evidence of the effectiveness of community based complex interventions, grouped according to their intervention components, to sustain independence for older people.

DESIGN

Systematic review and network meta-analysis.

DATA SOURCES

Medline, Embase, CINAHL, PsycINFO, CENTRAL, clinicaltrials.gov, and International Clinical Trials Registry Platform from inception to 9 August 2021 and reference lists of included studies.

ELIGIBILITY CRITERIA

Randomised controlled trials or cluster randomised controlled trials with ≥24 weeks' follow-up studying community based complex interventions for sustaining independence in older people (mean age ≥65 years) living at home, with usual care, placebo, or another complex intervention as comparators.

MAIN OUTCOMES

Living at home, activities of daily living (personal/instrumental), care home placement, and service/economic outcomes at 12 months.

DATA SYNTHESIS

Interventions were grouped according to a specifically developed typology. Random effects network meta-analysis estimated comparative effects; Cochrane's revised tool (RoB 2) structured risk of bias assessment. Grading of recommendations assessment, development and evaluation (GRADE) network meta-analysis structured certainty assessment.

RESULTS

The review included 129 studies (74 946 participants). Nineteen intervention components, including "multifactorial action from individualised care planning" (a process of multidomain assessment and management leading to tailored actions), were identified in 63 combinations. For living at home, compared with no intervention/placebo, evidence favoured multifactorial action from individualised care planning including medication review and regular follow-ups (routine review) (odds ratio 1.22, 95% confidence interval 0.93 to 1.59; moderate certainty); multifactorial action from individualised care planning including medication review without regular follow-ups (2.55, 0.61 to 10.60; low certainty); combined cognitive training, medication review, nutritional support, and exercise (1.93, 0.79 to 4.77; low certainty); and combined activities of daily living training, nutritional support, and exercise (1.79, 0.67 to 4.76; low certainty). Risk screening or the addition of education and self-management strategies to multifactorial action from individualised care planning and routine review with medication review may reduce odds of living at home. For instrumental activities of daily living, evidence favoured multifactorial action from individualised care planning and routine review with medication review (standardised mean difference 0.11, 95% confidence interval 0.00 to 0.21; moderate certainty). Two interventions may reduce instrumental activities of daily living: combined activities of daily living training, aids, and exercise; and combined activities of daily living training, aids, education, exercise, and multifactorial action from individualised care planning and routine review with medication review and self-management strategies. For personal activities of daily living, evidence favoured combined exercise, multifactorial action from individualised care planning, and routine review with medication review and self-management strategies (0.16, -0.51 to 0.82; low certainty). For homecare recipients, evidence favoured addition of multifactorial action from individualised care planning and routine review with medication review (0.60, 0.32 to 0.88; low certainty). High risk of bias and imprecise estimates meant that most evidence was low or very low certainty. Few studies contributed to each comparison, impeding evaluation of inconsistency and frailty.

CONCLUSIONS

The intervention most likely to sustain independence is individualised care planning including medicines optimisation and regular follow-up reviews resulting in multifactorial action. Homecare recipients may particularly benefit from this intervention. Unexpectedly, some combinations may reduce independence. Further research is needed to investigate which combinations of interventions work best for different participants and contexts.

REGISTRATION

PROSPERO CRD42019162195.

Discontinuation of anti-tumour necrosis factor alpha treatment owing to blood test abnormalities, and cost-effectiveness of alternate blood monitoring strategies

BACKGROUND

There is no evidence base to support the use of 6-monthly monitoring blood tests for the early detection of liver, blood and renal toxicity during established anti-tumour necrosis factor alpha (TNFα) treatment.

OBJECTIVES

To evaluate the incidence and risk factors of anti-TNFα treatment cessation owing to liver, blood and renal side-effects, and to estimate the cost-effectiveness of alternate intervals between monitoring blood tests.

METHODS

A secondary care-based retrospective cohort study was performed. Data from the British Association of Dermatologists Biologic and Immunomodulators Register (BADBIR) were used. Patients with at least moderate psoriasis prescribed their first anti-TNFα treatment were included. Treatment discontinuation due to a monitoring blood test abnormality was the primary outcome. Patients were followed-up from start of treatment to the outcome of interest, drug discontinuation, death, 31 July 2021 or up to 5 years, whichever came first. The incidence rate (IR) and 95% confidence intervals (CIs) of anti-TNFα discontinuation with monitoring blood test abnormality was calculated. Multivariate Cox regression was used to examine the association between risk factors and outcome. A mathematical model evaluated costs and quality-adjusted life years (QALYs) associated with increasing the length of time between monitoring blood tests during anti-TNFα treatment.

RESULTS

The cohort included 8819 participants [3710 (42.1%) female, mean (SD) age 44.76 (13.20) years] that contributed 25 058 person-years (PY) of follow-up and experienced 125 treatment discontinuations owing to a monitoring blood test abnormality at an IR of 5.85 (95% CI 4.91-6.97)/1000 PY. Of these, 64 and 61 discontinuations occurred within the first year and after the first year of treatment start, at IRs of 8.62 (95% CI 6.74-11.01) and 3.44 (95% CI 2.67-4.42)/1000 PY, respectively. Increasing age (in years), diabetes and liver disease were associated with anti-TNFα discontinuation after a monitoring blood test abnormality [adjusted hazard ratios of 1.02 (95% CI 1.01-1.04), 1.68 (95% CI 1.00-2.81) and 2.27 (95% CI 1.26-4.07), respectively]. Assuming a threshold of £20 000 per QALY gained, no monitoring was most cost-effective, but all extended periods were cost-effective vs. 3- or 6-monthly monitoring.

CONCLUSIONS

Anti-TNFα drugs were uncommonly discontinued owing to abnormal monitoring blood tests after the first year of treatment. Extending the duration between monitoring blood tests was cost-effective. Our results produce evidence for specialist society guidance to reduce patient monitoring burden and healthcare costs.

Development and external validation of the eFalls tool: a multivariable prediction model for the risk of ED attendance or hospitalisation with a fall or fracture in older adults

BACKGROUND

Falls are common in older adults and can devastate personal independence through injury such as fracture and fear of future falls. Methods to identify people for falls prevention interventions are currently limited, with high risks of bias in published prediction models. We have developed and externally validated the eFalls prediction model using routinely collected primary care electronic health records (EHR) to predict risk of emergency department attendance/hospitalisation with fall or fracture within 1 year.

METHODS

Data comprised two independent, retrospective cohorts of adults aged ≥65 years: the population of Wales, from the Secure Anonymised Information Linkage Databank (model development); the population of Bradford and Airedale, England, from Connected Bradford (external validation). Predictors included electronic frailty index components, supplemented with variables informed by literature reviews and clinical expertise. Fall/fracture risk was modelled using multivariable logistic regression with a Least Absolute Shrinkage and Selection Operator penalty. Predictive performance was assessed through calibration, discrimination and clinical utility. Apparent, internal-external cross-validation and external validation performance were assessed across general practices and in clinically relevant subgroups.

RESULTS

The model's discrimination performance (c-statistic) was 0.72 (95% confidence interval, CI: 0.68 to 0.76) on internal-external cross-validation and 0.82 (95% CI: 0.80 to 0.83) on external validation. Calibration was variable across practices, with some over-prediction in the validation population (calibration-in-the-large, -0.87; 95% CI: -0.96 to -0.78). Clinical utility on external validation was improved after recalibration.

CONCLUSION

The eFalls prediction model shows good performance and could support proactive stratification for falls prevention services if appropriately embedded into primary care EHR systems.

Association between pregnancy-related complications and development of type 2 diabetes and hypertension in women: an umbrella review

BACKGROUND

Despite many systematic reviews and meta-analyses examining the associations of pregnancy complications with risk of type 2 diabetes mellitus (T2DM) and hypertension, previous umbrella reviews have only examined a single pregnancy complication. Here we have synthesised evidence from systematic reviews and meta-analyses on the associations of a wide range of pregnancy-related complications with risk of developing T2DM and hypertension.

METHODS

Medline, Embase and Cochrane Database of Systematic Reviews were searched from inception until 26 September 2022 for systematic reviews and meta-analysis examining the association between pregnancy complications and risk of T2DM and hypertension. Screening of articles, data extraction and quality appraisal (AMSTAR2) were conducted independently by two reviewers using Covidence software. Data were extracted for studies that examined the risk of T2DM and hypertension in pregnant women with the pregnancy complication compared to pregnant women without the pregnancy complication. Summary estimates of each review were presented using tables, forest plots and narrative synthesis and reported following Preferred Reporting Items for Overviews of Reviews (PRIOR) guidelines.

RESULTS

Ten systematic reviews were included. Two pregnancy complications were identified. Gestational diabetes mellitus (GDM): One review showed GDM was associated with a 10-fold higher risk of T2DM at least 1 year after pregnancy (relative risk (RR) 9.51 (95% confidence interval (CI) 7.14 to 12.67) and although the association differed by ethnicity (white: RR 16.28 (95% CI 15.01 to 17.66), non-white: RR 10.38 (95% CI 4.61 to 23.39), mixed: RR 8.31 (95% CI 5.44 to 12.69)), the between subgroups difference were not statistically significant at 5% significance level. Another review showed GDM was associated with higher mean blood pressure at least 3 months postpartum (mean difference in systolic blood pressure: 2.57 (95% CI 1.74 to 3.40) mmHg and mean difference in diastolic blood pressure: 1.89 (95% CI 1.32 to 2.46) mmHg). Hypertensive disorders of pregnancy (HDP): Three reviews showed women with a history of HDP were 3 to 6 times more likely to develop hypertension at least 6 weeks after pregnancy compared to women without HDP (meta-analysis with largest number of studies: odds ratio (OR) 4.33 (3.51 to 5.33)) and one review reported a higher rate of T2DM after HDP (hazard ratio (HR) 2.24 (1.95 to 2.58)) at least a year after pregnancy. One of the three reviews and five other reviews reported women with a history of preeclampsia were 3 to 7 times more likely to develop hypertension at least 6 weeks postpartum (meta-analysis with the largest number of studies: OR 3.90 (3.16 to 4.82) with one of these reviews reporting the association was greatest in women from Asia (Asia: OR 7.54 (95% CI 2.49 to 22.81), Europe: OR 2.19 (95% CI 0.30 to 16.02), North and South America: OR 3.32 (95% CI 1.26 to 8.74)).

CONCLUSIONS

GDM and HDP are associated with a greater risk of developing T2DM and hypertension. Common confounders adjusted for across the included studies in the reviews were maternal age, body mass index (BMI), socioeconomic status, smoking status, pre-pregnancy and current BMI, parity, family history of T2DM or cardiovascular disease, ethnicity, and time of delivery. Further research is needed to evaluate the value of embedding these pregnancy complications as part of assessment for future risk of T2DM and chronic hypertension.

Regularized parametric survival modeling to improve risk prediction models

We propose to combine the benefits of flexible parametric survival modeling and regularization to improve risk prediction modeling in the context of time-to-event data. Thereto, we introduce ridge, lasso, elastic net, and group lasso penalties for both log hazard and log cumulative hazard models. The log (cumulative) hazard in these models is represented by a flexible function of time that may depend on the covariates (i.e., covariate effects may be time-varying). We show that the optimization problem for the proposed models can be formulated as a convex optimization problem and provide a user-friendly R implementation for model fitting and penalty parameter selection based on cross-validation. Simulation study results show the advantage of regularization in terms of increased out-of-sample prediction accuracy and improved calibration and discrimination of predicted survival probabilities, especially when sample size was relatively small with respect to model complexity. An applied example illustrates the proposed methods. In summary, our work provides both a foundation for and an easily accessible implementation of regularized parametric survival modeling and suggests that it improves out-of-sample prediction performance.

Risk of bias assessments in individual participant data meta-analyses of test accuracy and prediction models: a review shows improvements are needed

OBJECTIVES

Risk of bias assessments are important in meta-analyses of both aggregate and individual participant data (IPD). There is limited evidence on whether and how risk of bias of included studies or datasets in IPD meta-analyses (IPDMAs) is assessed. We review how risk of bias is currently assessed, reported, and incorporated in IPDMAs of test accuracy and clinical prediction model studies and provide recommendations for improvement.

STUDY DESIGN AND SETTING

We searched PubMed (January 2018-May 2020) to identify IPDMAs of test accuracy and prediction models, then elicited whether each IPDMA assessed risk of bias of included studies and, if so, how assessments were reported and subsequently incorporated into the IPDMAs.

RESULTS

Forty-nine IPDMAs were included. Nineteen of 27 (70%) test accuracy IPDMAs assessed risk of bias, compared to 5 of 22 (23%) prediction model IPDMAs. Seventeen of 19 (89%) test accuracy IPDMAs used Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2), but no tool was used consistently among prediction model IPDMAs. Of IPDMAs assessing risk of bias, 7 (37%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided details on the information sources (e.g., the original manuscript, IPD, primary investigators) used to inform judgments, and 4 (21%) test accuracy IPDMAs and 1 (20%) prediction model IPDMA provided information or whether assessments were done before or after obtaining the IPD of the included studies or datasets. Of all included IPDMAs, only seven test accuracy IPDMAs (26%) and one prediction model IPDMA (5%) incorporated risk of bias assessments into their meta-analyses. For future IPDMA projects, we provide guidance on how to adapt tools such as Prediction model Risk Of Bias ASsessment Tool (for prediction models) and QUADAS-2 (for test accuracy) to assess risk of bias of included primary studies and their IPD.

CONCLUSION

Risk of bias assessments and their reporting need to be improved in IPDMAs of test accuracy and, especially, prediction model studies. Using recommended tools, both before and after IPD are obtained, will address this.

Open science practices need substantial improvement in prognostic model studies in oncology using machine learning

OBJECTIVE

To describe the frequency of open science practices in a contemporary sample of studies developing prognostic models using machine learning methods in the field of oncology.

STUDY DESIGN AND SETTING

We conducted a systematic review, searching the MEDLINE database between December 1, 2022, and December 31, 2022, for studies developing a multivariable prognostic model using machine learning methods (as defined by the authors) in oncology. Two authors independently screened records and extracted open science practices.

RESULTS

We identified 46 publications describing the development of a multivariable prognostic model. The adoption of open science principles was poor. Only one study reported availability of a study protocol, and only one study was registered. Funding statements and conflicts of interest statements were common. Thirty-five studies (76%) provided data sharing statements, with 21 (46%) indicating data were available on request to the authors and seven declaring data sharing was not applicable. Two studies (4%) shared data. Only 12 studies (26%) provided code sharing statements, including 2 (4%) that indicated the code was available on request to the authors. Only 11 studies (24%) provided sufficient information to allow their model to be used in practice. The use of reporting guidelines was rare: eight studies (18%) mentioning using a reporting guideline, with 4 (10%) using the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis statement, 1 (2%) using Minimum Information About Clinical Artificial Intelligence Modeling and Consolidated Standards Of Reporting Trials-Artificial Intelligence, 1 (2%) using Strengthening The Reporting Of Observational Studies In Epidemiology, 1 (2%) using Standards for Reporting Diagnostic Accuracy Studies, and 1 (2%) using Transparent Reporting of Evaluations with Nonrandomized Designs.

CONCLUSION

The adoption of open science principles in oncology studies developing prognostic models using machine learning methods is poor. Guidance and an increased awareness of benefits and best practices of open science are needed for prediction research in oncology.

Clinical prediction models and the multiverse of madness

BACKGROUND

Each year, thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice.

MAIN BODY

We discuss how the creation of a prediction model (e.g. using regression or machine learning methods) is dependent on the sample and size of data used to develop it-were a different sample of the same size used from the same overarching population, the developed model could be very different even when the same model development methods are used. In other words, for each model created, there exists a multiverse of other potential models for that sample size and, crucially, an individual's predicted value (e.g. estimated risk) may vary greatly across this multiverse. The more an individual's prediction varies across the multiverse, the greater the instability. We show how small development datasets lead to more different models in the multiverse, often with vastly unstable individual predictions, and explain how this can be exposed by using bootstrapping and presenting instability plots. We recommend healthcare researchers seek to use large model development datasets to reduce instability concerns. This is especially important to ensure reliability across subgroups and improve model fairness in practice.

CONCLUSIONS

Instability is concerning as an individual's predicted value is used to guide their counselling, resource prioritisation, and clinical decision making. If different samples lead to different models with very different predictions for the same individual, then this should cast doubt into using a particular model for that individual. Therefore, visualising, quantifying and reporting the instability in individual-level predictions is essential when proposing a new model.

Effect of a doctor working during the festive period on population health: natural experiment using 60 years of Doctor Who episodes (the TARDIS study)

OBJECTIVE

To examine the effect of a (fictional) doctor working during the festive period on population health.

DESIGN

Natural experiment.

SETTING

England, Wales, and the UK.

MAIN OUTCOME MEASURES

Age standardised annual mortality rates in England, Wales, and the UK from 1963, when the BBC first broadcast Doctor Who, a fictional programme with a character called the Doctor who fights villains and intervenes to save others while travelling through space and time. Mortality rates were modelled in a time series analysis accounting for non-linear trends over time, and associations were estimated in relation to a new Doctor Who episode broadcast during the previous festive period, 24 December to 1 January. An interrupted time series analysis modelled the shift in mortality rates from 2005, when festive episodes of Doctor Who could be classed as a yearly Christmas intervention.

RESULTS

31 festive periods from 1963 have featured a new Doctor Who episode, including 14 broadcast on Christmas Day. In time series analyses, an association was found between broadcasts during the festive period and subsequent lower annual mortality rates. In particular, episodes shown on Christmas Day were associated with 0.60 fewer deaths per 1000 person years (95% confidence interval 0.21 to 0.99; P=0.003) in England and Wales and 0.40 fewer deaths per 1000 person years (0.08 to 0.73; P=0.02) in the UK. The interrupted time series analysis showed a strong shift (reduction) in mortality rates from 2005 onwards in association with the Doctor Who Christmas intervention, with a mean 0.73 fewer deaths per 1000 person years (0.21 to 1.26; P=0.01) in England and Wales and a mean 0.62 fewer deaths per 1000 person years (0.16 to 1.09; P=0.01) in the UK.

CONCLUSIONS

A new Doctor Who episode shown every festive period, especially on Christmas Day, was associated with reduced mortality rates in England, Wales, and the UK, suggesting that a doctor working over the festive period could lower mortality rates. This finding reinforces why healthcare provision should not be taken for granted and may prompt the BBC and Disney+ to televise new episodes of Doctor Who every festive period, ideally on Christmas Day.

Up Front and Open? Shrouded in Secrecy? Or Somewhere in Between? A Meta-Research Systematic Review of Open Science Practices in Sport Medicine Research

OBJECTIVE: To investigate open science practices in research published in the top 5 sports medicine journals from May 1, 2022, and October 1, 2022. DESIGN: A meta-research systematic review. LITERATURE SEARCH: Open science practices were searched in MEDLINE. STUDY SELECTION CRITERIA: We included original scientific research published in one of the identified top 5 sports medicine journals in 2022 as ranked by Clarivate: (1) British Journal of Sports Medicine, (2) Journal of Sport and Health Science, (3) American Journal of Sports Medicine, (4) Medicine and Science in Sports and Exercise, and (5) Sports Medicine-Open. Studies were excluded if they were systematic reviews, qualitative research, gray literature, or animal or cadaver models. DATA SYNTHESIS: Open science practices were extracted in accordance with the Transparency and Openness Promotion guidelines and patient and public involvement. RESULTS: Two hundred forty-three studies were included. The median number of open science practices in each study was 2, out of a maximum of 12 (range: 0-8; interquartile range: 2). Two hundred thirty-four studies (96%, 95% confidence interval [CI]: 94%-99%) provided an author conflict-of-interest statement and 163 (67%, 95% CI: 62%-73%) reported funding. Twenty-one studies (9%, 95% CI: 5%-12%) provided open-access data. Fifty-four studies (22%, 95% CI: 17%-27%) included a data availability statement and 3 (1%, 95% CI: 0%-3%) made code available. Seventy-six studies (32%, 95% CI: 25%-37%) had transparent materials and 30 (12%, 95% CI: 8%-16%) used a reporting guideline. Twenty-eight studies (12%, 95% CI: 8%-16%) were preregistered. Six studies (3%, 95% CI: 1%-4%) published a protocol. Four studies (2%, 95% CI: 0%-3%) reported an analysis plan a priori. Seven studies (3%, 95% CI: 1%-5%) reported patient and public involvement. CONCLUSION: Open science practices in the sports medicine field are extremely limited. The least followed practices were sharing code, data, and analysis plans. J Orthop Sports Phys Ther 2023;53(12):1-13. Epub 20 October 2023. doi:10.2519/jospt.2023.12016.

Stability of clinical prediction models developed using statistical or machine learning methods

Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model-building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model-building steps (those used to develop the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and deriving (i) a prediction instability plot of bootstrap model versus original model predictions; (ii) the mean absolute prediction error (mean absolute difference between individuals' original and bootstrap model predictions), and (iii) calibration, classification, and decision curve instability plots of bootstrap models applied in the original sample. A case study illustrates how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), while informing a model's critical appraisal (risk of bias rating), fairness, and further validation requirements.

Using temporal recalibration to improve the calibration of risk prediction models in competing risk settings when there are trends in survival over time

We have previously proposed temporal recalibration to account for trends in survival over time to improve the calibration of predictions from prognostic models for new patients. This involves first estimating the predictor effects using data from all individuals (full dataset) and then re-estimating the baseline using a subset of the most recent data whilst constraining the predictor effects to remain the same. In this article, we demonstrate how temporal recalibration can be applied in competing risk settings by recalibrating each cause-specific (or subdistribution) hazard model separately. We illustrate this using an example of colon cancer survival with data from the Surveillance Epidemiology and End Results (SEER) program. Data from patients diagnosed in 1995-2004 were used to fit two models for deaths due to colon cancer and other causes respectively. We discuss considerations that need to be made in order to apply temporal recalibration such as the choice of data used in the recalibration step. We also demonstrate how to assess the calibration of these models in new data for patients diagnosed subsequently in 2005. Comparison was made to a standard analysis (when improvements over time are not taken into account) and a period analysis which is similar to temporal recalibration but differs in the data used to estimate the predictor effects. The 10-year calibration plots demonstrated that using the standard approach over-estimated the risk of death due to colon cancer and the total risk of death and that calibration was improved using temporal recalibration or period analysis.

Predictors of outcome in sciatica patients following an epidural steroid injection: the POiSE prospective observational cohort study protocol

INTRODUCTION

Sciatica can be very painful and, in most cases, is due to pressure on a spinal nerve root from a disc herniation with associated inflammation. For some patients, the pain persists, and one management option is a spinal epidural steroid injection (ESI). The aim of an ESI is to relieve leg pain, improve function and reduce the need for surgery. ESIs work well in some patients but not in others, but we cannot identify these patient subgroups currently. This study aims to identify factors, including patient characteristics, clinical examination and imaging findings, that help in predicting who does well and who does not after an ESI. The overall objective is to develop a prognostic model to support individualised patient and clinical decision-making regarding ESI.

METHODS

POiSE is a prospective cohort study of 439 patients with sciatica referred by their clinician for an ESI. Participants will receive weekly text messages until 12 weeks following their ESIand then again at 24 weeks following their ESI to collect data on leg pain severity. Questionnaires will be sent to participants at baseline, 6, 12 and 24 weeks after their ESI to collect data on pain, disability, recovery and additional interventions. The prognosis for the cohort will be described. The primary outcome measure for the prognostic model is leg pain at 6 weeks. Prognostic models will also be developed for secondary outcomes of disability and recovery at 6 weeks and additional interventions at 24 weeks following ESI. Statistical analyses will include multivariable linear and logistic regression with mixed effects model.

ETHICS AND DISSEMINATION

The POiSE study has received ethical approval (South Central Berkshire B Research Ethics Committee 21/SC/0257). Dissemination will be guided by our patient and public engagement group and will include scientific publications, conference presentations and social media.

Two-stage or not two-stage? That is the question for IPD meta-analysis projects

Individual participant data meta-analysis (IPDMA) projects obtain, check, harmonise and synthesise raw data from multiple studies. When undertaking the meta-analysis, researchers must decide between a two-stage or a one-stage approach. In a two-stage approach, the IPD are first analysed separately within each study to obtain aggregate data (e.g., treatment effect estimates and standard errors); then, in the second stage, these aggregate data are combined in a standard meta-analysis model (e.g., common-effect or random-effects). In a one-stage approach, the IPD from all studies are analysed in a single step using an appropriate model that accounts for clustering of participants within studies and, potentially, between-study heterogeneity (e.g., a general or generalised linear mixed model). The best approach to take is debated in the literature, and so here we provide clearer guidance for a broad audience. Both approaches are important tools for IPDMA researchers and neither are a panacea. If most studies in the IPDMA are small (few participants or events), a one-stage approach is recommended due to using a more exact likelihood. However, in other situations, researchers can choose either approach, carefully following best practice. Some previous claims recommending to always use a one-stage approach are misleading, and the two-stage approach will often suffice for most researchers. When differences do arise between the two approaches, often it is caused by researchers using different modelling assumptions or estimation methods, rather than using one or two stages per se.

The Trade Secret Taboo: Open Science Methods are Required to Improve Prediction Models in Sports Medicine and Performance

Clinical prediction models in sports medicine that utilize regression or machine learning techniques have become more widely published, used, and disseminated. However, these models are typically characterized by poor methodology and incomplete reporting, and an inadequate evaluation of performance, leading to unreliable predictions and weak clinical utility within their intended sport population. Before implementation in practice, models require a thorough evaluation. Strong replicable methods and transparency reporting allow practitioners and researchers to make independent judgments as to the model's validity, performance, clinical usefulness, and confidence it will do no harm. However, this is not reflected in the sports medicine literature. As shown in a recent systematic review of models for predicting sports injury models, most were typically characterized by poor methodology, incomplete reporting, and inadequate performance evaluation. Because of constraints imposed by data from individual teams, the development of accurate, reliable, and useful models is highly reliant on external validation. However, a barrier to collaboration is a desire to maintain a competitive advantage; a team's proprietary information is often perceived as high value, and so these 'trade secrets' are frequently guarded. These 'trade secrets' also apply to commercially available models, as developers are unwilling to share proprietary (and potentially profitable) development and validation information. In this Current Opinion, we: (1) argue that open science is essential for improving sport prediction models and (2) critically examine sport prediction models for open science practices.

Risk-stratified monitoring for thiopurine toxicity in immune-mediated inflammatory diseases: prognostic model development, validation, and, health economic evaluation

Background

Patients established on thiopurines (e.g., azathioprine) are recommended to undergo three-monthly blood tests for the early detection of blood, liver, or kidney toxicity. These side-effects are uncommon during long-term treatment. We developed a prognostic model that could be used to inform risk-stratified decisions on frequency of monitoring blood-tests during long-term thiopurine treatment, and, performed health-economic evaluation of alternate monitoring intervals.

Methods

This was a retrospective cohort study set in the UK primary-care. Data from the Clinical Practice Research Datalink Aurum and Gold formed development and validation cohorts, respectively. People age ≥18 years, diagnosed with an immune mediated inflammatory disease, prescribed thiopurine by their general practitioner for at-least six-months between January 1, 2007 and December 31, 2019 were eligible. The outcome was thiopurine discontinuation with abnormal blood-test results. Patients were followed up from six-months after first primary-care thiopurine prescription to up to five-years. Penalised Cox regression developed the risk equation. Multiple imputation handled missing predictor data. Calibration and discrimination assessed model performance. A mathematical model evaluated costs and quality-adjusted life years associated with lengthening the interval between blood-tests.

Findings

Data from 5982 (405 events over 16,117 person-years) and 3573 (269 events over 9075 person-years) participants were included in the development and validation cohorts, respectively. Fourteen candidate predictors (21 parameters) were included. The optimism adjusted R2 and Royston D statistic in development data were 0.11 and 0.76, respectively. The calibration slope and Royston D statistic (95% Confidence Interval) in the validation data were 1.10 (0.84-1.36) and 0.72 (0.52-0.92), respectively. A 2-year period between monitoring blood-test was most cost-effective in all deciles of predicted risk but the gain between monitoring annually or biennially reduced in higher risk deciles.

Interpretation

This prognostic model requires information that is readily available during routine clinical care and may be used to risk-stratify blood-test monitoring for thiopurine toxicity. These findings should be considered by specialist societies when recommending blood monitoring during thiopurine prescription to bring about sustainable and equitable change in clinical practice.

Funding

National Institute for Health and Care Research.

Poor handling of continuous predictors in clinical prediction models using logistic regression: a systematic review

BACKGROUND AND OBJECTIVES

When developing a clinical prediction model, assuming a linear relationship between the continuous predictors and outcome is not recommended. Incorrect specification of the functional form of continuous predictors could reduce predictive accuracy. We examine how continuous predictors are handled in studies developing a clinical prediction model.

METHODS

We searched PubMed for clinical prediction model studies developing a logistic regression model for a binary outcome, published between July 01, 2020, and July 30, 2020.

RESULTS

In total, 118 studies were included in the review (18 studies (15%) assessed the linearity assumption or used methods to handle nonlinearity, and 100 studies (85%) did not). Transformation and splines were commonly used to handle nonlinearity, used in 7 (n = 7/18, 39%) and 6 (n = 6/18, 33%) studies, respectively. Categorization was most often used method to handle continuous predictors (n = 67/118, 56.8%) where most studies used dichotomization (n = 40/67, 60%). Only ten models included nonlinear terms in the final model (n = 10/18, 56%).

CONCLUSION

Though widely recommended not to categorize continuous predictors or assume a linear relationship between outcome and continuous predictors, most studies categorize continuous predictors, few studies assess the linearity assumption, and even fewer use methodology to account for nonlinearity. Methodological guidance is provided to guide researchers on how to handle continuous predictors when developing a clinical prediction model.

Calculating the power of a planned individual participant data meta-analysis of randomised trials to examine a treatment-covariate interaction with a time-to-event outcome

Before embarking on an individual participant data meta-analysis (IPDMA) project, researchers should consider the power of their planned IPDMA conditional on the studies promising their IPD and their characteristics. Such power estimates help inform whether the IPDMA project is worth the time and funding investment, before IPD are collected. Here, we suggest how to estimate the power of a planned IPDMA of randomised trials aiming to examine treatment-covariate interactions at the participant-level (i.e., treatment effect modifiers). We focus on a time-to-event (survival) outcome with a binary or continuous covariate, and propose an approximate analytic power calculation that conditions on the actual characteristics of trials, for example, in terms of sample sizes and covariate distributions. The proposed method has five steps: (i) extracting the following aggregate data for each group in each trial-the number of participants and events, the mean and SD for each continuous covariate, and the proportion of participants in each category for each binary covariate; (ii) specifying a minimally important interaction size; (iii) deriving an approximate estimate of Fisher's information matrix for each trial and the corresponding variance of the interaction estimate per trial, based on assuming an exponential survival distribution; (iv) deriving the estimated variance of the summary interaction estimate from the planned IPDMA, under a common-effect assumption, and (v) calculating the power of the IPDMA based on a two-sided Wald test. Stata and R code are provided and a real example provided for illustration. Further evaluation in real examples and simulations is needed.

Vitamin D supplementation for improving bone density in vitamin D-deficient children and adolescents: systematic review and individual participant data meta-analysis of randomized controlled trials

BACKGROUND

Vitamin D supplements are widely used for improving bone health in children and adolescents, but their effects in vitamin D-deficient children are unclear.

OBJECTIVES

This study aimed to examine whether the effect of vitamin D supplementation on bone mineral density (BMD) in children and adolescents differs by baseline vitamin D status and estimate the effect in vitamin D-deficient individuals.

METHODS

This is a systematic review and individual participant data (IPD) meta-analysis. We searched the Cochrane Central Register of Controlled Trials, MEDLINE, MBASE, CINAHL, AMED, and ISI Web of Science (until May 27, 2020) for randomized controlled trials (RCTs) of vitamin D supplementation reporting bone density outcomes after ≥6 mo in healthy individuals aged 1-19 y. We used two-stage IPD meta-analysis to determine treatment effects on total body bone mineral content and BMD at the hip, femoral neck, lumbar spine, and proximal and distal forearm after 1 y; examine whether effects varied by baseline serum 25-hydroxyvitamin D [25(OH)D] concentration, and estimate treatment effects for each 25(OH)D subgroup.

RESULTS

Eleven RCTs were included. Nine comprising 1439 participants provided IPD (86% females, mean baseline 25(OH)D = 36.3 nmol/L). Vitamin D supplementation had a small overall effect on total hip areal BMD (weighted mean difference = 6.8; 95% confidence interval: 0.7, 12.9 mg/cm2; I2 = 7.2%), but no effects on other outcomes. There was no clear evidence of linear or nonlinear interactions between baseline 25(OH)D and treatment; effects were similar in baseline 25(OH)D subgroups (cutoff of 35 or 50 nmol/L). The evidence was of high certainty.

CONCLUSIONS

Clinically important benefits for bone density from 1-y vitamin D supplementation in healthy children and adolescents, regardless of baseline vitamin D status, are unlikely. However, our findings are mostly generalizable to White postpubertal girls and do not apply to those with baseline 25(OH)D outside the studied range or with symptomatic vitamin D deficiency (e.g., rickets). This study was preregistered at PROSPERO as CRD42017068772. https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42017068772.

Propensity-based standardization to enhance the validation and interpretation of prediction model discrimination for a target population

External validation of the discriminative ability of prediction models is of key importance. However, the interpretation of such evaluations is challenging, as the ability to discriminate depends on both the sample characteristics (ie, case-mix) and the generalizability of predictor coefficients, but most discrimination indices do not provide any insight into their respective contributions. To disentangle differences in discriminative ability across external validation samples due to a lack of model generalizability from differences in sample characteristics, we propose propensity-weighted measures of discrimination. These weighted metrics, which are derived from propensity scores for sample membership, are standardized for case-mix differences between the model development and validation samples, allowing for a fair comparison of discriminative ability in terms of model characteristics in a target population of interest. We illustrate our methods with the validation of eight prediction models for deep vein thrombosis in 12 external validation data sets and assess our methods in a simulation study. In the illustrative example, propensity score standardization reduced between-study heterogeneity of discrimination, indicating that between-study variability was partially attributable to case-mix. The simulation study showed that only flexible propensity-score methods (allowing for non-linear effects) produced unbiased estimates of model discrimination in the target population, and only when the positivity assumption was met. Propensity score-based standardization may facilitate the interpretation of (heterogeneity in) discriminative ability of a prediction model as observed across multiple studies, and may guide model updating strategies for a particular target population. Careful propensity score modeling with attention for non-linear relations is recommended.

Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review

BACKGROUND

Having an appropriate sample size is important when developing a clinical prediction model. We aimed to review how sample size is considered in studies developing a prediction model for a binary outcome.

METHODS

We searched PubMed for studies published between 01/07/2020 and 30/07/2020 and reviewed the sample size calculations used to develop the prediction models. Using the available information, we calculated the minimum sample size that would be needed to estimate overall risk and minimise overfitting in each study and summarised the difference between the calculated and used sample size.

RESULTS

A total of 119 studies were included, of which nine studies provided sample size justification (8%). The recommended minimum sample size could be calculated for 94 studies: 73% (95% CI: 63-82%) used sample sizes lower than required to estimate overall risk and minimise overfitting including 26% studies that used sample sizes lower than required to estimate overall risk only. A similar number of studies did not meet the ≥ 10EPV criteria (75%, 95% CI: 66-84%). The median deficit of the number of events used to develop a model was 75 [IQR: 234 lower to 7 higher]) which reduced to 63 if the total available data (before any data splitting) was used [IQR:225 lower to 7 higher]. Studies that met the minimum required sample size had a median c-statistic of 0.84 (IQR:0.80 to 0.9) and studies where the minimum sample size was not met had a median c-statistic of 0.83 (IQR: 0.75 to 0.9). Studies that met the ≥ 10 EPP criteria had a median c-statistic of 0.80 (IQR: 0.73 to 0.84).

CONCLUSIONS

Prediction models are often developed with no sample size calculation, as a consequence many are too small to precisely estimate the overall risk. We encourage researchers to justify, perform and report sample size calculations when developing a prediction model.

Moderators of the effect of therapeutic exercise for knee and hip osteoarthritis: a systematic review and individual participant data meta-analysis

BACKGROUND

Many international clinical guidelines recommend therapeutic exercise as a core treatment for knee and hip osteoarthritis. We aimed to identify individual patient-level moderators of the effect of therapeutic exercise for reducing pain and improving physical function in people with knee osteoarthritis, hip osteoarthritis, or both.

METHODS

We did a systematic review and individual participant data (IPD) meta-analysis of randomised controlled trials comparing therapeutic exercise with non-exercise controls in people with knee osteoathritis, hip osteoarthritis, or both. We searched ten databases from March 1, 2012, to Feb 25, 2019, for randomised controlled trials comparing the effects of exercise with non-exercise or other exercise controls on pain and physical function outcomes among people with knee osteoarthritis, hip osteoarthritis, or both. IPD were requested from leads of all eligible randomised controlled trials. 12 potential moderators of interest were explored to ascertain whether they were associated with short-term (12 weeks), medium-term (6 months), and long-term (12 months) effects of exercise on self-reported pain and physical function, in comparison with non-exercise controls. Overall intervention effects were also summarised. This study is prospectively registered on PROSPERO (CRD42017054049).

FINDINGS

Of 91 eligible randomised controlled trials that compared exercise with non-exercise controls, IPD from 31 randomised controlled trials (n=4241 participants) were included in the meta-analysis. Randomised controlled trials included participants with knee osteoarthritis (18 [58%] of 31 trials), hip osteoarthritis (six [19%]), or both (seven [23%]) and tested heterogeneous exercise interventions versus heterogeneous non-exercise controls, with variable risk of bias. Summary meta-analysis results showed that, on average, compared with non-exercise controls, therapeutic exercise reduced pain on a standardised 0-100 scale (with 100 corresponding to worst pain), with a difference of -6·36 points (95% CI -8·45 to -4·27, borrowing of strength [BoS] 10·3%, between-study variance [τ2] 21·6) in the short term, -3·77 points (-5·97 to -1·57, BoS 30·0%, τ2 14·4) in the medium term, and -3·43 points (-5·18 to -1·69, BoS 31·7%, τ2 4·5) in the long term. Therapeutic exercise also improved physical function on a standardised 0-100 scale (with 100 corresponding to worst physical function), with a difference of -4·46 points in the short term (95% CI -5·95 to -2·98, BoS 10·5%, τ2 10·1), -2·71 points in the medium term (-4·63 to -0·78, BoS 33·6%, τ2 11·9), and -3·39 points in the long term (-4·97 to -1·81, BoS 34·1%, τ2 6·4). Baseline pain and physical function moderated the effect of exercise on pain and physical function outcomes. Those with higher self-reported pain and physical function scores at baseline (ie, poorer physical function) generally benefited more than those with lower self-reported pain and physical function scores at baseline, with the evidence most certain in the short term (12 weeks).

INTERPRETATION

There was evidence of a small, positive overall effect of therapeutic exercise on pain and physical function compared with non-exercise controls. However, this effect is of questionable clinical importance, particularly in the medium and long term. As individuals with higher pain severity and poorer physical function at baseline benefited more than those with lower pain severity and better physical function at baseline, targeting individuals with higher levels of osteoarthritis-related pain and disability for therapeutic exercise might be of merit.

FUNDING

Chartered Society of Physiotherapy Charitable Trust and the National Institute for Health and Care Research.

Nonlinear effects and effect modification at the participant-level in IPD meta-analysis part 2: methodological guidance is available

OBJECTIVES

To review methodological guidance for nonlinear covariate-outcome associations (NL), and linear effect modification and nonlinear effect modification (LEM and NLEM) at the participant level in individual participant data meta-analyses (IPDMAs) and their power requirements.

STUDY DESIGN AND SETTING

We searched Medline, Embase, Web of Science, Scopus, PsycINFO and the Cochrane Library to identify methodology publications on IPDMA of LEM, NL or NLEM (PROSPERO CRD42019126768).

RESULTS

Through screening 6,466 records we identified 54 potential articles of which 23 full texts were relevant. Nine further relevant publications were published before or after the literature search and were added. Of these 32 references, 21 articles considered LEM, 6 articles NL or NLEM and 6 articles described sample size calculations. A book described all four. Sample size may be calculated through simulation or closed form. Assessments of LEM or NLEM at the participant level need to be based on within-trial information alone. Nonlinearity (NL or NLEM) can be modeled using polynomials or splines to avoid categorization.

CONCLUSION

Detailed methodological guidance on IPDMA of effect modification at participant-level is available. However, methodology papers for sample size and nonlinearity are rarer and may not cover all scenarios. On these aspects, further guidance is needed.

Nonlinear effects and effect modification at the participant-level in IPD meta-analysis part 1: analysis methods are often substandard

OBJECTIVES

To review analysis methods used for linear effect modification (LEM), nonlinear covariate-outcome associations (NL) and nonlinear effect modification (NLEM) at the participant-level in individual participant data meta-analysis (IPDMA).

STUDY DESIGN AND SETTING

We searched Medline, Embase, Web of Science, Scopus, PsycINFO and the Cochrane Library to identify IPDMA of randomized controlled trials (PROSPERO CRD42019126768). We investigated if and how IPDMA examined LEM, NL and NLEM, including whether aggregation bias was addressed and if power was considered.

RESULTS

We screened 6,466 records, randomly sampled 207 and identified 100 IPDMA of LEM, NL or NLEM. Power for LEM was calculated a priori in 3 IPDMA. Of 100 IPDMA, 94 analyzed LEM, 4 NLEM and 8 NL. One-stage models were favoured for all three (56%, 100%, 50%, respectively). Two-stage models were used in 15%, 0% and 25% of IPDMA with unclear descriptions in 30%, 0% and 25%, respectively. Only 12% of one-stage LEM and NLEM IPDMA provided sufficient detail to confirm they had addressed aggregation bias.

CONCLUSION

Investigation of effect modification at the participant-level is common in IPDMA projects, but methods are often open to bias or lack detailed descriptions. Nonlinearity of continuous covariates and power of IPDMA are rarely assessed.

Systematic review finds "spin" practices and poor reporting standards in studies on machine learning-based prediction models

OBJECTIVES

We evaluated the presence and frequency of spin practices and poor reporting standards in studies that developed and/or validated clinical prediction models using supervised machine learning techniques.

STUDY DESIGN AND SETTING

We systematically searched PubMed from 01/2018 to 12/2019 to identify diagnostic and prognostic prediction model studies using supervised machine learning. No restrictions were placed on data source, outcome, or clinical specialty.

RESULTS

We included 152 studies: 38% reported diagnostic models and 62% prognostic models. When reported, discrimination was described without precision estimates in 53/71 abstracts (74.6% [95% CI 63.4-83.3]) and 53/81 main texts (65.4% [95% CI 54.6-74.9]). Of the 21 abstracts that recommended the model to be used in daily practice, 20 (95.2% [95% CI 77.3-99.8]) lacked any external validation of the developed models. Likewise, 74/133 (55.6% [95% CI 47.2-63.8]) studies made recommendations for clinical use in their main text without any external validation. Reporting guidelines were cited in 13/152 (8.6% [95% CI 5.1-14.1]) studies.

CONCLUSION

Spin practices and poor reporting standards are also present in studies on prediction models using machine learning techniques. A tailored framework for the identification of spin will enhance the sound reporting of prediction model studies.

Risk stratified monitoring for methotrexate toxicity in immune mediated inflammatory diseases: prognostic model development and validation using primary care data from the UK

OBJECTIVE

To develop and validate a prognostic model to inform risk stratified decisions on frequency of monitoring blood tests during long term methotrexate treatment.

DESIGN

Retrospective cohort study.

SETTING

Electronic health records within the UK's Clinical Practice Research Datalink (CPRD) Gold and CPRD Aurum.

PARTICIPANTS

Adults (≥18 years) with a diagnosis of an immune mediated inflammatory disease who were prescribed methotrexate by their general practitioner for six months or more during 2007-19.

MAIN OUTCOME MEASURE

Discontinuation of methotrexate owing to abnormal monitoring blood test result. Patients were followed-up from six months after their first prescription for methotrexate in primary care to the earliest of outcome, drug discontinuation for any other reason, leaving the practice, last data collection from the practice, death, five years, or 31 December 2019. Cox regression was performed to develop the risk equation, with bootstrapping used to shrink predictor effects for optimism. Multiple imputation handled missing predictor data. Model performance was assessed in terms of calibration and discrimination.

RESULTS

Data from 13 110 (854 events) and 23 999 (1486 events) participants were included in the development and validation cohorts, respectively. 11 candidate predictors (17 parameters) were included. In the development dataset, the optimism adjusted R² was 0.13 and the optimism adjusted Royston D statistic was 0.79. The calibration slope and Royston D statistic in the validation dataset for the entire follow-up period was 0.94 (95% confidence interval 0.85 to 1.02) and 0.75 (95% confidence interval 0.67 to 0.83), respectively. The prognostic model performed well in predicting outcomes in clinically relevant subgroups defined by age group, type of immune mediated inflammatory disease, and methotrexate dose.

CONCLUSION

A prognostic model was developed and validated that uses information collected during routine clinical care and may be used to risk stratify the frequency of monitoring blood test during long term methotrexate treatment.

Developing prediction models to estimate the risk of two survival outcomes both occurring: A comparison of techniques

INTRODUCTION

This study considers the prediction of the time until two survival outcomes have both occurred. We compared a variety of analytical methods motivated by a typical clinical problem of multimorbidity prognosis.

METHODS

We considered five methods: product (multiply marginal risks), dual-outcome (directly model the time until both events occur), multistate models (msm), and a range of copula and frailty models. We assessed calibration and discrimination under a variety of simulated data scenarios, varying outcome prevalence, and the amount of residual correlation. The simulation focused on model misspecification and statistical power. Using data from the Clinical Practice Research Datalink, we compared model performance when predicting the risk of cardiovascular disease and type 2 diabetes both occurring.

RESULTS

Discrimination was similar for all methods. The product method was poorly calibrated in the presence of residual correlation. The msm and dual-outcome models were the most robust to model misspecification but suffered a drop in performance at small sample sizes due to overfitting, which the copula and frailty model were less susceptible to. The copula and frailty model's performance were highly dependent on the underlying data structure. In the clinical example, the product method was poorly calibrated when adjusting for 8 major cardiovascular risk factors.

DISCUSSION

We recommend the dual-outcome method for predicting the risk of two survival outcomes both occurring. It was the most robust to model misspecification, although was also the most prone to overfitting. The clinical example motivates the use of the methods considered in this study.

Overinterpretation of findings in machine learning prediction model studies in oncology: a systematic review

OBJECTIVES

In biomedical research, spin is the overinterpretation of findings, and it is a growing concern. To date, the presence of spin has not been evaluated in prognostic model research in oncology, including studies developing and validating models for individualized risk prediction.

STUDY DESIGN AND SETTING

We conducted a systematic review, searching MEDLINE and EMBASE for oncology-related studies that developed and validated a prognostic model using machine learning published between 1st January, 2019, and 5th September, 2019. We used existing spin frameworks and described areas of highly suggestive spin practices.

RESULTS

We included 62 publications (including 152 developed models; 37 validated models). Reporting was inconsistent between methods and the results in 27% of studies due to additional analysis and selective reporting. Thirty-two studies (out of 36 applicable studies) reported comparisons between developed models in their discussion and predominantly used discrimination measures to support their claims (78%). Thirty-five studies (56%) used an overly strong or leading word in their title, abstract, results, discussion, or conclusion.

CONCLUSION

The potential for spin needs to be considered when reading, interpreting, and using studies that developed and validated prognostic models in oncology. Researchers should carefully report their prognostic model research using words that reflect their actual results and strength of evidence.

Minimum sample size for developing a multivariable prediction model using multinomial logistic regression

AIMS

Multinomial logistic regression models allow one to predict the risk of a categorical outcome with > 2 categories. When developing such a model, researchers should ensure the number of participants (n ) is appropriate relative to the number of events (E k ) and the number of predictor parameters (p k ) for each category k. We propose three criteria to determine the minimum n required in light of existing criteria developed for binary outcomes.

PROPOSED CRITERIA

The first criterion aims to minimise the model overfitting. The second aims to minimise the difference between the observed and adjusted R 2 Nagelkerke. The third criterion aims to ensure the overall risk is estimated precisely. For criterion (i), we show the sample size must be based on the anticipated Cox-snell R 2 of distinct 'one-to-one' logistic regression models corresponding to the sub-models of the multinomial logistic regression, rather than on the overall Cox-snell R 2 of the multinomial logistic regression.

EVALUATION OF CRITERIA

We tested the performance of the proposed criteria (i) through a simulation study and found that it resulted in the desired level of overfitting. Criterion (ii) and (iii) were natural extensions from previously proposed criteria for binary outcomes and did not require evaluation through simulation.

SUMMARY

We illustrated how to implement the sample size criteria through a worked example considering the development of a multinomial risk prediction model for tumour type when presented with an ovarian mass. Code is provided for the simulation and worked example. We will embed our proposed criteria within the pmsampsize R library and Stata modules.

Maternal and child outcomes for pregnant women with pre-existing multiple long-term conditions: protocol for an observational study in the UK

INTRODUCTION

One in five pregnant women has multiple pre-existing long-term conditions in the UK. Studies have shown that maternal multiple long-term conditions are associated with adverse outcomes. This observational study aims to compare maternal and child outcomes for pregnant women with multiple long-term conditions to those without multiple long-term conditions (0 or 1 long-term conditions).

METHODS AND ANALYSIS

Pregnant women aged 15-49 years old with a conception date between 2000 and 2019 in the UK will be included with follow-up till 2019. The data source will be routine health records from all four UK nations (Clinical Practice Research Datalink (England), Secure Anonymised Information Linkage (Wales), Scotland routine health records and Northern Ireland Maternity System) and the Born in Bradford birth cohort. The exposure of two or more pre-existing, long-term physical or mental health conditions will be defined from a list of health conditions predetermined by women and clinicians. The association of maternal multiple long-term conditions with (a) antenatal, (b) peripartum, (c) postnatal and long-term and (d) mental health outcomes, for both women and their children will be examined. Outcomes of interest will be guided by a core outcome set. Comparisons will be made between pregnant women with and without multiple long-term conditions using modified Poisson and Cox regression. Generalised estimating equation will account for the clustering effect of women who had more than one pregnancy episode. Where appropriate, multiple imputation with chained equation will be used for missing data. Federated analysis will be conducted for each dataset and results will be pooled using random-effects meta-analyses.

ETHICS AND DISSEMINATION

Approval has been obtained from the respective data sources in each UK nation. Study findings will be submitted for publications in peer-reviewed journals and presented at key conferences.

Minimal reporting improvement after peer review in reports of COVID-19 prediction models: systematic review

OBJECTIVES

To assess improvement in the completeness of reporting coronavirus (COVID-19) prediction models after the peer review process.

STUDY DESIGN AND SETTING

Studies included in a living systematic review of COVID-19 prediction models, with both preprint and peer-reviewed published versions available, were assessed. The primary outcome was the change in percentage adherence to the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) reporting guidelines between pre-print and published manuscripts.

RESULTS

Nineteen studies were identified including seven (37%) model development studies, two external validations of existing models (11%), and 10 (53%) papers reporting on both development and external validation of the same model. Median percentage adherence among preprint versions was 33% (min-max: 10 to 68%). The percentage adherence of TRIPOD components increased from preprint to publication in 11/19 studies (58%), with adherence unchanged in the remaining eight studies. The median change in adherence was just 3 percentage points (pp, min-max: 0-14 pp) across all studies. No association was observed between the change in percentage adherence and preprint score, journal impact factor, or time between journal submission and acceptance.

CONCLUSIONS

The preprint reporting quality of COVID-19 prediction modeling studies is poor and did not improve much after peer review, suggesting peer review had a trivial effect on the completeness of reporting during the pandemic.

Systematic review identifies the design and methodological conduct of studies on machine learning-based prediction models

BACKGROUND AND OBJECTIVES

We sought to summarize the study design, modelling strategies, and performance measures reported in studies on clinical prediction models developed using machine learning techniques.

METHODS

We search PubMed for articles published between 01/01/2018 and 31/12/2019, describing the development or the development with external validation of a multivariable prediction model using any supervised machine learning technique. No restrictions were made based on study design, data source, or predicted patient-related health outcomes.

RESULTS

We included 152 studies, 58 (38.2% [95% CI 30.8-46.1]) were diagnostic and 94 (61.8% [95% CI 53.9-69.2]) prognostic studies. Most studies reported only the development of prediction models (n = 133, 87.5% [95% CI 81.3-91.8]), focused on binary outcomes (n = 131, 86.2% [95% CI 79.8-90.8), and did not report a sample size calculation (n = 125, 82.2% [95% CI 75.4-87.5]). The most common algorithms used were support vector machine (n = 86/522, 16.5% [95% CI 13.5-19.9]) and random forest (n = 73/522, 14% [95% CI 11.3-17.2]). Values for area under the Receiver Operating Characteristic curve ranged from 0.45 to 1.00. Calibration metrics were often missed (n = 494/522, 94.6% [95% CI 92.4-96.3]).

CONCLUSION

Our review revealed that focus is required on handling of missing values, methods for internal validation, and reporting of calibration to improve the methodological conduct of studies on machine learning-based prediction models.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO, CRD42019161764.

Targeted validation: validating clinical prediction models in their intended population and setting

Clinical prediction models must be appropriately validated before they can be used. While validation studies are sometimes carefully designed to match an intended population/setting of the model, it is common for validation studies to take place with arbitrary datasets, chosen for convenience rather than relevance. We call estimating how well a model performs within the intended population/setting "targeted validation". Use of this term sharpens the focus on the intended use of a model, which may increase the applicability of developed models, avoid misleading conclusions, and reduce research waste. It also exposes that external validation may not be required when the intended population for the model matches the population used to develop the model; here, a robust internal validation may be sufficient, especially if the development dataset was large.

Development and external validation of a risk prediction model for falls in patients with an indication for antihypertensive treatment: retrospective cohort study

OBJECTIVE

To develop and externally validate the STRAtifying Treatments In the multi-morbid Frail elderlY (STRATIFY)-Falls clinical prediction model to identify the risk of hospital admission or death from a fall in patients with an indication for antihypertensive treatment.

DESIGN

Retrospective cohort study.

SETTING

Primary care data from electronic health records contained within the UK Clinical Practice Research Datalink (CPRD).

PARTICIPANTS

Patients aged 40 years or older with at least one blood pressure measurement between 130 mm Hg and 179 mm Hg.

MAIN OUTCOME MEASURE

First serious fall, defined as hospital admission or death with a primary diagnosis of a fall within 10 years of the index date (12 months after cohort entry). Model development was conducted using a Fine-Gray approach in data from CPRD GOLD, accounting for the competing risk of death from other causes, with subsequent recalibration at one, five, and 10 years using pseudo values. External validation was conducted using data from CPRD Aurum, with performance assessed through calibration curves and the observed to expected ratio, C statistic, and D statistic, pooled across general practices, and clinical utility using decision curve analysis at thresholds around 10%.

RESULTS

Analysis included 1 772 600 patients (experiencing 62 691 serious falls) from CPRD GOLD used in model development, and 3 805 366 (experiencing 206 956 serious falls) from CPRD Aurum in the external validation. The final model consisted of 24 predictors, including age, sex, ethnicity, alcohol consumption, living in an area of high social deprivation, a history of falls, multiple sclerosis, and prescriptions of antihypertensives, antidepressants, hypnotics, and anxiolytics. Upon external validation, the recalibrated model showed good discrimination, with pooled C statistics of 0.833 (95% confidence interval 0.831 to 0.835) and 0.843 (0.841 to 0.844) at five and 10 years, respectively. Original model calibration was poor on visual inspection and although this was improved with recalibration, under-prediction of risk remained (observed to expected ratio at 10 years 1.839, 95% confidence interval 1.811 to 1.865). Nevertheless, decision curve analysis suggests potential clinical utility, with net benefit larger than other strategies.

CONCLUSIONS

This prediction model uses commonly recorded clinical characteristics and distinguishes well between patients at high and low risk of falls in the next 1-10 years. Although miscalibration was evident on external validation, the model still had potential clinical utility around risk thresholds of 10% and so could be useful in routine clinical practice to help identify those at high risk of falls who might benefit from closer monitoring or early intervention to prevent future falls. Further studies are needed to explore the appropriate thresholds that maximise the model's clinical utility and cost effectiveness.

Calculating the power to examine treatment-covariate interactions when planning an individual participant data meta-analysis of randomized trials with a binary outcome

Before embarking on an individual participant data meta-analysis (IPDMA) project, researchers and funders need assurance it is worth their time and cost. This should include consideration of how many studies are promising their IPD and, given the characteristics of these studies, the power of an IPDMA including them. Here, we show how to estimate the power of a planned IPDMA of randomized trials to examine treatment-covariate interactions at the participant level (ie, treatment effect modifiers). We focus on a binary outcome with binary or continuous covariates, and propose a three-step approach, which assumes the true interaction size is common to all trials. In step one, the user must specify a minimally important interaction size and, for each trial separately (eg, as obtained from trial publications), the following aggregate data: the number of participants and events in control and treatment groups, the mean and SD for each continuous covariate, and the proportion of participants in each category for each binary covariate. This allows the variance of the interaction estimate to be calculated for each trial, using an analytic solution for Fisher's information matrix from a logistic regression model. Step 2 calculates the variance of the summary interaction estimate from the planned IPDMA (equal to the inverse of the sum of the inverse trial variances from step 1), and step 3 calculates the corresponding power based on a two-sided Wald test. Stata and R code are provided, and two examples given for illustration. Extension to allow for between-study heterogeneity is also considered.

External validation of a prediction model for estimating fat mass in children and adolescents in 19 countries: individual participant data meta-analysis

OBJECTIVE

To evaluate the performance of a UK based prediction model for estimating fat-free mass (and indirectly fat mass) in children and adolescents in non-UK settings.

DESIGN

Individual participant data meta-analysis.

SETTING

19 countries.

PARTICIPANTS

5693 children and adolescents (49.7% boys) aged 4 to 15 years with complete data on the predictors included in the UK based model (weight, height, age, sex, and ethnicity) and on the independently assessed outcome measure (fat-free mass determined by deuterium dilution assessment).

MAIN OUTCOME MEASURES

The outcome of the UK based prediction model was natural log transformed fat-free mass (lnFFM). Predictive performance statistics of R², calibration slope, calibration-in-the-large, and root mean square error were assessed in each of the 19 countries and then pooled through random effects meta-analysis. Calibration plots were also derived for each country, including flexible calibration curves.

RESULTS

The model showed good predictive ability in non-UK populations of children and adolescents, providing R² values of >75% in all countries and >90% in 11 of the 19 countries, and with good calibration (ie, agreement) of observed and predicted values. Root mean square error values (on fat-free mass scale) were <4 kg in 17 of the 19 settings. Pooled values (95% confidence intervals) of R², calibration slope, and calibration-in-the-large were 88.7% (85.9% to 91.4%), 0.98 (0.97 to 1.00), and 0.01 (-0.02 to 0.04), respectively. Heterogeneity was evident in the R² and calibration-in-the-large values across settings, but not in the calibration slope. Model performance did not vary markedly between boys and girls, age, ethnicity, and national income groups. To further improve the accuracy of the predictions, the model equation was recalibrated for the intercept in each setting so that country specific equations are available for future use.

CONCLUSION

The UK based prediction model, which is based on readily available measures, provides predictions of childhood fat-free mass, and hence fat mass, in a range of non-UK settings that explain a large proportion of the variability in observed fat-free mass, and exhibit good calibration performance, especially after recalibration of the intercept for each population. The model demonstrates good generalisability in both low-middle income and high income populations of healthy children and adolescents aged 4-15 years.

Using individual participant data to improve network meta-analysis projects

A network meta-analysis combines the evidence from existing randomised trials about the comparative efficacy of multiple treatments. It allows direct and indirect evidence about each comparison to be included in the same analysis, and provides a coherent framework to compare and rank treatments. A traditional network meta-analysis uses aggregate data (eg, treatment effect estimates and standard errors) obtained from publications or trial investigators. An alternative approach is to obtain, check, harmonise and meta-analyse the individual participant data (IPD) from each trial. In this article, we describe potential advantages of IPD for network meta-analysis projects, emphasising five key benefits: (1) improving the quality and scope of information available for inclusion in the meta-analysis, (2) examining and plotting distributions of covariates across trials (eg, for potential effect modifiers), (3) standardising and improving the analysis of each trial, (4) adjusting for prognostic factors to allow a network meta-analysis of conditional treatment effects and (5) including treatment-covariate interactions (effect modifiers) to allow relative treatment effects to vary by participant-level covariate values (eg, age, baseline depression score). A running theme of all these benefits is that they help examine and reduce heterogeneity (differences in the true treatment effect between trials) and inconsistency (differences in the true treatment effect between direct and indirect evidence) in the network. As a consequence, an IPD network meta-analysis has the potential for more precise, reliable and informative results for clinical practice and even allows treatment comparisons to be made for individual patients and targeted populations conditional on their particular characteristics.

Predicting relapse or recurrence of depression: systematic review of prognostic models

BACKGROUND

Relapse and recurrence of depression are common, contributing to the overall burden of depression globally. Accurate prediction of relapse or recurrence while patients are well would allow the identification of high-risk individuals and may effectively guide the allocation of interventions to prevent relapse and recurrence.

AIMS

To review prognostic models developed to predict the risk of relapse, recurrence, sustained remission, or recovery in adults with remitted major depressive disorder.

METHOD

We searched the Cochrane Library (current issue); Ovid MEDLINE (1946 onwards); Ovid Embase (1980 onwards); Ovid PsycINFO (1806 onwards); and Web of Science (1900 onwards) up to May 2021. We included development and external validation studies of multivariable prognostic models. We assessed risk of bias of included studies using the Prediction model risk of bias assessment tool (PROBAST).

RESULTS

We identified 12 eligible prognostic model studies (11 unique prognostic models): 8 model development-only studies, 3 model development and external validation studies and 1 external validation-only study. Multiple estimates of performance measures were not available and meta-analysis was therefore not necessary. Eleven out of the 12 included studies were assessed as being at high overall risk of bias and none examined clinical utility.

CONCLUSIONS

Due to high risk of bias of the included studies, poor predictive performance and limited external validation of the models identified, presently available clinical prediction models for relapse and recurrence of depression are not yet sufficiently developed for deploying in clinical settings. There is a need for improved prognosis research in this clinical area and future studies should conform to best practice methodological and reporting guidelines.