Overinterpretation of findings in machine learning prediction model studies in oncology: a systematic review

Larger sample sizes are needed when developing a clinical prediction model using machine learning in oncology: methodological systematic review

BACKGROUND AND OBJECTIVES

Having a sufficient sample size is crucial when developing a clinical prediction model. We reviewed details of sample size in studies developing prediction models for binary outcomes using machine learning (ML) methods within oncology and compared the sample size used to develop the models with the minimum required sample size needed when developing a regression-based model (Nmin).

METHODS

We searched the Medline (via OVID) database for studies developing a prediction model using ML methods published in December 2022. We reviewed how sample size was justified. We calculated Nmin, which is the Nmin, and compared this with the sample size that was used to develop the models.

RESULTS

Only one of 36 included studies justified their sample size. We were able to calculate Nmin for 17 (47%) studies. 5/17 studies met Nmin, allowing to precisely estimate the overall risk and minimize overfitting. There was a median deficit of 302 participants with the event (n = 17; range: -21,331 to 2298) when developing the ML models. An additional three out of the 17 studies met the required sample size to precisely estimate the overall risk only.

CONCLUSION

Studies developing a prediction model using ML in oncology seldom justified their sample size and sample sizes were often smaller than Nmin. As ML models almost certainly require a larger sample size than regression models, the deficit is likely larger. We recommend that researchers consider and report their sample size and at least meet the minimum sample size required when developing a regression-based model.

Public Disclosure of Results From Artificial Intelligence/Machine Learning Research in Health Care: Comprehensive Analysis of ClinicalTrials.gov, PubMed, and Scopus Data (2010-2023)

BACKGROUND

Despite the rapid growth of research in artificial intelligence/machine learning (AI/ML), little is known about how often study results are disclosed years after study completion.

OBJECTIVE

We aimed to estimate the proportion of AI/ML research that reported results through ClinicalTrials.gov or peer-reviewed publications indexed in PubMed or Scopus.

METHODS

Using data from the Clinical Trials Transformation Initiative Aggregate Analysis of ClinicalTrials.gov, we identified studies initiated and completed between January 2010 and December 2023 that contained AI/ML-specific terms in the official title, brief summary, interventions, conditions, detailed descriptions, primary outcomes, or keywords. For 842 completed studies, we searched PubMed and Scopus for publications containing study identifiers and AI/ML-specific terms in relevant fields, such as the title, abstract, and keywords. We calculated disclosure rates within 3 years of study completion and median times to disclosure-from the "primary completion date" to the "results first posted date" on ClinicalTrials.gov or the earliest date of journal publication.

RESULTS

When restricted to studies completed before 2021, ensuring at least 3 years of follow-up in which to report results, 7.0% (22/316) disclosed results on ClinicalTrials.gov, 16.5% (52/316) in journal publications, and 20.6% (65/316) through either route within 3 years of completion. Higher disclosure rates were observed for trials: 11.0% (15/136) on ClinicalTrials.gov, 25.0% (34/136) in journal publications, and 30.1% (41/136) through either route. Randomized controlled trials had even higher disclosure rates: 12.2% (9/74) on ClinicalTrials.gov, 31.1% (23/74) in journal publications, and 36.5% (27/74) through either route. Nevertheless, most study findings (79.4%; 251/316) remained undisclosed 3 years after study completion. Trials using randomization (vs nonrandomized) or masking (vs open label) had higher disclosure rates and shorter times to disclosure. Most trials (85%; 305/357) had sample sizes of ≤1000, yet larger trials (n>1000) had higher publication rates (30.8%; 16/52) than smaller trials (n≤1000) (17.4%; 53/305). Hospitals (12.4%; 42/340), academia (15.1%; 39/259), and industry (13.7%; 20/146) published the most. High-income countries accounted for 82.4% (89/108) of all published studies. Of studies with disclosed results, the median times to report through ClinicalTrials.gov and in journal publications were 505 days (IQR 399-676) and 407 days (IQR 257-674), respectively. Open-label trials were common (60%; 214/357). Single-center designs were prevalent in both trials (83.3%; 290/348) and observational studies (82.3%; 377/458).

CONCLUSIONS

For nearly 80% of completed studies, findings remained undisclosed within the 3 years of follow-up, raising questions about the representativeness of publicly available evidence. While methodological rigor was generally associated with higher publication rates, the predominance of single-center designs and high-income countries may limit the generalizability of the results currently accessible.

A systematic review of cardiac surgery clinical prediction models that include intra-operative variables

BACKGROUND

Most cardiac surgery clinical prediction models (CPMs) are developed using pre-operative variables to predict post-operative outcomes. Some CPMs are developed with intra-operative variables, but none are widely used. The objective of this systematic review was to identify CPMs with intra-operative variables that predict short-term outcomes following adult cardiac surgery.

METHODS

Ovid MEDLINE and EMBASE databases were searched from inception to December 2022, for studies developing a CPM with at least one intra-operative variable. Data were extracted using a critical appraisal framework and bias assessment tool. Model performance was analysed using discrimination and calibration measures.

RESULTS

A total of 24 models were identified. Frequent predicted outcomes were acute kidney injury (9/24 studies) and peri-operative mortality (6/24 studies). Frequent pre-operative variables were age (18/24 studies) and creatinine/eGFR (18/24 studies). Common intra-operative variables were cardiopulmonary bypass time (16/24 studies) and transfusion (13/24 studies). Model discrimination was acceptable for all internally validated models (AUC 0.69-0.91). Calibration was poor (15/24 studies) or unreported (8/24 studies). Most CPMs were at a high or indeterminate risk of bias (23/24 models). The added value of intra-operative variables was assessed in six studies with statistically significantly improved discrimination demonstrated in two.

CONCLUSION

Weak reporting and methodological limitations may restrict wider applicability and adoption of existing CPMs that include intra-operative variables. There is some evidence that CPM discrimination is improved with the addition of intra-operative variables. Further work is required to understand the role of intra-operative CPMs in the management of cardiac surgery patients.

A systematic review of machine learning-based prognostic models for acute pancreatitis: Towards improving methods and reporting quality

BACKGROUND

An accurate prognostic tool is essential to aid clinical decision-making (e.g., patient triage) and to advance personalized medicine. However, such a prognostic tool is lacking for acute pancreatitis (AP). Increasingly machine learning (ML) techniques are being used to develop high-performing prognostic models in AP. However, methodologic and reporting quality has received little attention. High-quality reporting and study methodology are critical for model validity, reproducibility, and clinical implementation. In collaboration with content experts in ML methodology, we performed a systematic review critically appraising the quality of methodology and reporting of recently published ML AP prognostic models.

METHODS/FINDINGS

Using a validated search strategy, we identified ML AP studies from the databases MEDLINE and EMBASE published between January 2021 and December 2023. We also searched pre-print servers medRxiv, bioRxiv, and arXiv for pre-prints registered between January 2021 and December 2023. Eligibility criteria included all retrospective or prospective studies that developed or validated new or existing ML models in patients with AP that predicted an outcome following an episode of AP. Meta-analysis was considered if there was homogeneity in the study design and in the type of outcome predicted. For risk of bias (ROB) assessment, we used the Prediction Model Risk of Bias Assessment Tool. Quality of reporting was assessed using the Transparent Reporting of a Multivariable Prediction Model of Individual Prognosis or Diagnosis-Artificial Intelligence (TRIPOD+AI) statement that defines standards for 27 items that should be reported in publications using ML prognostic models. The search strategy identified 6,480 publications of which 30 met the eligibility criteria. Studies originated from China (22), the United States (4), and other (4). All 30 studies developed a new ML model and none sought to validate an existing ML model, producing a total of 39 new ML models. AP severity (23/39) or mortality (6/39) were the most common outcomes predicted. The mean area under the curve for all models and endpoints was 0.91 (SD 0.08). The ROB was high for at least one domain in all 39 models, particularly for the analysis domain (37/39 models). Steps were not taken to minimize over-optimistic model performance in 27/39 models. Due to heterogeneity in the study design and in how the outcomes were defined and determined, meta-analysis was not performed. Studies reported on only 15/27 items from TRIPOD+AI standards, with only 7/30 justifying sample size and 13/30 assessing data quality. Other reporting deficiencies included omissions regarding human-AI interaction (28/30), handling low-quality or incomplete data in practice (27/30), sharing analytical codes (25/30), study protocols (25/30), and reporting source data (19/30).

CONCLUSIONS

There are significant deficiencies in the methodology and reporting of recently published ML based prognostic models in AP patients. These undermine the validity, reproducibility, and implementation of these prognostic models despite their promise of superior predictive accuracy.

REGISTRATION

Research Registry (reviewregistry1727).

Quality and completeness of, and spin in reporting of, pilot and feasibility studies in hip and knee arthroplasty: a protocol for a methodological survey

INTRODUCTION

Pilot or feasibility trials examine the feasibility, viability and recruitment potential of larger, main trials. Specifically, a pilot trial can be instrumental in identifying methodological issues essential to the development of an effective research protocol. However, numerous studies published as pilot or feasibility studies have demonstrated notable inconsistencies in the nature of information reported, resulting in poor-quality and incomplete reporting. It is unclear whether such low quality or incompleteness of reporting is also prevalent in arthroplasty pilot trials.

METHODS AND ANALYSIS

This protocol outlines a methodological survey examining the completeness of reporting among hip and knee arthroplasty pilot trials in accordance with the Consolidated Standards of Reporting Trials (CONSORT) 2010 extension to pilot trials. Secondary objectives include: (1) determining the prevalence of 'spin' practices, defined as: (a) placing a focus on statistical significance rather than feasibility, (b) presenting results that show the trial to be non-feasible as feasible or (c) emphasising the effectiveness or potential intervention benefits rather than feasibility; (2) determining factors associated with incomplete reporting, and 'spin'. A search of PubMed will be conducted for pilot trials in hip or knee arthroplasty published between 01 January 2017 and 31 December 2023. Following screening, appropriate data will be extracted from eligible publications and reported as descriptive statistics, encompassing elements of the CONSORT checklist associated with completeness of reporting. Logistic regression analysis and Poisson regression will be used to analyse factors associated with completeness of reporting and spin.

ETHICS AND DISSEMINATION

This methodological review does not require formal ethical approval, as it will solely involve the use of published and publicly reported literature. The results of this study will be disseminated through submission to peer-reviewed journals and academic conference presentations. Study details will be sent to McMaster University's media coordinators to be shared through the institution's research-focused platforms.

Identifying and Minimizing Incentives for Competing Interests in Sports Medicine Publications

Academics in sports medicine as well as other medical fields are generally expected to publish research and opinions in peer-reviewed journals. The peer-review process is intended to protect against the publication of flawed research and unsubstantiated claims. However, both financial and non-financial competing interests may result in sub-optimal results by affecting investigators, editors, peer reviewers, academic institutions, and publishers. In this article, we focus on the non-financial competing interests created in our current academic system. Because these competing interests are embedded in our current scholastic framework, the potential biases are difficult to quantify. To minimize the effect of these competing interests, we review and highlight some underlying incentives for each stakeholder and some potential solutions to mitigate their effects.

Identifying patients with undiagnosed small intestinal neuroendocrine tumours in primary care using statistical and machine learning: model development and validation study

BACKGROUND

Neuroendocrine tumours (NETs) are increasing in incidence, often diagnosed at advanced stages, and individuals may experience years of diagnostic delay, particularly when arising from the small intestine (SI). Clinical prediction models could present novel opportunities for case finding in primary care.

METHODS

An open cohort of adults (18+ years) contributing data to the Optimum Patient Care Research Database between 1st Jan 2000 and 30th March 2023 was identified. This database collects de-identified data from general practices in the UK. Model development approaches comprised logistic regression, penalised regression, and XGBoost. Performance (discrimination and calibration) was assessed using internal-external cross-validation. Decision analysis curves compared clinical utility.

RESULTS

Of 11.7 million individuals, 382 had recorded SI NET diagnoses (0.003%). The XGBoost model had the highest AUC (0.869, 95% confidence interval [CI]: 0.841-0.898) but was mildly miscalibrated (slope 1.165, 95% CI: 1.088-1.243; calibration-in-the-large 0.010, 95% CI: -0.164 to 0.185). Clinical utility was similar across all models.

DISCUSSION

Multivariable prediction models may have clinical utility in identifying individuals with undiagnosed SI NETs using information in their primary care records. Further evaluation including external validation and health economics modelling may identify cost-effective strategies for case finding for this uncommon tumour.

PET/CT deep learning prognosis for treatment decision support in esophageal squamous cell carcinoma

OBJECTIVES

The clinical decision-making regarding choosing surgery alone (SA) or surgery followed by postoperative adjuvant chemotherapy (SPOCT) in esophageal squamous cell carcinoma (ESCC) remains controversial. We aim to propose a pre-therapy PET/CT image-based deep learning approach to improve the survival benefit and clinical management of ESCC patients.

METHODS

This retrospective multicenter study included 837 ESCC patients from three institutions. Prognostic biomarkers integrating six networks were developed to build an ESCC prognosis (ESCCPro) model and predict the survival probability of ESCC patients treated with SA and SPOCT. Patients who did not undergo surgical resection were in a control group. Overall survival (OS) was the primary end-point event. The expected improvement in survival prognosis with the application of ESCCPro to assign treatment protocols was estimated by comparing the survival of patients in each subgroup. Seven clinicians with varying experience evaluated how ESCCPro performed in assisting clinical decision-making.

RESULTS

In this retrospective multicenter study, patients receiving SA had a median OS 9.2 months longer than controls. No significant differences in survival were found between SA patients with predicted poor outcomes and the controls (p > 0.05). It was estimated that if ESCCPro was used to determine SA and SPOCT eligibility, the median OS in the ESCCPro-recommended SA group and SPOCT group would have been 15.3 months and 24.9 months longer, respectively. In addition, ESCCPro also significantly improved prognosis accuracy, certainty, and the efficiency of clinical experts.

CONCLUSION

ESCCPro assistance improved the survival benefit of ESCC patients and the clinical decision-making among the two treatment approaches.

CRITICAL RELEVANCE STATEMENT

The ESCCPro model for treatment decision-making is promising to improve overall survival in ESCC patients undergoing surgical resection and patients undergoing surgery followed by postoperative adjuvant chemotherapy.

KEY POINTS

ESCC is associated with a poor prognosis and unclear ideal treatments. ESCCPro predicts the survival of patients with ESCC and the expected benefit from SA. ESCCPro improves clinicians' stratification of patients' prognoses.

SPIN-PM: a consensus framework to evaluate the presence of spin in studies on prediction models

OBJECTIVES

To develop a framework to identify and evaluate spin practices and its facilitators in studies on clinical prediction model regardless of the modeling technique.

STUDY DESIGN AND SETTING

We followed a three-phase consensus process: (1) premeeting literature review to generate items to be included; (2) a series of structured meetings to provide comments discussed and exchanged viewpoints on items to be included with a panel of experienced researchers; and (3) postmeeting review on final list of items and examples to be included. Through this iterative consensus process, a framework was derived after all panel's researchers agreed.

RESULTS

This consensus process involved a panel of eight researchers and resulted in SPIN-Prediction Models which consists of two categories of spin (misleading interpretation and misleading transportability), and within these categories, two forms of spin (spin practices and facilitators of spin). We provide criteria and examples.

CONCLUSION

We proposed this guidance aiming to facilitate not only the accurate reporting but also an accurate interpretation and extrapolation of clinical prediction models which will likely improve the reporting quality of subsequent research, as well as reduce research waste.

Artificial Intelligence in Orthodontics: Critical Review

With increasing digitalization in orthodontics, certain orthodontic manufacturing processes such as the fabrication of indirect bonding trays, aligner production, or wire bending can be automated. However, orthodontic treatment planning and evaluation remains a specialist's task and responsibility. As the prediction of growth in orthodontic patients and response to orthodontic treatment is inherently complex and individual, orthodontists make use of features gathered from longitudinal, multimodal, and standardized orthodontic data sets. Currently, these data sets are used by the orthodontist to make informed, rule-based treatment decisions. In research, artificial intelligence (AI) has been successfully applied to assist orthodontists with the extraction of relevant data from such data sets. Here, AI has been applied for the analysis of clinical imagery, such as automated landmark detection in lateral cephalograms but also for evaluation of intraoral scans or photographic data. Furthermore, AI is applied to help orthodontists with decision support for treatment decisions such as the need for orthognathic surgery or for orthodontic tooth extractions. One major challenge in current AI research in orthodontics is the limited generalizability, as most studies use unicentric data with high risks of bias. Moreover, comparing AI across different studies and tasks is virtually impossible as both outcomes and outcome metrics vary widely, and underlying data sets are not standardized. Notably, only few AI applications in orthodontics have reached full clinical maturity and regulatory approval, and researchers in the field are tasked with tackling real-world evaluation and implementation of AI into the orthodontic workflow.

Critical appraisal of machine learning prognostic models for acute pancreatitis: protocol for a systematic review

Acute pancreatitis (AP) is an acute inflammatory disorder that is common, costly, and is increasing in incidence worldwide with over 300,000 hospitalizations occurring yearly in the United States alone. As its course and outcomes vary widely, a critical knowledge gap in the field has been a lack of accurate prognostic tools to forecast AP patients' outcomes. Despite several published studies in the last three decades, the predictive performance of published prognostic models has been found to be suboptimal. Recently, non-regression machine learning models (ML) have garnered intense interest in medicine for their potential for better predictive performance. Each year, an increasing number of AP models are being published. However, their methodologic quality relating to transparent reporting and risk of bias in study design has never been systematically appraised. Therefore, through collaboration between a group of clinicians and data scientists with appropriate content expertise, we will perform a systematic review of papers published between January 2021 and December 2023 containing artificial intelligence prognostic models in AP. To systematically assess these studies, the authors will leverage the CHARMS checklist, PROBAST tool for risk of bias assessment, and the most current version of the TRIPOD-AI. (Research Registry ( http://www.reviewregistry1727 .).

The application and use of artificial intelligence in cancer nursing: A systematic review

PURPOSE

Artificial Intelligence is being applied in oncology to improve patient and service outcomes. Yet, there is a limited understanding of how these advanced computational techniques are employed in cancer nursing to inform clinical practice. This review aimed to identify and synthesise evidence on artificial intelligence in cancer nursing.

METHODS

CINAHL, MEDLINE, PsycINFO, and PubMed were searched using key terms between January 2010 and December 2022. Titles, abstracts, and then full texts were screened against eligibility criteria, resulting in twenty studies being included. Critical appraisal was undertaken, and relevant data extracted and analysed. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines were followed.

RESULTS

Artificial intelligence was used in numerous areas including breast, colorectal, liver, and ovarian cancer care among others. Algorithms were trained and tested on primary and secondary datasets to build predictive models of health problems related to cancer. Studies reported this led to improvements in the accuracy of predicting health outcomes or identifying variables that improved outcome prediction. While nurses led most studies, few deployed an artificial intelligence based digital tool with cancer nurses in a real-world setting as studies largely focused on developing and validating predictive models.

CONCLUSION

Electronic cancer nursing datasets should be established to enable artificial intelligence techniques to be tested and if effective implemented in digital prediction and other AI-based tools. Cancer nurses need more education on machine learning and natural language processing, so they can lead and contribute to artificial intelligence developments in oncology.

Leveraging machine learning to distinguish between bacterial and viral induced pharyngitis using hematological markers: a retrospective cohort study

Accurate differentiation between bacterial and viral-induced pharyngitis is recognized as essential for personalized treatment and judicious antibiotic use. From a cohort of 693 patients with pharyngitis, data from 197 individuals clearly diagnosed with bacterial or viral infections were meticulously analyzed in this study. By integrating detailed hematological insights with several machine learning algorithms, including Random Forest, Neural Networks, Decision Trees, Support Vector Machine, Naive Bayes, and Lasso Regression, for potential biomarkers were identified, with an emphasis being placed on the diagnostic significance of the Monocyte-to-Lymphocyte Ratio. Distinct inflammatory signatures associated with bacterial infections were spotlighted in this study. An innovation introduced in this research was the adaptation of the high-accuracy Lasso Regression model for the TI-84 calculator, with an AUC (95% CI) of 0.94 (0.925-0.955) being achieved. Using this adaptation, pivotal laboratory parameters can be input on-the-spot and infection probabilities can be computed subsequently. This methodology embodies an improvement in diagnostics, facilitating more effective distinction between bacterial and viral infections while fostering judicious antibiotic use.