Developing, validating, updating and judging the impact of prognostic models for respiratory diseases

Temporal validation of machine learning models for pre-eclampsia prediction using routinely collected maternal characteristics: A validation study

BACKGROUND

Pre-eclampsia (PE) contributes to more than one-fourth of all maternal deaths and half a million newborn deaths worldwide every year. Early screening and interventions can reduce PE incidence and related complications. We aim to 1) temporally validate three existing models (two machine learning (ML) and one logistic regression) developed in the same region and 2) compare the performances of the validated ML models with the logistic regression model in PE prediction. This work addresses a gap in the literature by undertaking a comprehensive evaluation of existing risk prediction models, which is an important step to advancing this field.

METHODS

We obtained a dataset of routinely collected antenatal data from three maternity hospitals in South-East Melbourne, Australia, extracted between July 2021 and December 2022. We temporally validated three existing models: extreme gradient boosting (XGBoost, 'model 1'), random forest ('model 2') ML models, and a logistic regression model ('model 3'). Area under the receiver-operating characteristic (ROC) curve (AUC) was evaluated discrimination performance, and calibration was assessed. The AUCs were compared using the 'bootstrapping' test.

RESULTS

The temporal evaluation dataset consisted of 12,549 singleton pregnancies, of which 431 (3.43 %, 95 % confidence interval (CI) 3.13-3.77) developed PE. The characteristics of the temporal evaluation dataset were similar to the original development dataset. The XGBoost 'model 1' and the logistic regression 'model 3' exhibited similar discrimination performance with an AUC of 0.75 (95 % CI 0.73-0.78) and 0.76 (95 % CI 0.74-0.78), respectively. The random forest 'model 2' showed a discrimination performance of AUC 0.71 (95 % CI 0.69-0.74). Model 3 showed perfect calibration performance with a slope of 1.02 (95 % CI 0.92-1.12). Models 1 and 2 showed a calibration slope of 1.15 (95 % CI 1.03-1.28) and 0.62 (95 % CI 0.54-0.70), respectively. Compared to the original development models, the temporally validated models 1 and 3 showed stable discrimination performance, whereas model 2 showed significantly lower discrimination performance. Models 1 and 3 showed better clinical net benefits between 3 % and 22 % threshold probabilities than default strategies.

CONCLUSIONS

During temporal validation of PE prediction models, logistic regression and XGBoost models exhibited stable prediction performance; however, both ML models did not outperform the logistic regression model. To facilitate insights into interpretability and deployment, the logistic regression model could be integrated into routine practice as a first-step in a two-stage screening approach to identify a higher-risk woman for further second stage screening with a more accurate test.

Multimodal machine learning to predict surgical site infection with healthcare workload impact assessment

Remote monitoring is essential for healthcare digital transformation, however, this poses greater burdens on healthcare providers to review and respond as the data collected expands. This study developed a multimodal neural network to automate assessments of patient-generated data from remote postoperative wound monitoring. Two interventional studies including adult gastrointestinal surgery patients collected wound images and patient-reported outcome measures (PROMs) for 30-days postoperatively. Neural networks for PROMs and images were combined to predict surgical site infection (SSI) diagnosis within 48 h. The multimodal neural network model to predict confirmed SSI within 48 h remained comparable to clinician triage (0.762 [0.690-0.835] vs 0.777 [0.721-0.832]), with an excellent performance on external validation. Simulated usage indicated an 80% reduction in staff time (51.5 to 9.1 h) without compromising diagnostic accuracy. This multimodal approach can effectively support remote monitoring, alleviating provider burden while ensuring high-quality postoperative care.

Finding Consensus on Trust in AI in Health Care: Recommendations From a Panel of International Experts

BACKGROUND

The integration of artificial intelligence (AI) into health care has become a crucial element in the digital transformation of health systems worldwide. Despite the potential benefits across diverse medical domains, a significant barrier to the successful adoption of AI systems in health care applications remains the prevailing low user trust in these technologies. Crucially, this challenge is exacerbated by the lack of consensus among experts from different disciplines on the definition of trust in AI within the health care sector.

OBJECTIVE

We aimed to provide the first consensus-based analysis of trust in AI in health care based on an interdisciplinary panel of experts from different domains. Our findings can be used to address the problem of defining trust in AI in health care applications, fostering the discussion of concrete real-world health care scenarios in which humans interact with AI systems explicitly.

METHODS

We used a combination of framework analysis and a 3-step consensus process involving 18 international experts from the fields of computer science, medicine, philosophy of technology, ethics, and social sciences. Our process consisted of a synchronous phase during an expert workshop where we discussed the notion of trust in AI in health care applications, defined an initial framework of important elements of trust to guide our analysis, and agreed on 5 case studies. This was followed by a 2-step iterative, asynchronous process in which the authors further developed, discussed, and refined notions of trust with respect to these specific cases.

RESULTS

Our consensus process identified key contextual factors of trust, namely, an AI system's environment, the actors involved, and framing factors, and analyzed causes and effects of trust in AI in health care. Our findings revealed that certain factors were applicable across all discussed cases yet also pointed to the need for a fine-grained, multidisciplinary analysis bridging human-centered and technology-centered approaches. While regulatory boundaries and technological design features are critical to successful AI implementation in health care, ultimately, communication and positive lived experiences with AI systems will be at the forefront of user trust. Our expert consensus allowed us to formulate concrete recommendations for future research on trust in AI in health care applications.

CONCLUSIONS

This paper advocates for a more refined and nuanced conceptual understanding of trust in the context of AI in health care. By synthesizing insights into commonalities and differences among specific case studies, this paper establishes a foundational basis for future debates and discussions on trusting AI in health care.

Updating methods for artificial intelligence-based clinical prediction models: a scoping review

OBJECTIVES

To give an overview of methods for updating artificial intelligence (AI)-based clinical prediction models based on new data.

STUDY DESIGN AND SETTING

We comprehensively searched Scopus and Embase up to August 2022 for articles that addressed developments, descriptions, or evaluations of prediction model updating methods. We specifically focused on articles in the medical domain involving AI-based prediction models that were updated based on new data, excluding regression-based updating methods as these have been extensively discussed elsewhere. We categorized and described the identified methods used to update the AI-based prediction model as well as the use cases in which they were used.

RESULTS

We included 78 articles. The majority of the included articles discussed updating for neural network methods (93.6%) with medical images as input data (65.4%). In many articles (51.3%) existing, pretrained models for broad tasks were updated to perform specialized clinical tasks. Other common reasons for model updating were to address changes in the data over time and cross-center differences; however, more unique use cases were also identified, such as updating a model from a broad population to a specific individual. We categorized the identified model updating methods into four categories: neural network-specific methods (described in 92.3% of the articles), ensemble-specific methods (2.5%), model-agnostic methods (9.0%), and other (1.3%). Variations of neural network-specific methods are further categorized based on the following: (1) the part of the original neural network that is kept, (2) whether and how the original neural network is extended with new parameters, and (3) to what extent the original neural network parameters are adjusted to the new data. The most frequently occurring method (n = 30) involved selecting the first layer(s) of an existing neural network, appending new, randomly initialized layers, and then optimizing the entire neural network.

CONCLUSION

We identified many ways to adjust or update AI-based prediction models based on new data, within a large variety of use cases. Updating methods for AI-based prediction models other than neural networks (eg, random forest) appear to be underexplored in clinical prediction research.

PLAIN LANGUAGE SUMMARY

AI-based prediction models are increasingly used in health care, helping clinicians with diagnosing diseases, guiding treatment decisions, and informing patients. However, these prediction models do not always work well when applied to hospitals, patient populations, or times different from those used to develop the models. Developing new models for every situation is neither practical nor desired, as it wastes resources, time, and existing knowledge. A more efficient approach is to adjust existing models to new contexts ('updating'), but there is limited guidance on how to do this for AI-based clinical prediction models. To address this, we reviewed 78 studies in detail to understand how researchers are currently updating AI-based clinical prediction models, and the types of situations in which these updating methods are used. Our findings provide a comprehensive overview of the available methods to update existing models. This is intended to serve as guidance and inspiration for researchers. Ultimately, this can lead to better reuse of existing models and improve the quality and efficiency of AI-based prediction models in health care.

Precautions to Consider in the Analysis of Prognostic and Predictive Indices

Understanding the differences between prognostic and predictive indices is imperative for medical research advances. We have developed a new prognostic measure that will identify the strengths, limitations, and potential applications in clinical practice.

An interrater reliability analysis of preoperative mortality risk calculators used for elective high-risk noncardiac surgical patients shows poor to moderate reliability

BACKGROUND

Multiple preoperative calculators are available online to predict preoperative mortality risk for noncardiac surgical patients. However, it is currently unknown how these risk calculators perform across different raters. The current study investigated the interrater reliability of three preoperative mortality risk calculators in an elective high-risk noncardiac surgical patient population to evaluate if these calculators can be safely used for identification of high-risk noncardiac surgical patients for a preoperative multidisciplinary team discussion.

METHODS

Five anesthesiologists assessed the preoperative mortality risk of 34 high-risk patients using the preoperative score to calculate postoperative mortality risks (POSPOM), the American College of Surgeons surgical risk calculator (SRC), and the surgical outcome risk tool (SORT). In total, 170 calculations per calculator were gathered.

RESULTS

Interrater reliability was poor for SORT (ICC (C.I. 95%) = 0.46 (0.30-0.63)) and moderate for SRC (ICC = 0.65 (0.51-0.78)) and POSPOM (ICC = 0.63 (0.49-0.77). The absolute range of calculated mortality risk was 0.2-72% for POSPOM, 0-36% for SRC, and 0.4-17% for SORT. The coefficient of variation increased in higher risk classes for POSPOM and SORT. The extended Bland-Altman limits of agreement suggested that all raters contributed to the variation in calculated risks.

CONCLUSION

The current results indicate that the preoperative risk calculators POSPOM, SRC, and SORT exhibit poor to moderate interrater reliability. These calculators are not sufficiently accurate for clinical identification and preoperative counseling of high-risk surgical patients. Clinicians should be trained in using mortality risk calculators. Also, clinicians should be cautious when using predicted mortality estimates from these calculators to identify high-risk noncardiac surgical patients for elective surgery.

Planned but ever published? A retrospective analysis of clinical prediction model studies registered on clinicaltrials.gov since 2000

OBJECTIVES

To describe the characteristics and publication outcomes of clinical prediction model studies registered on clinicaltrials.gov since 2000.

STUDY DESIGN AND SETTING

Observational studies registered on clinicaltrials.gov between January 1, 2000, and March 2, 2022, describing the development of a new clinical prediction model or the validation of an existing model for predicting individual-level prognostic or diagnostic risk were analyzed. Eligible clinicaltrials.gov records were classified by modeling study type (development, validation) and the model outcome being predicted (prognostic, diagnostic). Recorded characteristics included study status, sample size information, Medical Subject Headings, and plans to share individual participant data. Publication outcomes were analyzed by linking National Clinical Trial numbers for eligible records with PubMed abstracts.

RESULTS

Nine hundred twenty-eight records were analyzed from a possible 89,896 observational study records. Publications searches found 170 matching peer-reviewed publications for 137 clinicaltrials.gov records. The estimated proportion of records with 1 or more matching publications after accounting for time since study start was 2.8% at 2 years (95% CI: 1.7%, 3.9%), 12.3% at 5 years (9.8% to 14.9%) and 27% at 10 years (23% to 33%). Stratifying records by study start year indicated that publication proportions improved over time. Records tended to prioritize the development of new prediction models over the validation of existing models (76%; 704/928 vs. 24%; 182/928). At the time of download, 27% of records were marked as complete, 35% were still recruiting, and 14.7% had unknown status. Only 7.4% of records stated plans to share individual participant data.

CONCLUSION

Published clinical prediction model studies are only a fraction of overall research efforts, with many studies planned but not completed or published. Improving the uptake of study preregistration and follow-up will increase the visibility of planned research. Introducing additional registry features and guidance may improve the identification of clinical prediction model studies posted to clinical registries.

The PROgnostic ModEl for chronic lung disease (PRO-MEL): development and temporal validation

BACKGROUND

Patients with chronic lung diseases (CLDs), defined as progressive and life-limiting respiratory conditions, experience a heavy symptom burden as the conditions become more advanced, but palliative referral rates are low and late. Prognostic tools can help clinicians identify CLD patients at high risk of deterioration for needs assessments and referral to palliative care. As current prognostic tools may not generalize well across all CLD conditions, we aim to develop and validate a general model to predict one-year mortality in patients presenting with any CLD.

METHODS

A retrospective cohort study of patients with a CLD diagnosis at a public hospital from July 2016 to October 2017 was conducted. The outcome of interest was all-cause mortality within one-year of diagnosis. Potential prognostic factors were identified from reviews of prognostic studies in CLD, and data was extracted from electronic medical records. Missing data was imputed using multiple imputation by chained equations. Logistic regression models were developed using variable selection methods and validated in patients seen from January 2018 to December 2019. Discriminative ability, calibration and clinical usefulness of the model was assessed. Model coefficients and performance were pooled across all imputed datasets and reported.

RESULTS

Of the 1000 patients, 122 (12.2%) died within one year. Patients had chronic obstructive pulmonary disease or emphysema (55%), bronchiectasis (38%), interstitial lung diseases (12%), or multiple diagnoses (6%). The model selected through forward stepwise variable selection had the highest AUC (0.77 (0.72-0.82)) and consisted of ten prognostic factors. The model AUC for the validation cohort was 0.75 (0.70, 0.81), and the calibration intercept and slope were - 0.14 (-0.54, 0.26) and 0.74 (0.53, 0.95) respectively. Classifying patients with a predicted risk of death exceeding 0.30 as high risk, the model would correctly identify 3 out 10 decedents and 9 of 10 survivors.

CONCLUSIONS

We developed and validated a prognostic model for one-year mortality in patients with CLD using routinely available administrative data. The model will support clinicians in identifying patients across various CLD etiologies who are at risk of deterioration for a basic palliative care assessment to identify unmet needs and trigger an early referral to palliative medicine.

TRIAL REGISTRATION

Not applicable (retrospective study).

Preoperative Atelectasis in Patients with Obesity Undergoing Bariatric Surgery: A Cross-Sectional Study

BACKGROUND

Pulmonary atelectasis is present even before surgery in patients with obesity. We aimed to estimate the prevalence and extension of preoperative atelectasis in patients with obesity undergoing bariatric surgery and to determine if variation in preoperative Spo2 values in the seated position at room air is explained by the extent of atelectasis coverage in the supine position.

METHODS

This was a cross-sectional study in a single center specialized in laparoscopic bariatric surgery. Preoperative chest computed tomographies were reassessed by a senior radiologist to quantify the extent of atelectasis coverage as a percentage of total lung volume. Patients were classified as having atelectasis when the affection was ≥2.5%, to estimate the prevalence of atelectasis. Crude and adjusted prevalence ratios (aPRs) and odds ratios (aORs) were obtained to assess the relative prevalence of atelectasis and percentage coverage, respectively, with increasing obesity category. Inverse probability weighting was used to assess the total, direct (not mediated), and indirect (mediated through atelectasis) effects of body mass index (BMI) on preoperative Spo2, and to quantify the magnitude of mediation (proportion mediated). E-values were calculated, to represent the minimum magnitude of association that an unmeasured confounder with the same directionality of the effect should have to drive the observed point estimates or lower confidence intervals (CIs) to 1, respectively.

RESULTS

In 236 patients with a median BMI of 40.3 kg/m2 (interquartile range [IQR], 34.6-46.0, range: 30.0-77.3), the overall prevalence of atelectasis was 32.6% (95% CI, 27.0-38.9) and by BMI category: 30 to 35 kg/m2, 12.7% (95% CI, 6.1-24.4); 35 to 40 kg/m2, 28.3% (95% CI, 17.2-42.6); 40 to 45 kg/m2, 12.3% (95% CI, 5.5-24.3); 45 to 50 kg/m2, 48.4% (95% CI, 30.6-66.6); and ≥50 units, 100% (95% CI, 86.7-100). Compared to the 30 to 35 kg/m2 group, only the categories with BMI ≥45 kg/m2 had significantly higher relative prevalence of atelectasis-45 to 50 kg/m2, aPR = 3.52 (95% CI, 1.63-7.61, E-value lower bound: 2.64) and ≥50 kg/m2, aPR = 8.0 (95% CI, 4.22-15.2, E-value lower bound: 7.91)-and higher odds of greater atelectasis percentage coverage: 45-50 kg/m2, aOR = 7.5 (95% CI, 2.7-20.9) and ≥50 kg/m2, aOR = 91.5 (95% CI, 30.0-279.3). Atelectasis percent alone explained 70.2% of the variation in preoperative Spo2. The proportion of the effect of BMI on preoperative Spo2 values <96% mediated through atelectasis was 81.5% (95% CI, 56.0-100).

CONCLUSIONS

The prevalence and extension of atelectasis increased with higher BMI, being significantly higher at BMI ≥45 kg/m2. Preoperative atelectasis mediated the effect of BMI on Spo2 at room air in the seated position.

Don't be misled: 3 misconceptions about external validation of clinical prediction models

Clinical prediction models provide risks of health outcomes that can inform patients and support medical decisions. However, most models never make it to actual implementation in practice. A commonly heard reason for this lack of implementation is that prediction models are often not externally validated. While we generally encourage external validation, we argue that an external validation is often neither sufficient nor required as an essential step before implementation. As such, any available external validation should not be perceived as a license for model implementation. We clarify this argument by discussing 3 common misconceptions about external validation. We argue that there is not one type of recommended validation design, not always a necessity for external validation, and sometimes a need for multiple external validations. The insights from this paper can help readers to consider, design, interpret, and appreciate external validation studies.

The Improved Kidney Risk Score in ANCA-Associated Vasculitis for Clinical Practice and Trials

SIGNIFICANCE STATEMENT

Reliable prediction tools are needed to personalize treatment in ANCA-associated GN. More than 1500 patients were collated in an international longitudinal study to revise the ANCA kidney risk score. The score showed satisfactory performance, mimicking the original study (Harrell's C=0.779). In the development cohort of 959 patients, no additional parameters aiding the tool were detected, but replacing the GFR with creatinine identified an additional cutoff. The parameter interstitial fibrosis and tubular atrophy was modified to allow wider access, risk points were reweighted, and a fourth risk group was created, improving predictive ability (C=0.831). In the validation, the new model performed similarly well with excellent calibration and discrimination ( n =480, C=0.821). The revised score optimizes prognostication for clinical practice and trials.

BACKGROUND

Reliable prediction tools are needed to personalize treatment in ANCA-associated GN. A retrospective international longitudinal cohort was collated to revise the ANCA renal risk score.

METHODS

The primary end point was ESKD with patients censored at last follow-up. Cox proportional hazards were used to reweight risk factors. Kaplan-Meier curves, Harrell's C statistic, receiver operating characteristics, and calibration plots were used to assess model performance.

RESULTS

Of 1591 patients, 1439 were included in the final analyses, 2:1 randomly allocated per center to development and validation cohorts (52% male, median age 64 years). In the development cohort ( n =959), the ANCA renal risk score was validated and calibrated, and parameters were reinvestigated modifying interstitial fibrosis and tubular atrophy allowing semiquantitative reporting. An additional cutoff for kidney function (K) was identified, and serum creatinine replaced GFR (K0: <250 µ mol/L=0, K1: 250-450 µ mol/L=4, K2: >450 µ mol/L=11 points). The risk points for the percentage of normal glomeruli (N) and interstitial fibrosis and tubular atrophy (T) were reweighted (N0: >25%=0, N1: 10%-25%=4, N2: <10%=7, T0: none/mild or <25%=0, T1: ≥ mild-moderate or ≥25%=3 points), and four risk groups created: low (0-4 points), moderate (5-11), high (12-18), and very high (21). Discrimination was C=0.831, and the 3-year kidney survival was 96%, 79%, 54%, and 19%, respectively. The revised score performed similarly well in the validation cohort with excellent calibration and discrimination ( n =480, C=0.821).

CONCLUSIONS

The updated score optimizes clinicopathologic prognostication for clinical practice and trials.

National Multi-Institutional Validation of a Surgical Transfusion Risk Prediction Model

BACKGROUND

Accurate estimation of surgical transfusion risk is important for many aspects of surgical planning, yet few methods for estimating are available for estimating such risk. There is a need for reliable validated methods for transfusion risk stratification to support effective perioperative planning and resource stewardship.

STUDY DESIGN

This study was conducted using the American College of Surgeons NSQIP datafile from 2019. S-PATH performance was evaluated at each contributing hospital, with and without hospital-specific model tuning. Linear regression was used to assess the relationship between hospital characteristics and area under the receiver operating characteristic (AUROC) curve.

RESULTS

A total of 1,000,927 surgical cases from 414 hospitals were evaluated. Aggregate AUROC was 0.910 (95% CI 0.904 to 0.916) without model tuning and 0.925 (95% CI 0.919 to 0.931) with model tuning. AUROC varied across individual hospitals (median 0.900, interquartile range 0.849 to 0.944), but no statistically significant relationships were found between hospital-level characteristics studied and model AUROC.

CONCLUSIONS

S-PATH demonstrated excellent discriminative performance, although there was variation across hospitals that was not well-explained by hospital-level characteristics. These results highlight the S-PATH's viability as a generalizable surgical transfusion risk prediction tool.

Prognostic models for surgical-site infection in gastrointestinal surgery: systematic review

BACKGROUND

Identification of patients at high risk of surgical-site infection may allow clinicians to target interventions and monitoring to minimize associated morbidity. The aim of this systematic review was to identify and evaluate prognostic tools for the prediction of surgical-site infection in gastrointestinal surgery.

METHODS

This systematic review sought to identify original studies describing the development and validation of prognostic models for 30-day SSI after gastrointestinal surgery (PROSPERO: CRD42022311019). MEDLINE, Embase, Global Health, and IEEE Xplore were searched from 1 January 2000 to 24 February 2022. Studies were excluded if prognostic models included postoperative parameters or were procedure specific. A narrative synthesis was performed, with sample-size sufficiency, discriminative ability (area under the receiver operating characteristic curve), and prognostic accuracy compared.

RESULTS

Of 2249 records reviewed, 23 eligible prognostic models were identified. A total of 13 (57 per cent) reported no internal validation and only 4 (17 per cent) had undergone external validation. Most identified operative contamination (57 per cent, 13 of 23) and duration (52 per cent, 12 of 23) as important predictors; however, there remained substantial heterogeneity in other predictors identified (range 2-28). All models demonstrated a high risk of bias due to the analytic approach, with overall low applicability to an undifferentiated gastrointestinal surgical population. Model discrimination was reported in most studies (83 per cent, 19 of 23); however, calibration (22 per cent, 5 of 23) and prognostic accuracy (17 per cent, 4 of 23) were infrequently assessed. Of externally validated models (of which there were four), none displayed 'good' discrimination (area under the receiver operating characteristic curve greater than or equal to 0.7).

CONCLUSION

The risk of surgical-site infection after gastrointestinal surgery is insufficiently described by existing risk-prediction tools, which are not suitable for routine use. Novel risk-stratification tools are required to target perioperative interventions and mitigate modifiable risk factors.

Fixing the Leaky Pipe: How to Improve the Uptake of Patient-Reported Outcomes-Based Prognostic and Predictive Models in Cancer Clinical Practice

PURPOSE

This discussion paper outlines challenges and proposes solutions for successfully implementing prediction models that incorporate patient-reported outcomes (PROs) in cancer practice.

METHODS

We organized a full-day multidisciplinary meeting of people with expertise in cancer care delivery, PRO collection, PRO use in prediction modeling, computing, implementation, and decision science. The discussions presented here focused on identifying challenges to the development, implementation and use of prediction models incorporating PROs, and suggesting possible solutions.

RESULTS

Specific challenges and solutions were identified across three broad areas. (1) Understanding decision making and implementation: necessitating multidisciplinary collaboration in the early stages and throughout; early stakeholder engagement to define the decision problem and ensure acceptability of PROs in prediction; understanding patient/clinician interpretation of PRO predictions and uncertainty to optimize prediction impact; striving for model integration into existing electronic health records; and early regulatory alignment. (2) Recognizing the limitations to PRO collection and their impact on prediction: incorporating validated, clinically important PROs to maximize model generalizability and clinical engagement; and minimizing missing PRO data (resulting from both structural digital exclusion and time-varying factors) to avoid exacerbating existing inequalities. (3) Statistical and modeling challenges: incorporating statistical methods to address missing data; ensuring predictive modeling recognizes complex causal relationships; and considering temporal and geographic recalibration so that model predictions reflect the relevant population.

CONCLUSION

Developing and implementing PRO-based prediction models in cancer care requires extensive multidisciplinary working from the earliest stages, recognition of implementation challenges because of PRO collection and model presentation, and robust statistical methods to manage missing data, causality, and calibration. Prediction models incorporating PROs should be viewed as complex interventions, with their development and impact assessment carried out to reflect this.

Development and internal validation of a prediction model for the presence of tinnitus in a Dutch population-based cohort

Objectives

In this study we aim to develop and internally validate a prediction model on tinnitus experience in a representative sample of the Dutch general population.

Methods

We developed a multivariable prediction model using elastic net logistic regression with data from the Dutch Lifelines Cohort Study. This is a multigenerational cohort study on adults who are located in the northern parts of the Netherlands. The model was internally validated using 10-fold cross-validation. The outcome of the model was tinnitus presence, for which we used 24 candidate predictors on different domains (among others demographic, hearing specific, and mental health variables). We assessed the overall predictive performance, discrimination, and calibration of the model.

Results

Data on 122.884 different participants were included, of which 7,965 (6.5%, 0 missing) experienced tinnitus. Nine variables were included in the final model: sex, hearing aids, hearing limitations, arterial blood pressure, quality of sleep, general health, symptom checklist of somatic complaints, cardiovascular risk factors, and age. In the final model, the Brier score was 0.056 and 0.787 in internal validation.

Conclusion

We developed and internally validated a prediction model on tinnitus presence in a multigenerational cohort of the Dutch general population. From the 24 candidate predictors, the final model included nine predictors.

Vital signs-based deterioration prediction model assumptions can lead to losses in prediction performance

OBJECTIVE

Vital signs-based models are complicated by repeated measures per patient and frequently missing data. This paper investigated the impacts of common vital signs modeling assumptions during clinical deterioration prediction model development.

STUDY DESIGN AND SETTING

Electronic medical record (EMR) data from five Australian hospitals (1 January 2019-31 December 2020) were used. Summary statistics for each observation's prior vital signs were created. Missing data patterns were investigated using boosted decision trees, then imputed with common methods. Two example models predicting in-hospital mortality were developed, as follows: logistic regression and eXtreme Gradient Boosting. Model discrimination and calibration were assessed using the C-statistic and nonparametric calibration plots.

RESULTS

The data contained 5,620,641 observations from 342,149 admissions. Missing vitals were associated with observation frequency, vital sign variability, and patient consciousness. Summary statistics improved discrimination slightly for logistic regression and markedly for eXtreme Gradient Boosting. Imputation method led to notable differences in model discrimination and calibration. Model calibration was generally poor.

CONCLUSION

Summary statistics and imputation methods can improve model discrimination and reduce bias during model development, but it is questionable whether these differences are clinically significant. Researchers should consider why data are missing during model development and how this may impact clinical utility.

Digital Transformation of Cancer Care in the Era of Big Data, Artificial Intelligence and Data-Driven Interventions: Navigating the Field

OBJECTIVES

To navigate the field of digital cancer care and define and discuss key aspects and applications of big data analytics, artificial intelligence (AI), and data-driven interventions.

DATA SOURCES

Peer-reviewed scientific publications and expert opinion.

CONCLUSION

The digital transformation of cancer care, enabled by big data analytics, AI, and data-driven interventions, presents a significant opportunity to revolutionize the field. An increased understanding of the lifecycle and ethics of data-driven interventions will enhance development of innovative and applicable products to advance digital cancer care services.

IMPLICATIONS FOR NURSING PRACTICE

As digital technologies become integrated into cancer care, nurse practitioners and scientists will be required to increase their knowledge and skills to effectively use these tools to the patient's benefit. An enhanced understanding of the core concepts of AI and big data, confident use of digital health platforms, and ability to interpret the outputs of data-driven interventions are key competencies. Nurses in oncology will play a crucial role in patient education around big data and AI, with a focus on addressing any arising questions, concerns, or misconceptions to foster trust in these technologies. Successful integration of data-driven innovations into oncology nursing practice will empower practitioners to deliver more personalized, effective, and evidence-based care.

Machine Learning and Statistics in Clinical Research Articles-Moving Past the False Dichotomy

This Viewpoint describes the false dichotomy between statistics and machine learning and suggests considerations in building and evaluating clinical prediction models.

Prediction Models for Tinnitus Presence and the Impact of Tinnitus on Daily Life: A Systematic Review

The presence of tinnitus does not necessarily imply associated suffering. Prediction models on the impact of tinnitus on daily life could aid medical professionals to direct specific medical resources to those (groups of) tinnitus patients with specific levels of impact. Models of tinnitus presence could possibly identify risk factors for tinnitus. We systematically searched the PubMed and EMBASE databases for articles published up to January 2021. We included all studies that reported on multivariable prediction models for tinnitus presence or the impact of tinnitus on daily life. Twenty-one development studies were included, with a total of 31 prediction models. Seventeen studies made a prediction model for the impact of tinnitus on daily life, three studies made a prediction model for tinnitus presence and one study made models for both. The risk of bias was high and reporting was poor in all studies. The most used predictors in the final impact on daily life models were depression- or anxiety-associated questionnaire scores. Demographic predictors were most common in final presence models. No models were internally or externally validated. All published prediction models were poorly reported and had a high risk of bias. This hinders the usability of the current prediction models. Methodological guidance is available for the development and validation of prediction models. Researchers should consider the importance and clinical relevance of the models they develop and should consider validation of existing models before developing new ones.