Extending the c-statistic to nominal polytomous outcomes: the Polytomous Discrimination Index

Diagnostic accuracy validation study of the M6 model without initial serum progesterone (M6NP) in triage of pregnancy of unknown location

OBJECTIVES

The M6 prediction model stratifies the risk of development of ectopic pregnancy (EP) for women with pregnancy of unknown location (PUL) into low risk or high risk, using human chorionic gonadotrophin (hCG) and progesterone levels at the initial visit to a gynaecological emergency room and hCG level at 48 h. This study evaluated a second model, the M6NP model, which does not include the progesterone level at the initial visit. The main aim of this study was to validate the diagnostic accuracy of the M6NP model in a population of French women.

STUDY DESIGN

Between January and December 2021, all women with an hCG measurement from the gynaecological emergency department of a teaching hospital were screened for inclusion in this study. Women with a pregnancy location determined before or at the second visit were excluded. The diagnostic test was based on logistic regression of the M6NP model, with classification into two groups: high risk of EP (≥5%) and low risk of EP (<5%). The reference test was the final outcome based on clinical, biological and sonographic results: failed PUL (FPUL), intrauterine pregnancy (IUP) or EP. Diagnostic performance for risk prediction of EP, and also IUP and FPUL, was calculated.

RESULTS

In total, 759 women with possible PUL were identified. After screening, 341 women with PUL were included in the main analysis. Of these, 186 (54.5%) were classified as low risk, including three (1.6%) with a final outcome of EP. The remaining 155 women with PUL were classified as high risk, of whom 60 (38.7%), 66 (42.8%) and 29 (18.7%) had a final outcome of FPUL, IUP and EP, respectively. Of the 32 women with PUL with a final outcome of EP, 29 (90.6%) were classified as high risk and three (9.4%) were classified as low risk. Therefore, the performance of the M6NP model to predict EP had a negative predictive value of 98.4%, a positive predictive value of 18.7%, sensitivity of 90.6% and specificity of 59.2%. If the prediction model had been used, it is estimated that 4.5 visits per patient could have been prevented.

CONCLUSION

The M6NP model could be used safely in the French population for risk stratification of PUL. Its use in clinical practice should result in a substantial reduction in the number of visits to a gynaecological emergency room.

The Clinical Relevance of the Infiltrating Immune Cell Composition in Kidney Transplant Rejection

Key Points

Background

The link between the histology of kidney transplant rejection, especially antibody-mediated rejection, T-cell–mediated rejection, and mixed rejection, and the types of infiltrating immune cells is currently not well charted. Cost and technical complexity of single-cell analysis hinder large-scale studies of the relationship between cell infiltrate profiles and histological heterogeneity.

Methods

In this cross-sectional study, we assessed the composition of nine intragraft immune cell types by using a validated kidney transplant–specific signature matrix for deconvolution of bulk transcriptomics in three different kidney transplant biopsy datasets (N=403, N=224, N=282). The association and discrimination of the immune cell types with the Banff histology and the association with graft failure were assessed individually and with multivariable models. Unsupervised clustering algorithms were applied on the overall immune cell composition and compared with the Banff phenotypes.

Results

Banff-defined rejection was related to high presence of CD8+ effector T cells, natural killer cells, monocytes/macrophages, and, to a lesser extent, B cells, whereas CD4+ memory T cells were lower in rejection compared with no rejection. Estimated intragraft effector memory–expressing CD45RA (TEMRA) CD8+ T cells were strongly and consistently associated with graft failure. The large heterogeneity in immune cell composition across rejection types prevented supervised and unsupervised methods to accurately recover the Banff phenotypes solely on the basis of immune cell estimates. The lack of correlation between immune cell composition and Banff-defined rejection types was validated using multiplex immunohistochemistry.

Conclusions

Although some specific cell types (FCGR3A + myeloid cells, CD14 + monocytes/macrophages, and NK cells) partly discriminated between rejection phenotypes, the overall estimated immune cell composition of kidney transplants was ill related to main Banff-defined rejection categories and added to the Banff lesion scoring and evaluation of rejection severity. The estimated intragraft CD8temra cells bore strong and consistent association with graft failure and were independent of Banff-grade rejection.

Developing and externally validating multinomial prediction models for methotrexate treatment outcomes in patients with rheumatoid arthritis: results from an international collaboration

OBJECTIVES

In rheumatology, there is a clinical need to identify patients at high risk (>50%) of not responding to the first-line therapy methotrexate (MTX) due to lack of disease control or discontinuation due to adverse events (AEs). Despite this need, previous prediction models in this context are at high risk of bias and ignore AEs. Our objectives were to (i) develop a multinomial model for outcomes of low disease activity and discontinuing due to AEs 6 months after starting MTX, (ii) update prognosis 3-month following treatment initiation, and (iii) externally validate these models.

STUDY DESIGN AND SETTING

A multinomial model for low disease activity (submodel 1) and discontinuing due to AEs (submodel 2) was developed using data from the UK Rheumatoid Arthritis Medication Study, updated using landmarking analysis, internally validated using bootstrapping, and externally validated in the Norwegian Disease-Modifying Antirheumatic Drug register. Performance was assessed using calibration (calibration-slope and calibration-in-the-large), and discrimination (concordance-statistic and polytomous discriminatory index).

RESULTS

The internally validated model showed good calibration in the development setting with a calibration-slope of 1.01 (0.87, 1.14) (submodel 1) and 0.83 (0.30, 1.34) (submodel 2), and moderate discrimination with a c-statistic of 0.72 (0.69, 0.74) and 0.53 (0.48, 0.59), respectively. Predictive performance decreased after external validation (calibration-slope 0.78 (0.64, 0.93) (submodel 1) and 0.86 (0.34, 1.38) (submodel 2)), which may be due to differences in disease-specific characteristics and outcome prevalence.

CONCLUSION

We addressed previously identified methodological limitations of prediction models for outcomes of MTX therapy. The multinomial approach predicted outcomes of disease activity more accurately than AEs, which should be addressed in future work to aid implementation into clinical practice.

External validation of a multivariable prediction model for identification of pneumonia and other serious bacterial infections in febrile immunocompromised children

OBJECTIVE

To externally validate and update the Feverkids tool clinical prediction model for differentiating bacterial pneumonia and other serious bacterial infections (SBIs) from non-SBI causes of fever in immunocompromised children.

DESIGN

International, multicentre, prospective observational study embedded in PErsonalised Risk assessment in Febrile illness to Optimise Real-life Management across the European Union (PERFORM).

SETTING

Fifteen teaching hospitals in nine European countries.

PARTICIPANTS

Febrile immunocompromised children aged 0-18 years.

METHODS

The Feverkids clinical prediction model predicted the probability of bacterial pneumonia, other SBI or no SBI. Model discrimination, calibration and diagnostic performance at different risk thresholds were assessed. The model was then re-fitted and updated.

RESULTS

Of 558 episodes, 21 had bacterial pneumonia, 104 other SBI and 433 no SBI. Discrimination was 0.83 (95% CI 0.71 to 0.90) for bacterial pneumonia, with moderate calibration and 0.67 (0.61 to 0.72) for other SBIs, with poor calibration. After model re-fitting, discrimination improved to 0.88 (0.79 to 0.96) and 0.71 (0.65 to 0.76) and calibration improved. Predicted risk <1% ruled out bacterial pneumonia with sensitivity 0.95 (0.86 to 1.00) and negative likelihood ratio (LR) 0.09 (0.00 to 0.32). Predicted risk >10% ruled in bacterial pneumonia with specificity 0.91 (0.88 to 0.94) and positive LR 6.51 (3.71 to 10.3). Predicted risk <10% ruled out other SBIs with sensitivity 0.92 (0.87 to 0.97) and negative LR 0.32 (0.13 to 0.57). Predicted risk >30% ruled in other SBIs with specificity 0.89 (0.86 to 0.92) and positive LR 2.86 (1.91 to 4.25).

CONCLUSION

Discrimination and calibration were good for bacterial pneumonia but poorer for other SBIs. The rule-out thresholds have the potential to reduce unnecessary investigations and antibiotics in this high-risk group.

Multiclass risk models for ovarian malignancy: an illustration of prediction uncertainty due to the choice of algorithm

BACKGROUND

Assessing malignancy risk is important to choose appropriate management of ovarian tumors. We compared six algorithms to estimate the probabilities that an ovarian tumor is benign, borderline malignant, stage I primary invasive, stage II-IV primary invasive, or secondary metastatic.

METHODS

This retrospective cohort study used 5909 patients recruited from 1999 to 2012 for model development, and 3199 patients recruited from 2012 to 2015 for model validation. Patients were recruited at oncology referral or general centers and underwent an ultrasound examination and surgery ≤ 120 days later. We developed models using standard multinomial logistic regression (MLR), Ridge MLR, random forest (RF), XGBoost, neural networks (NN), and support vector machines (SVM). We used nine clinical and ultrasound predictors but developed models with or without CA125.

RESULTS

Most tumors were benign (3980 in development and 1688 in validation data), secondary metastatic tumors were least common (246 and 172). The c-statistic (AUROC) to discriminate benign from any type of malignant tumor ranged from 0.89 to 0.92 for models with CA125, from 0.89 to 0.91 for models without. The multiclass c-statistic ranged from 0.41 (SVM) to 0.55 (XGBoost) for models with CA125, and from 0.42 (SVM) to 0.51 (standard MLR) for models without. Multiclass calibration was best for RF and XGBoost. Estimated probabilities for a benign tumor in the same patient often differed by more than 0.2 (20% points) depending on the model. Net Benefit for diagnosing malignancy was similar for algorithms at the commonly used 10% risk threshold, but was slightly higher for RF at higher thresholds. Comparing models, between 3% (XGBoost vs. NN, with CA125) and 30% (NN vs. SVM, without CA125) of patients fell on opposite sides of the 10% threshold.

CONCLUSION

Although several models had similarly good performance, individual probability estimates varied substantially.

Diagnostic Performance of Ultrasonography-Based Risk Models in Differentiating Between Benign and Malignant Ovarian Tumors in a US Cohort

Importance

Ultrasonography-based risk models can help nonexpert clinicians evaluate adnexal lesions and reduce surgical interventions for benign tumors. Yet, these models have limited uptake in the US, and studies comparing their diagnostic accuracy are lacking.

Objective

To evaluate, in a US cohort, the diagnostic performance of 3 ultrasonography-based risk models for differentiating between benign and malignant adnexal lesions: International Ovarian Tumor Analysis (IOTA) Simple Rules with inconclusive cases reclassified as malignant or reevaluated by an expert, IOTA Assessment of Different Neoplasias in the Adnexa (ADNEX), and Ovarian-Adnexal Reporting and Data System (O-RADS).

Design, Setting, and Participants

This retrospective diagnostic study was conducted at a single US academic medical center and included consecutive patients aged 18 to 89 years with adnexal masses that were managed surgically or conservatively between January 2017 and October 2022.

Exposure

Evaluation of adnexal lesions using the Simple Rules, ADNEX, and O-RADS.

Main Outcomes and Measures

The main outcome was diagnostic performance, including area under the receiver operating characteristic (ROC) curve (AUC), sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratios. Surgery or follow-up were reference standards. Secondary analyses evaluated the models' performances stratified by menopause status and race.

Results

The cohort included 511 female patients with a 15.9% malignant tumor prevalence (81 patients). Mean (SD) ages of patients with benign and malignant adnexal lesions were 44.1 (14.4) and 52.5 (15.2) years, respectively, and 200 (39.1%) were postmenopausal. In the ROC analysis, the AUCs for discriminative performance of the ADNEX and O-RADS models were 0.96 (95% CI, 0.93-0.98) and 0.92 (95% CI, 0.90-0.95), respectively. After converting the ADNEX continuous individualized risk into the discrete ordinal categories of O-RADS, the ADNEX performance was reduced to an AUC of 0.93 (95% CI, 0.90-0.96), which was similar to that for O-RADS. The Simple Rules combined with expert reevaluation had 93.8% sensitivity (95% CI, 86.2%-98.0%) and 91.9% specificity (95% CI, 88.9%-94.3%), and the Simple Rules combined with malignant classification had 93.8% sensitivity (95% CI, 86.2%-98.0%) and 88.1% specificity (95% CI, 84.7%-91.0%). At a 10% risk threshold, ADNEX had 91.4% sensitivity (95% CI, 83.0%-96.5%) and 86.3% specificity (95% CI, 82.7%-89.4%) and O-RADS had 98.8% sensitivity (95% CI, 93.3%-100%) and 74.4% specificity (95% CI, 70.0%-78.5%). The specificities of all models were significantly lower in the postmenopausal group. Subgroup analysis revealed high performances independent of race.

Conclusions and Relevance

In this diagnostic study of a US cohort, the Simple Rules, ADNEX, and O-RADS models performed well in differentiating between benign and malignant adnexal lesions; this outcome has been previously reported primarily in European populations. Risk stratification models can lead to more accurate and consistent evaluations of adnexal masses, especially when used by nonexpert clinicians, and may reduce unnecessary surgeries.

The impact of COVID-19 on antibiotic prescribing in primary care in England: Evaluation and risk prediction of appropriateness of type and repeat prescribing

BACKGROUND

This study aimed to predict risks of potentially inappropriate antibiotic type and repeat prescribing and assess changes during COVID-19.

METHODS

With the approval of NHS England, we used OpenSAFELY platform to access the TPP SystmOne electronic health record (EHR) system and selected patients prescribed antibiotics from 2019 to 2021. Multinomial logistic regression models predicted patient's probability of receiving inappropriate antibiotic type or repeat antibiotic course for each common infection.

RESULTS

The population included 9.1 million patients with 29.2 million antibiotic prescriptions. 29.1% of prescriptions were identified as repeat prescribing. Those with same day incident infection coded in the EHR had considerably lower rates of repeat prescribing (18.0%) and 8.6% had potentially inappropriate type. No major changes in the rates of repeat antibiotic prescribing during COVID-19 were found. In the 10 risk prediction models, good levels of calibration and moderate levels of discrimination were found.

CONCLUSIONS

Our study found no evidence of changes in level of inappropriate or repeat antibiotic prescribing after the start of COVID-19. Repeat antibiotic prescribing was frequent and varied according to regional and patient characteristics. There is a need for treatment guidelines to be developed around antibiotic failure and clinicians provided with individualised patient information.

Minimum sample size for developing a multivariable prediction model using multinomial logistic regression

AIMS

Multinomial logistic regression models allow one to predict the risk of a categorical outcome with > 2 categories. When developing such a model, researchers should ensure the number of participants (n ) is appropriate relative to the number of events (E k ) and the number of predictor parameters (p k ) for each category k. We propose three criteria to determine the minimum n required in light of existing criteria developed for binary outcomes.

PROPOSED CRITERIA

The first criterion aims to minimise the model overfitting. The second aims to minimise the difference between the observed and adjusted R 2 Nagelkerke. The third criterion aims to ensure the overall risk is estimated precisely. For criterion (i), we show the sample size must be based on the anticipated Cox-snell R 2 of distinct 'one-to-one' logistic regression models corresponding to the sub-models of the multinomial logistic regression, rather than on the overall Cox-snell R 2 of the multinomial logistic regression.

EVALUATION OF CRITERIA

We tested the performance of the proposed criteria (i) through a simulation study and found that it resulted in the desired level of overfitting. Criterion (ii) and (iii) were natural extensions from previously proposed criteria for binary outcomes and did not require evaluation through simulation.

SUMMARY

We illustrated how to implement the sample size criteria through a worked example considering the development of a multinomial risk prediction model for tumour type when presented with an ovarian mass. Code is provided for the simulation and worked example. We will embed our proposed criteria within the pmsampsize R library and Stata modules.

A network approach to compute hypervolume under receiver operating characteristic manifold for multi-class biomarkers

Computation of hypervolume under ROC manifold (HUM) is necessary to evaluate biomarkers for their capability to discriminate among multiple disease types or diagnostic groups. However the original definition of HUM involves multiple integration and thus a medical investigation for multi-class receiver operating characteristic (ROC) analysis could suffer from huge computational cost when the formula is implemented naively. We introduce a novel graph-based approach to compute HUM efficiently in this article. The computational method avoids the time-consuming multiple summation when sample size or the number of categories is large. We conduct extensive simulation studies to demonstrate the improvement of our method over existing R packages. We apply our method to two real biomedical data sets to illustrate its application.

Development and validation of novel multimorbidity indices for older adults

BACKGROUND

Measuring multimorbidity in claims data is used for risk adjustment and identifying populations at high risk for adverse events. Multimorbidity indices such as Charlson and Elixhauser scores have important limitations. We sought to create a better method of measuring multimorbidity using claims data by incorporating geriatric conditions, markers of disease severity, and disease-disease interactions, and by tailoring measures to different outcomes.

METHODS

Health conditions were assessed using Medicare inpatient and outpatient claims from subjects age 67 and older in the Health and Retirement Study. Separate indices were developed for ADL decline, IADL decline, hospitalization, and death, each over 2 years of follow-up. We validated these indices using data from Medicare claims linked to the National Health and Aging Trends Study.

RESULTS

The development cohort included 5012 subjects with median age 76 years; 58% were female. Claims-based markers of disease severity and disease-disease interactions yielded minimal gains in predictive power and were not included in the final indices. In the validation cohort, after adjusting for age and sex, c-statistics for the new multimorbidity indices were 0.72 for ADL decline, 0.69 for IADL decline, 0.72 for hospitalization, and 0.77 for death. These c-statistics were 0.02-0.03 higher than c-statistics from Charlson and Elixhauser indices for predicting ADL decline, IADL decline, and hospitalization, and <0.01 higher for death (p < 0.05 for each outcome except death), and were similar to those from the CMS-HCC model. On decision curve analysis, the new indices provided minimal benefit compared with legacy approaches. C-statistics for both new and legacy indices varied substantially across derivation and validation cohorts.

CONCLUSIONS

A new series of claims-based multimorbidity measures were modestly better at predicting hospitalization and functional decline than several legacy indices, and no better at predicting death. There may be limited opportunity in claims data to measure multimorbidity better than older methods.

AutoScore-Ordinal: an interpretable machine learning framework for generating scoring models for ordinal outcomes

Background

Risk prediction models are useful tools in clinical decision-making which help with risk stratification and resource allocations and may lead to a better health care for patients. AutoScore is a machine learning–based automatic clinical score generator for binary outcomes. This study aims to expand the AutoScore framework to provide a tool for interpretable risk prediction for ordinal outcomes.

Methods

The AutoScore-Ordinal framework is generated using the same 6 modules of the original AutoScore algorithm including variable ranking, variable transformation, score derivation (from proportional odds models), model selection, score fine-tuning, and model evaluation. To illustrate the AutoScore-Ordinal performance, the method was conducted on electronic health records data from the emergency department at Singapore General Hospital over 2008 to 2017. The model was trained on 70% of the data, validated on 10% and tested on the remaining 20%.

Results

This study included 445,989 inpatient cases, where the distribution of the ordinal outcome was 80.7% alive without 30-day readmission, 12.5% alive with 30-day readmission, and 6.8% died inpatient or by day 30 post discharge. Two point-based risk prediction models were developed using two sets of 8 predictor variables identified by the flexible variable selection procedure. The two models indicated reasonably good performance measured by mean area under the receiver operating characteristic curve (0.758 and 0.793) and generalized c-index (0.737 and 0.760), which were comparable to alternative models.

Conclusion

AutoScore-Ordinal provides an automated and easy-to-use framework for development and validation of risk prediction models for ordinal outcomes, which can systematically identify potential predictors from high-dimensional data.

The polytomous discrimination index for prediction involving multistate processes under intermittent observation

With the increasing importance of predictive modeling in health research comes the need for methods to rigorously assess predictive accuracy. We consider the problem of evaluating the accuracy of predictive models for nominal outcomes when outcome data are coarsened at random. We first consider the problem in the context of a multinomial response modeled by polytomous logistic regression. Attention is then directed to the motivating setting in which class membership corresponds to the state occupied in a multistate disease process at a time horizon of interest. Here, class (state) membership may be unknown at the time horizon since disease processes are under intermittent observation. We propose a novel extension to the polytomous discrimination index to address this and evaluate the predictive accuracy of an intensity-based model in the context of a study involving patients with arthritis from a registry at the University of Toronto Centre for Prognosis Studies in Rheumatic Diseases.

Risk prediction models for discrete ordinal outcomes: Calibration and the impact of the proportional odds assumption

Calibration is a vital aspect of the performance of risk prediction models, but research in the context of ordinal outcomes is scarce. This study compared calibration measures for risk models predicting a discrete ordinal outcome, and investigated the impact of the proportional odds assumption on calibration and overfitting. We studied the multinomial, cumulative, adjacent category, continuation ratio, and stereotype logit/logistic models. To assess calibration, we investigated calibration intercepts and slopes, calibration plots, and the estimated calibration index. Using large sample simulations, we studied the performance of models for risk estimation under various conditions, assuming that the true model has either a multinomial logistic form or a cumulative logit proportional odds form. Small sample simulations were used to compare the tendency for overfitting between models. As a case study, we developed models to diagnose the degree of coronary artery disease (five categories) in symptomatic patients. When the true model was multinomial logistic, proportional odds models often yielded poor risk estimates, with calibration slopes deviating considerably from unity even on large model development datasets. The stereotype logistic model improved the calibration slope, but still provided biased risk estimates for individual patients. When the true model had a cumulative logit proportional odds form, multinomial logistic regression provided biased risk estimates, although these biases were modest. Nonproportional odds models require more parameters to be estimated from the data, and hence suffered more from overfitting. Despite larger sample size requirements, we generally recommend multinomial logistic regression for risk prediction modeling of discrete ordinal outcomes.

Assessing the performance of clinical diagnostic models for dehydration among patients with cholera and undernutrition in Bangladesh

OBJECTIVE

Accurately assessing dehydration severity is a critical step in reducing mortality from diarrhoea, but is complicated by cholera and undernutrition. This study seeks to assess the accuracy of two clinical diagnostic models for dehydration among patients over five years with cholera and undernutrition and compare their respective performance to the World Health Organization (WHO) algorithm.

METHODS

In this secondary analysis of data collected from the NIRUDAK study, accuracy of the full and simplified NIRUDAK models for predicting severe and any dehydration was measured using the area under the Receiver Operator Characteristic curve (AUC) among patients over five with/without cholera and with/without wasting. Bootstrap with 1000 iterations was used to compare the m-index for each NIRUDAK model to that of the WHO algorithm.

RESULTS

A total of 2,139 and 2,108 patients were included in the nutrition and cholera subgroups respectively with an overall median age of 35 years (IQR = 42) and 49.6% female. All subgroups had acceptable discrimination in diagnosing severe or any dehydration (AUC > 0.60); though the full NIRUDAK model performed best among patients without cholera, with an AUC of 0.82 (95%CI:0.79, 0.85) and among patients without wasting, with an AUC of 0.79 (95%CI:0.76, 0.81). Compared with the WHO's algorithm, both the full and simplified NIRUDAK models performed significantly better in terms of their m-index (p < 0.001) for all comparisons, except for the simplified NIRUDAK model in the wasting group.

CONCLUSIONS

Both the full and simplified NIRUDAK models performed less well in patients over five years with cholera and/or wasting; however, both performed better than the WHO algorithm.

Evaluation of competing risks prediction models using polytomous discrimination index

For competing risks data, it is often important to predict a patient's outcome status at a clinically meaningful time point after incorporating the informative censoring due to competing risks. This can be done by adopting a regression model that relates the cumulative incidence probabilities to a set of covariates. To assess the performance of the resulting prediction tool, we propose an estimator of the polytomous discrimination index applicable to competing risks data, which can quantify a prognostic model's ability to discriminate among subjects from different outcome groups. The proposed estimator allows the prediction model to be subject to model misspecification and enjoys desirable asymptotic properties. We also develop an efficient computation algorithm that features a computational complexity of O(n log n). A perturbation resampling scheme is developed to achieve consistent variance estimation. Numerical results suggest that the estimator performs well under realistic sample sizes. We apply the proposed methods to a study of monoclonal gammopathy of undetermined significance.

Development of Classification Criteria for the Uveitides

PURPOSE

To develop classification criteria for 25 of the most common uveitides.

DESIGN

Machine learning using 5,766 cases of 25 uveitides.

METHODS

Cases were collected in an informatics-designed preliminary database. Using formal consensus techniques, a final database was constructed of 4,046 cases achieving supermajority agreement on the diagnosis. Cases were analyzed within uveitic class and were split into a training set and a validation set. Machine learning used multinomial logistic regression with lasso regularization on the training set to determine a parsimonious set of criteria for each disease and to minimize misclassification rates. The resulting criteria were evaluated in the validation set. Accuracy of the rules developed to express the machine learning criteria was evaluated by a masked observer in a 10% random sample of cases.

RESULTS

Overall accuracy estimates by uveitic class in the validation set were as follows: anterior uveitides 96.7% (95% confidence interval [CI] 92.4, 98.6); intermediate uveitides 99.3% (95% CI 96.1, 99.9); posterior uveitides 98.0% (95% CI 94.3, 99.3); panuveitides 94.0% (95% CI 89.0, 96.8); and infectious posterior uveitides / panuveitides 93.3% (95% CI 89.1, 96.3). Accuracies of the masked evaluation of the "rules" were anterior uveitides 96.5% (95% CI 91.4, 98.6) intermediate uveitides 98.4% (91.5, 99.7), posterior uveitides 99.2% (95% CI 95.4, 99.9), panuveitides 98.9% (95% CI 94.3, 99.8), and infectious posterior uveitides / panuveitides 98.8% (95% CI 93.4, 99.9).

CONCLUSIONS

The classification criteria for these 25 uveitides had high overall accuracy (ie, low misclassification rates) and seemed to perform well enough for use in clinical and translational research.

Radiomics analysis of contrast-enhanced CT for classification of hepatic focal lesions in colorectal cancer patients: its limitations compared to radiologists

OBJECTIVE

To evaluate diagnostic performance of a radiomics model for classifying hepatic cyst, hemangioma, and metastasis in patients with colorectal cancer (CRC) from portal-phase abdominopelvic CT images.

METHODS

This retrospective study included 502 CRC patients who underwent contrast-enhanced CT and contrast-enhanced liver MRI between January 2005 and December 2010. Portal-phase CT images of training (n = 386) and validation (n = 116) cohorts were used to develop a radiomics model for differentiating three classes of liver lesions. Among multiple handcrafted features, the feature selection was performed using ReliefF method, and random forest classifiers were used to train the selected features. Diagnostic performance of the developed model was compared with that of four radiologists. A subgroup analysis was conducted based on lesion size.

RESULTS

The radiomics model demonstrated significantly lower overall and hemangioma- and metastasis-specific polytomous discrimination index (PDI) (overall, 0.8037; hemangioma-specific, 0.6653; metastasis-specific, 0.8027) than the radiologists (overall, 0.9622-0.9680; hemangioma-specific, 0.9452-0.9630; metastasis-specific, 0.9511-0.9869). For subgroup analysis, the PDI of the radiomics model was different according to the lesion size (< 10 mm, 0.6486; ≥ 10 mm, 0.8264) while that of the radiologists was relatively maintained. For classifying metastasis from benign lesions, the radiomics model showed excellent diagnostic performance, with an accuracy of 84.36% and an AUC of 0.9426.

CONCLUSION

Albeit inferior to the radiologists, the radiomics model achieved substantial diagnostic performance when differentiating hepatic lesions from portal-phase CT images of CRC patients. This model was limited particularly to classifying hemangiomas and subcentimeter lesions.

KEY POINTS

• Albeit inferior to the radiologists, the radiomics model could differentiate cyst, hemangioma, and metastasis with substantial diagnostic performance using portal-phase CT images of colorectal cancer patients. • The radiomics model demonstrated limitations especially in classifying hemangiomas and subcentimeter liver lesions.

Degrees of necessity and of sufficiency: Further results and extensions, with an application to covid-19 mortality in Austria

The purpose of this paper is to extend to ordinal and nominal outcomes the measures of degree of necessity and of sufficiency defined by the authors for dichotomous and survival outcomes in a previous paper. A cause, represented by certain values of prognostic factors, is considered necessary for an event if, without the cause, the event cannot develop. It is considered sufficient for an event if the event is unavoidable in the presence of the cause. The degrees of necessity and sufficiency, ranging from zero to one, are simple, intuitive functions of unconditional and conditional probabilities of an event such as disease or death. These probabilities often will be derived from logistic regression models; the measures, however, do not require any particular model. In addition, we study in detail the relationship between the proposed measures and the related explained variation summary for dichotomous outcomes, which are the common root for the developments for ordinal, nominal, and survival outcomes. We introduce and analyze the Austrian covid-19 data, with the aim of quantifying effects of age and other potentially prognostic factors on covid-19 mortality. This is achieved by standard regression methods but also in terms of the newly proposed measures. It is shown how they complement the toolbox of prognostic factor studies, in particular when comparing the importance of prognostic factors of different types. While the full model's degree of necessity is extremely high (0.933), its low degree of sufficiency (0.179) is responsible for the low proportion of explained variation (0.193).

Computing the polytomous discrimination index

Polytomous regression models generalize logistic models for the case of a categorical outcome variable with more than two distinct categories. These models are currently used in clinical research, and it is essential to measure their abilities to distinguish between the categories of the outcome. In 2012, van Calster et al proposed the polytomous discrimination index (PDI) as an extension of the binary discrimination c-statistic to unordered polytomous regression. The PDI is a summary of the simultaneous discrimination between all outcome categories. Previous implementations of the PDI are not capable of running on "big data." This article shows that the PDI formula can be manipulated to depend only on the distributions of the predicted probabilities evaluated for each outcome category and within each observed level of the outcome, which substantially improves the computation time. We present a SAS macro and R function that can rapidly evaluate the PDI and its components. The routines are evaluated on several simulated datasets after varying the number of categories of the outcome and size of the data and two real-world large administrative health datasets. We compare PDI with two other discrimination indices: M-index and hypervolume under the manifold (HUM) on simulated examples. We describe situations where the PDI and HUM, indices based on multiple comparisons, are superior to the M-index, an index based on pairwise comparisons, to detect predictions that are no different than random selection or erroneous due to incorrect ranking.

Derivation of the first clinical diagnostic models for dehydration severity in patients over five years with acute diarrhea

Diarrheal diseases lead to an estimated 1.3 million deaths each year, with the majority of those deaths occurring in patients over five years of age. As the severity of diarrheal disease can vary widely, accurately assessing dehydration status remains the most critical step in acute diarrhea management. The objective of this study is to empirically derive clinical diagnostic models for assessing dehydration severity in patients over five years with acute diarrhea in low resource settings. We enrolled a random sample of patients over five years with acute diarrhea presenting to the icddr,b Dhaka Hospital. Two blinded nurses independently assessed patients for symptoms/signs of dehydration on arrival. Afterward, consecutive weights were obtained to determine the percent weight change with rehydration, our criterion standard for dehydration severity. Full and simplified ordinal logistic regression models were derived to predict the outcome of none (<3%), some (3-9%), or severe (>9%) dehydration. The reliability and accuracy of each model were assessed. Bootstrapping was used to correct for over-optimism and compare each model's performance to the current World Health Organization (WHO) algorithm. 2,172 patients were enrolled, of which 2,139 (98.5%) had complete data for analysis. The Inter-Class Correlation Coefficient (reliability) was 0.90 (95% CI = 0.87, 0.91) for the full model and 0.82 (95% CI = 0.77, 0.86) for the simplified model. The area under the Receiver-Operator Characteristic curve (accuracy) for severe dehydration was 0.79 (95% CI: 0.76-0.82) for the full model and 0.73 (95% CI: 0.70, 0.76) for the simplified model. The accuracy for both the full and simplified models were significantly better than the WHO algorithm (p<0.001). This is the first study to empirically derive clinical diagnostic models for dehydration severity in patients over five years. Once prospectively validated, the models may improve management of patients with acute diarrhea in low resource settings.

Preoperative Evaluation of the ADNEX Model for the Prediction of the Ovarian Cancer Risk of Adnexal Masses at Siriraj Hospital

INTRODUCTION

Distinguishing benign adnexal masses from malignant tumors plays an important role in preoperative planning and improving patients' survival rates. The International Ovarian Tumor Analysis (IOTA) group developed a model termed the Assessment of Different NEoplasias in the adneXa (ADNEX).

OBJECTIVE

Our objective was to evaluate the performance of the ADNEX model in distinguishing between benign and malignant tumors at a cutoff value of 10%.

METHODS

This was a prospective diagnostic study. 357 patients with an adnexal mass who were scheduled for surgery at Siriraj Hospital were included from May 1, 2018, to May 30, 2019. All patients were undergoing ultrasonography, and serum CA125 was measured. Data were calculated by the ADNEX model via an IOTA ADNEX calculator.

RESULTS

Of the 357 patients, 296 had benign tumors and 61 had malignant tumors. The area under the receiver operating characteristic curve for using the ADNEX model was 0.975 (95% confidence interval, 0.953-0.988). At a 10% cutoff, the sensitivity was 98.4% and specificity was 87.2%. The best cutoff value was at 16.6% in our population.

CONCLUSIONS

The performance of the ADNEX model in differentiating benign and malignant tumors was excellent.

Development and validation of a screening model for diabetes mellitus in patients with periodontitis in dental settings

OBJECTIVES

To identify predictors in patient profiles and to develop, internally validate, and calibrate a screening model for diabetes mellitus (DM) in patients with periodontitis in dental settings MATERIALS AND METHODS: The study included 204 adult patients with periodontitis. Patients' socio-demographic characteristics, general health status, and periodontal status were recorded as potential predictors. The diabetic status was considered the outcome, classified into no DM, prediabetes (pre-DM), or DM. Multinomial logistic regression analysis was used to develop the model. The performance and clinical values of the model were determined.

RESULTS

Seventeen percent and 47% of patients were diagnosed with DM and pre-DM, respectively. Patients' age, BMI, European background, cholesterol levels, previous periodontal treatment, percentage of the number of teeth with mobility, and with gingival recession were significantly associated with the diabetic status of the patients. The model showed a reasonable calibration and moderate to good discrimination with area under the curve (AUC) values of 0.67 to 0.80. The added predictive values for ruling in the risk of DM and pre-DM were 0.42 and 0.11, respectively, and those for ruling it out were 0.05 and 0.17, respectively.

CONCLUSIONS

Predictors in patient profiles for screening of DM and pre-DM in patients with periodontitis were identified. The calibration, discrimination, and clinical values of the model were acceptable.

CLINICAL RELEVANCE

The model may well assist clinicians in screening of diabetic status of patients with periodontitis. The model can be used as a reliable screening tool for DM and pre-DM in patients with periodontitis in dental settings.

Clinical prediction models to predict the risk of multiple binary outcomes: a comparison of approaches

Clinical prediction models (CPMs) can predict clinically relevant outcomes or events. Typically, prognostic CPMs are derived to predict the risk of a single future outcome. However, there are many medical applications where two or more outcomes are of interest, meaning this should be more widely reflected in CPMs so they can accurately estimate the joint risk of multiple outcomes simultaneously. A potentially naïve approach to multi-outcome risk prediction is to derive a CPM for each outcome separately, then multiply the predicted risks. This approach is only valid if the outcomes are conditionally independent given the covariates, and it fails to exploit the potential relationships between the outcomes. This paper outlines several approaches that could be used to develop CPMs for multiple binary outcomes. We consider four methods, ranging in complexity and conditional independence assumptions: namely, probabilistic classifier chain, multinomial logistic regression, multivariate logistic regression, and a Bayesian probit model. These are compared with methods that rely on conditional independence: separate univariate CPMs and stacked regression. Employing a simulation study and real-world example, we illustrate that CPMs for joint risk prediction of multiple outcomes should only be derived using methods that model the residual correlation between outcomes. In such a situation, our results suggest that probabilistic classification chains, multinomial logistic regression or the Bayesian probit model are all appropriate choices. We call into question the development of CPMs for each outcome in isolation when multiple correlated or structurally related outcomes are of interest and recommend more multivariate approaches to risk prediction.

External validation of models to predict the outcome of pregnancies of unknown location: a multicentre cohort study

Objective

To validate externally five approaches to predict ectopic pregnancy (EP) in pregnancies of unknown location (PUL): the M6P and M6NP risk models, the two‐step triage strategy (2ST, which incorporates M6P), the M4 risk model, and beta human chorionic gonadotropin ratio cut‐offs (BhCG‐RC).

Design

Secondary analysis of a prospective cohort study.

Setting

Eight UK early pregnancy assessment units.

Population

Women presenting with a PUL and BhCG >25 IU/l.

Methods

Women were managed using the 2ST protocol: PUL were classified as low risk of EP if presenting progesterone ≤2 nmol/l; the remaining cases returned 2 days later for triage based on M6P. EP risk ≥5% was used to classify PUL as high risk. Missing values were imputed, and predictions for the five approaches were calculated post hoc. We meta‐analysed centre‐specific results.

Main outcome measures

Discrimination, calibration and clinical utility (decision curve analysis) for predicting EP.

Results

Of 2899 eligible women, the primary analysis excluded 297 (10%) women who were lost to follow up. The area under the ROC curve for EP was 0.89 (95% CI 0.86–0.91) for M6P, 0.88 (0.86–0.90) for 2ST, 0.86 (0.83–0.88) for M6NP and 0.82 (0.78–0.85) for M4. Sensitivities for EP were 96% (M6P), 94% (2ST), 92% (N6NP), 80% (M4) and 58% (BhCG‐RC); false‐positive rates were 35%, 33%, 39%, 24% and 13%. M6P and 2ST had the best clinical utility and good overall calibration, with modest variability between centres.

Conclusions

2ST and M6P performed best for prediction and triage in PUL.

Tweetable abstract

The M6 model, as part of a two‐step triage strategy, is the best approach to characterise and triage PULs.

The M6 model, as part of a two‐step triage strategy, is the best approach to characterise and triage PULs.

Use of the Activity Measure for Post-Acute Care "6 Clicks" Basic Mobility Inpatient Short Form and National Institutes of Health Stroke Scale to Predict Hospital Discharge Disposition After Stroke

OBJECTIVE

Therapists in the hospital are charged with making timely discharge recommendations to improve access to rehabilitation after stroke. The objective of this study was to identify the predictive ability of the Activity Measure for Post-Acute Care "6 Clicks" Basic Mobility Inpatient Short Form (6 Clicks mobility) score and the National Institutes of Health Stroke Scale (NIHSS) score for actual hospital discharge disposition after stroke.

METHODS

In this retrospective cohort study, data were collected from an academic hospital in the United States for 1543 patients with acute stroke and a 6 Clicks mobility score. Discharge to home, a skilled nursing facility (SNF), or an inpatient rehabilitation facility (IRF) was the primary outcome. Associations among these outcomes and 6 Clicks mobility and NIHSS scores, alone or together, were tested using multinomial logistic regression, and the predictive ability of these scores was calculated using concordance statistics.

RESULTS

A higher 6 Clicks mobility score alone was associated with a decreased odds of actual discharge to an IRF or an SNF. The 6 Clicks mobility score alone was a strong predictor of discharge to home versus an IRF or an SNF. However, predicting discharge to an IRF versus an SNF was stronger when the 6 Clicks mobility score was considered in combination with the NIHSS score, age, sex, and race.

CONCLUSION

The 6 Clicks mobility score alone can guide discharge decision making after stroke, particularly for discharge to home versus an SNF or an IRF. Determining discharge to an SNF versus an IRF could be improved by also considering the NIHSS score, age, sex, and race. Future studies should seek to identify which additional characteristics improve predictability for these separate discharge destinations.

IMPACT

The use of outcome measures can improve therapist confidence in making discharge recommendations for people with stroke, can enhance hospital throughput, and can expedite access to rehabilitation, ultimately affecting functional outcomes.

Using pharmacy dispensing data to predict falls in older individuals

Aims

Associations between individual medication use and falling in older individuals are well‐documented. However, a comprehensive risk score that takes into account overall medication use and that can be used in daily pharmacy practice is lacking. We, therefore, aimed to determine whether pharmacy dispensing records can be used to predict falls.

Methods

A retrospective cohort study was conducted using pharmacy dispensing data and self‐reported falls among 3454 Dutch individuals aged ≥65 years. Two different methods were used to classify medication exposure for each person: the drug burden index (DBI) for cumulative anticholinergic and sedative medication exposure as well as exposure to fall risk‐increasing drugs (FRIDs). Multinomial regression analyses, adjusted for age and sex, were conducted to investigate the association between medication exposure and falling classified as nonfalling, single falling and recurrent falling. The predictive performances of the DBI and FRIDs exposure were estimated by the polytomous discrimination index (PDI).

Results

There were 521 single fallers (15%) and 485 recurrent fallers (14%). We found significant associations between a DBI ≥1 and single falling (adjusted odds ratio: 1.30 [95% confidence interval {CI}: 1.02–1.66]) and recurrent falling (adjusted odds ratio: 1.60 [95%CI: 1.25–2.04]). The PDI of the DBI model was 0.41 (95%CI: 0.39–0.42) and the PDI of the FRIDs model was 0.45 (95%CI: 0.43–0.47), indicating poor discrimination between fallers and nonfallers.

Conclusion

The study shows significant associations between medication use and falling. However, the medication‐based models were insufficient and other factors should be included to develop a risk score for pharmacy practice.

Criteria for evaluating risk prediction of multiple outcomes

Risk prediction models have been developed in many contexts to classify individuals according to a single outcome, such as risk of a disease. Emerging “-omic” biomarkers provide panels of features that can simultaneously predict multiple outcomes from a single biological sample, creating issues of multiplicity reminiscent of exploratory hypothesis testing. Here I propose definitions of some basic criteria for evaluating prediction models of multiple outcomes. I define calibration in the multivariate setting and then distinguish between outcome-wise and individual-wise prediction, and within the latter between joint and panel-wise prediction. I give examples such as screening and early detection in which different senses of prediction may be more appropriate. In each case I propose definitions of sensitivity, specificity, concordance, positive and negative predictive value and relative utility. I link the definitions through a multivariate probit model, showing that the accuracy of a multivariate prediction model can be summarised by its covariance with a liability vector. I illustrate the concepts on a biomarker panel for early detection of eight cancers, and on polygenic risk scores for six common diseases.

Automatic Triage of 12-Lead ECGs Using Deep Convolutional Neural Networks

BACKGROUND

The correct interpretation of the ECG is pivotal for the accurate diagnosis of many cardiac abnormalities, and conventional computerized interpretation has not been able to reach physician‐level accuracy in detecting (acute) cardiac abnormalities. This study aims to develop and validate a deep neural network for comprehensive automated ECG triage in daily practice.

METHODS AND RESULTS

We developed a 37‐layer convolutional residual deep neural network on a data set of free‐text physician‐annotated 12‐lead ECGs. The deep neural network was trained on a data set with 336.835 recordings from 142.040 patients and validated on an independent validation data set (n=984), annotated by a panel of 5 cardiologists electrophysiologists. The 12‐lead ECGs were acquired in all noncardiology departments of the University Medical Center Utrecht. The algorithm learned to classify these ECGs into the following 4 triage categories: normal, abnormal not acute, subacute, and acute. Discriminative performance is presented with overall and category‐specific concordance statistics, polytomous discrimination indexes, sensitivities, specificities, and positive and negative predictive values. The patients in the validation data set had a mean age of 60.4 years and 54.3% were men. The deep neural network showed excellent overall discrimination with an overall concordance statistic of 0.93 (95% CI, 0.92–0.95) and a polytomous discriminatory index of 0.83 (95% CI, 0.79–0.87).

CONCLUSIONS

This study demonstrates that an end‐to‐end deep neural network can be accurately trained on unstructured free‐text physician annotations and used to consistently triage 12‐lead ECGs. When further fine‐tuned with other clinical outcomes and externally validated in clinical practice, the demonstrated deep learning–based ECG interpretation can potentially improve time to treatment and decrease healthcare burden.

The ADNEX model to triage adnexal masses: An external validation study and comparison with the IOTA two-step strategy and subjective assessment by an experienced ultrasound operator

OBJECTIVES

The ADNEX (Assessment of Different NEoplasias in the adneXa) model was developed using parameters collected by experienced (level III) ultrasound examiners. Our primary aim was to externally validate the ADNEX model. Then, the discriminatory performance of ADNEX was compared with the two-step strategy and subjective assessment by an experienced ultrasound operator.

METHODS

Between February 2013 and January 2017, all patients who were scheduled for surgery for an adnexal mass at the Sant'Anna Hospital in Turin were enrolled in this study. Preoperative transvaginal sonography was performed, and the two-step strategy was applied for triage of the adnexal mass. Two ultrasound examiners, IOTA certified, applied the ADNEX model to all the collected masses based on the ultrasound reports. Finally, an experienced operator assigned the subjective assessment based on recorded ultrasound images. The discrimination and calibration performance of ADNEX were evaluated. The AUC was calculated for the basic discrimination between benign and malignant tumours. In addition, AUCs were computed for each pair of tumour types using the conditional risk method.

RESULTS

A total of 577 patients were included in the analysis: the overall prevalence of malignancy was 25 %. With ADNEX, the AUC to differentiate between benign and malignant masses was 0.9111 (95 % CI 0. 8788-0.9389). At risk cut-offs of 1%, 10 % and 30 %, sensitivities were 100 %, 89.6 % and 79.2 %, respectively, and specificities were 2.8 %, 76.2 % and 89.6 %, respectively. Discrimination between benign and stage II-IV tumours was good (AUC 0.935). The model had the most difficulties discriminating between borderline and stage I tumours (AUC 0.666), and between stages II-IV invasive and secondary metastatic tumours (AUC 0.736). The polytomous discrimination index (PDI) was 0.61 for ADNEX, whereas PDI for random performance would be 0.25. ADNEX proved to be equally or more accurate than the subjective assessment or the two-step strategy in the discrimination between benign and malignant adnexal masses.

CONCLUSIONS

the ADNEX model could probably be successfully applied when an expert examiner is not available and, therefore both a subjective assessment and the two-step strategy cannot be performed.

Developmental risk among Aboriginal children living in urban areas in Australia: the Study of Environment on Aboriginal Resilience and Child Health (SEARCH)

Background

Most Australian Aboriginal children are on track with their development, however, the prevalence of children at risk of or with a developmental or behavioural problem is higher than in other children. Aboriginal child development data mostly comes from remote communities, whereas most Aboriginal children live in urban settings. We quantified the proportion of participating children at moderate and high developmental risk as identified by caregivers’ concerns, and determined the factors associated with developmental risk among urban Aboriginal communities.

Methods

Study methods were co-designed and implemented with four participating urban Aboriginal Community Controlled Health Services in New South Wales, Australia, between 2008 and 2012. Caregiver-reported data on children < 8 years old enrolled in a longitudinal cohort study (Study of Environment on Aboriginal Resilience and Child Health: SEARCH) were collected by interview. The Parents’ Evaluation of Developmental Status (PEDS) was used to assess developmental risk through report of caregiver concerns. Odds ratios (OR) were calculated using multinomial logistic regression to investigate risk factors and develop a risk prediction model.

Results

Of 725 children in SEARCH with PEDS data (69% of eligible), 405 (56%) were male, and 336 (46%) were aged between 4.5 and 8 years. Using PEDS, 32% were at high, 28% moderate, and 40% low/no developmental risk. Compared with low/no risk, factors associated with high developmental risk in a mutually-adjusted model, with additional adjustment for study site, were male sex (OR 2.42, 95% confidence intervals 1.62–3.61), being older (4.5 to < 8 years versus < 3 years old, 3.80, 2.21–6.54), prior history of ear infection (1.95, 1.21–3.15), having lived in 4 or more houses versus one house (4.13, 2.04–8.35), foster care versus living with a parent (5.45, 2.32–12.78), and having a caregiver with psychological distress (2.40, 1.37–4.20).

Conclusion

In SEARCH, 40% of urban Aboriginal children younger than 8 years were at no or low developmental risk. Several factors associated with higher developmental risk were modifiable. Aboriginal community-driven programs to improve detection of developmental problems and facilitate early intervention are needed.

Statistical methods for building better biomarkers of chronic kidney disease

The last two decades have witnessed an explosion in research focused on the development and assessment of novel biomarkers for improved prognosis of diseases. As a result, best practice standards guiding biomarker research have undergone extensive development. Currently, there is great interest in the promise of biomarkers to enhance research efforts and clinical practice in the setting of chronic kidney disease, acute kidney injury, and glomerular disease. However, some have questioned whether biomarkers currently add value to the clinical practice of nephrology. The current state of the art pertaining to statistical analyses regarding the use of such measures is critical. In December 2014, the National Institute of Diabetes and Digestive and Kidney Diseases convened a meeting, "Toward Building Better Biomarker Statistical Methodology," with the goals of summarizing the current best practice recommendations and articulating new directions for methodological research. This report summarizes its conclusions and describes areas that need attention. Suggestions are made regarding metrics that should be commonly reported. We outline the methodological issues related to traditional metrics and considerations in prognostic modeling, including discrimination and case mix, calibration, validation, and cost-benefit analysis. We highlight the approach to improved risk communication and the value of graphical displays. Finally, we address some "new frontiers" in prognostic biomarker research, including the competing risk framework, the use of longitudinal biomarkers, and analyses in distributed research networks.

Sample size considerations and predictive performance of multinomial logistic prediction models

Multinomial Logistic Regression (MLR) has been advocated for developing clinical prediction models that distinguish between three or more unordered outcomes. We present a full‐factorial simulation study to examine the predictive performance of MLR models in relation to the relative size of outcome categories, number of predictors and the number of events per variable. It is shown that MLR estimated by Maximum Likelihood yields overfitted prediction models in small to medium sized data. In most cases, the calibration and overall predictive performance of the multinomial prediction model is improved by using penalized MLR. Our simulation study also highlights the importance of events per variable in the multinomial context as well as the total sample size. As expected, our study demonstrates the need for optimism correction of the predictive performance measures when developing the multinomial logistic prediction model. We recommend the use of penalized MLR when prediction models are developed in small data sets or in medium sized data sets with a small total sample size (ie, when the sizes of the outcome categories are balanced). Finally, we present a case study in which we illustrate the development and validation of penalized and unpenalized multinomial prediction models for predicting malignancy of ovarian cancer.

AR(1) latent class models for longitudinal count data

In a variety of applications involving longitudinal or repeated-measurements data, it is desired to uncover natural groupings or clusters that exist among study subjects. Motivated by the need to recover clusters of longitudinal trajectories of conduct problems in the field of developmental psychopathology, we propose a method to address this goal when the response data in question are counts. We assume the subject-specific observations are generated from a first-order autoregressive process that is appropriate for count data. A key advantage of our approach is that the class-specific likelihood function arising from each subject's data can be expressed in closed form, circumventing common computational issues associated with random effects models. To further improve computational efficiency, we propose an approximate EM procedure for estimating the model parameters where, within each EM iteration, the maximization step is approximated by solving an appropriately chosen set of estimating equations. We explore the effectiveness of our procedures through simulations based on a four-class model, placing a special emphasis on recovery of the latent trajectories. Finally, we analyze data and recover trajectories of conduct problems in an important nationally representative sample. The methods discussed here are implemented in the R package inarmix, which is available from the Comprehensive R Archive Network (http://cran.r-project.org).

Validation of the Feverkidstool and procalcitonin for detecting serious bacterial infections in febrile children

BackgroundTo validate the Feverkidstool, a prediction model consisting of clinical signs and symptoms and C-reactive protein (CRP) to identify serious bacterial infections (SBIs) in febrile children, and to determine the incremental diagnostic value of procalcitonin.MethodsThis prospective observational study that was carried out at two Dutch emergency departments included children with fever, aged 1 month to 16 years. The prediction models were developed with polytomous logistic regression differentiating "pneumonia" and "other SBIs" from "non-SBIs" using standardized, routinely collected data on clinical signs and symptoms, CRP, and procalcitonin.ResultsA total of 1,085 children were included with a median age of 1.6 years (interquartile range 0.8-3.4); 73 children (7%) had pneumonia and 98 children (9%) had other SBIs. The Feverkidstool showed good discriminative ability in this new population. After adding procalcitonin to the Feverkidstool, c-statistic for "pneumonia" increased from 0.85 (95% confidence interval (CI) 0.76-0.94) to 0.86 (0.77-0.94) and for "other SBI" from 0.81 (0.73-0.90) to 0.83 (0.75- 0.91). A model with clinical features and procalcitonin performed similar to the Feverkidstool.ConclusionThis study confirms the external validity of the Feverkidstool, with CRP and procalcitonin being equally valuable for predicting SBI in our population of febrile children. Our findings do not support routine dual use of CRP and procalcitonin.

Interobserver Agreement Among Uveitis Experts on Uveitic Diagnoses: The Standardization of Uveitis Nomenclature Experience

PURPOSE

To evaluate the interobserver agreement among uveitis experts on the diagnosis of the specific uveitic disease.

DESIGN

Interobserver agreement analysis.

METHODS

Five committees, each comprised of 9 individuals and working in parallel, reviewed cases from a preliminary database of 25 uveitic diseases, collected by disease, and voted independently online whether the case was the disease in question or not. The agreement statistic, κ, was calculated for the 36 pairwise comparisons for each disease, and a mean κ was calculated for each disease. After the independent online voting, committee consensus conference calls, using nominal group techniques, reviewed all cases not achieving supermajority agreement (>75%) on the diagnosis in the online voting to attempt to arrive at a supermajority agreement.

RESULTS

A total of 5766 cases for the 25 diseases were evaluated. The overall mean κ for the entire project was 0.39, with disease-specific variation ranging from 0.23 to 0.79. After the formalized consensus conference calls to address cases that did not achieve supermajority agreement in the online voting, supermajority agreement overall was reached on approximately 99% of cases, with disease-specific variation ranging from 96% to 100%.

CONCLUSIONS

Agreement among uveitis experts on diagnosis is moderate at best but can be improved by discussion among them. These data suggest the need for validated and widely used classification criteria in the field of uveitis.

Using ordinal outcomes to construct and select biomarker combinations for single-level prediction

BACKGROUND

Biomarker studies may involve an ordinal outcome, such as no, mild, or severe disease. There is often interest in predicting one particular level of the outcome due to its clinical significance.

METHODS

A simple approach to constructing biomarker combinations in this context involves dichotomizing the outcome and using a binary logistic regression model. We assessed whether more sophisticated methods offer advantages over this simple approach. It is often necessary to select among several candidate biomarker combinations. One strategy involves selecting a combination based on its ability to predict the outcome level of interest. We propose an algorithm that leverages the ordinal outcome to inform combination selection. We apply this algorithm to data from a study of acute kidney injury after cardiac surgery, where kidney injury may be absent, mild, or severe.

RESULTS

Using more sophisticated modeling approaches to construct combinations provided gains over the simple binary logistic regression approach in specific settings. In the examples considered, the proposed algorithm for combination selection tended to reduce the impact of bias due to selection and to provide combinations with improved performance.

CONCLUSIONS

Methods that utilize the ordinal nature of the outcome in the construction and/or selection of biomarker combinations have the potential to yield better combinations.

Nonparametric estimation and inference for polytomous discrimination index

Polytomous discrimination index is a novel and important diagnostic accuracy measure for multi-category classification. After reconstructing its probabilistic definition, we propose a nonparametric approach to the estimation of polytomous discrimination index based on an empirical sample of biomarker values. In this paper, we provide the finite-sample and asymptotic properties of the proposed estimators and such analytic results may facilitate the statistical inference. Simulation studies are performed to examine the performance of the nonparametric estimators. Two real data examples are analysed to illustrate our methodology.

Validation and updating of risk models based on multinomial logistic regression

BACKGROUND

Risk models often perform poorly at external validation in terms of discrimination or calibration. Updating methods are needed to improve performance of multinomial logistic regression models for risk prediction.

METHODS

We consider simple and more refined updating approaches to extend previously proposed methods for dichotomous outcomes. These include model recalibration (adjustment of intercept and/or slope), revision (re-estimation of individual model coefficients), and extension (revision with additional markers). We suggest a closed testing procedure to assist in deciding on the updating complexity. These methods are demonstrated on a case study of women with pregnancies of unknown location (PUL). A previously developed risk model predicts the probability that a PUL is a failed, intra-uterine, or ectopic pregnancy. We validated and updated this model on more recent patients from the development setting (temporal updating; n = 1422) and on patients from a different hospital (geographical updating; n = 873). Internal validation of updated models was performed through bootstrap resampling.

RESULTS

Contrary to dichotomous models, we noted that recalibration can also affect discrimination for multinomial risk models. If the number of outcome categories is higher than the number of variables, logistic recalibration is obsolete because straightforward model refitting does not require the estimation of more parameters. Although recalibration strongly improved performance in the case study, the closed testing procedure selected model revision. Further, revision of functional form of continuous predictors had a positive effect on discrimination, whereas penalized estimation of changes in model coefficients was beneficial for calibration.

CONCLUSIONS

Methods for updating of multinomial risk models are now available to improve predictions in new settings. A closed testing procedure is helpful to decide whether revision is preferred over recalibration. Because multicategory outcomes increase the number of parameters to be estimated, we recommend full model revision only when the sample size for each outcome category is large.

Evaluating the risk of ovarian cancer before surgery using the ADNEX model: a multicentre external validation study

BACKGROUND

The International Ovarian Tumour Analysis (IOTA) group have developed the ADNEX (The Assessment of Different NEoplasias in the adneXa) model to predict the risk that an ovarian mass is benign, borderline, stage I, stages II-IV or metastatic. We aimed to externally validate the ADNEX model in the hands of examiners with varied training and experience.

METHODS

This was a multicentre cross-sectional cohort study for diagnostic accuracy. Patients were recruited from three cancer centres in Europe. Patients who underwent transvaginal ultrasonography and had a histological diagnosis of surgically removed tissue were included. The diagnostic performance of the ADNEX model with and without the use of CA125 as a predictor was calculated.

RESULTS

Data from 610 women were analysed. The overall prevalence of malignancy was 30%. The area under the receiver operator curve (AUC) for the ADNEX diagnostic performance to differentiate between benign and malignant masses was 0.937 (95% CI: 0.915-0.954) when CA125 was included, and 0.925 (95% CI: 0.902-0.943) when CA125 was excluded. The calibration plots suggest good correspondence between the total predicted risk of malignancy and the observed proportion of malignancies. The model showed good discrimination between the different subtypes.

CONCLUSIONS

The performance of the ADNEX model retains its performance on external validation in the hands of ultrasound examiners with varied training and experience.

The clinical performance of the M4 decision support model to triage women with a pregnancy of unknown location as at low or high risk of complications

STUDY QUESTION

What are the adverse outcomes associated with using the M4 model in everyday clinical practice for women with pregnancy of unknown location (PUL)?

SUMMARY ANSWER

There were 17/835 (2.0%) adverse events and no serious adverse events associated with the performance of the M4 model in clinical practice.

WHAT IS KNOWN ALREADY

The M4 model has previously been shown to stratify women classified as a PUL as at low or high risk of complications with a good level of test performance. The triage performance of the M4 model is better than single measurements of serum progesterone or the hCG ratio (serum hCG at 48 h/hCG at presentation).

STUDY DESIGN, SIZE, DURATION

A prospective multi-centre cohort study of 1022 women with a PUL carried out between August 2012 and December 2013 across 2 university teaching hospitals and 1 district general hospital.

PARTICIPANTS/MATERIALS, SETTING, METHODS

All women presenting with a PUL to the early pregnancy units of the three hospitals were recruited. The final outcome for PUL was either a failed PUL (FPUL), intrauterine pregnancy (IUP) or ectopic pregnancy (EP) (including persistent PUL (PPUL)), with EP and PPUL considered high-risk PUL. Their hCG results at 0 and 48 h were entered into the M4 model algorithm. If the risk of EP was ≥5%, the PUL was predicted to be high-risk and the participant was asked to re-attend 48 h later for a repeat hCG and transvaginal ultrasound scan by a senior clinician. If the PUL was classified as 'low risk, likely failed PUL', the participant was asked to perform a urinary pregnancy test 2 weeks later. If the PUL was classified as 'low risk, likely intrauterine', the participant was scheduled for a repeat scan in 1 week. Deviations from the management protocol were recorded as either an 'unscheduled visit (participant reason)', 'unscheduled visit (clinician reason)' or 'differences in timing (blood test/ultrasound)'. Adverse events were assessed using definitions outlined in the UK Good Clinical Practice Guidelines' document.

MAIN RESULTS AND THE ROLE OF CHANCE

A total of 835 (82%) women classified as a PUL were managed according to the M4 model (9 met the exclusion criteria, 69 were lost to follow-up, 109 had no hCG result at 48 h). Of these, 443 (53%) had a final outcome of FPUL, 298 (36%) an IUP and 94 (11%) an EP. The M4 model predicted 70% (585/835) PUL as low risk, of which 568 (97%) were confirmed as FPUL or IUP. Of the 17 EP and PPUL misclassified as low risk, 5 had expectant management, 7 medical management with methotrexate and 5 surgical intervention.Nineteen PUL had an unscheduled visit (participant reason), 38 PUL had an unscheduled visit (clinician reason) and 68 PUL had deviations from protocol due to a difference in timing (blood test/ultrasound).Adverse events were reported in 26 PUL and 1 participant had a serious adverse event. A total of 17/26 (65%) adverse events were misclassifications of a high risk PUL as low risk by the M4 model, while 5/26 (19%) adverse events were related to incorrect clinical decisions. Four of the 26 adverse events (15%) were secondary to unscheduled admissions for pain/bleeding. The serious adverse event was due to an incorrect clinical decision.

LIMITATIONS, REASONS FOR CAUTION

A limitation of the study was that 69/1022 (7%) of PUL were lost to follow-up. A 48 h hCG level was missing for 109/1022 (11%) participants.

WIDER IMPLICATIONS OF THE FINDINGS

The low number of adverse events (2.0%) suggests that expectant management of PUL using the M4 prediction model is safe. The model is an effective way of triaging women with a PUL as being at high- and low-risk of complications and rationalizing follow-up. The multi-centre design of the study is more likely to make the performance of the M4 model generalizable in other populations.

STUDY FUNDING/COMPETING INTERESTS

None.

TRIAL REGISTRATION NUMBER

Not applicable.

How to Develop, Validate, and Compare Clinical Prediction Models Involving Radiological Parameters: Study Design and Statistical Methods

Clinical prediction models are developed to calculate estimates of the probability of the presence/occurrence or future course of a particular prognostic or diagnostic outcome from multiple clinical or non-clinical parameters. Radiologic imaging techniques are being developed for accurate detection and early diagnosis of disease, which will eventually affect patient outcomes. Hence, results obtained by radiological means, especially diagnostic imaging, are frequently incorporated into a clinical prediction model as important predictive parameters, and the performance of the prediction model may improve in both diagnostic and prognostic settings. This article explains in a conceptual manner the overall process of developing and validating a clinical prediction model involving radiological parameters in relation to the study design and statistical methods. Collection of a raw dataset; selection of an appropriate statistical model; predictor selection; evaluation of model performance using a calibration plot, Hosmer-Lemeshow test and c-index; internal and external validation; comparison of different models using c-index, net reclassification improvement, and integrated discrimination improvement; and a method to create an easy-to-use prediction score system will be addressed. This article may serve as a practical methodological reference for clinical researchers.

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration

The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) Statement includes a 22-item checklist, which aims to improve the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD Statement is explained in detail and accompanied by published examples of good reporting. The document also provides a valuable reference of issues to consider when designing, conducting, and analyzing prediction model studies. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, it is recommended that authors include a completed checklist in their submission. The TRIPOD checklist can also be downloaded from www.tripod-statement.org.

Evaluating the risk of ovarian cancer before surgery using the ADNEX model to differentiate between benign, borderline, early and advanced stage invasive, and secondary metastatic tumours: prospective multicentre diagnostic study

OBJECTIVES

To develop a risk prediction model to preoperatively discriminate between benign, borderline, stage I invasive, stage II-IV invasive, and secondary metastatic ovarian tumours.

DESIGN

Observational diagnostic study using prospectively collected clinical and ultrasound data.

SETTING

24 ultrasound centres in 10 countries.

PARTICIPANTS

Women with an ovarian (including para-ovarian and tubal) mass and who underwent a standardised ultrasound examination before surgery. The model was developed on 3506 patients recruited between 1999 and 2007, temporally validated on 2403 patients recruited between 2009 and 2012, and then updated on all 5909 patients.

MAIN OUTCOME MEASURES

Histological classification and surgical staging of the mass.

RESULTS

The Assessment of Different NEoplasias in the adneXa (ADNEX) model contains three clinical and six ultrasound predictors: age, serum CA-125 level, type of centre (oncology centres v other hospitals), maximum diameter of lesion, proportion of solid tissue, more than 10 cyst locules, number of papillary projections, acoustic shadows, and ascites. The area under the receiver operating characteristic curve (AUC) for the classic discrimination between benign and malignant tumours was 0.94 (0.93 to 0.95) on temporal validation. The AUC was 0.85 for benign versus borderline, 0.92 for benign versus stage I cancer, 0.99 for benign versus stage II-IV cancer, and 0.95 for benign versus secondary metastatic. AUCs between malignant subtypes varied between 0.71 and 0.95, with an AUC of 0.75 for borderline versus stage I cancer and 0.82 for stage II-IV versus secondary metastatic. Calibration curves showed that the estimated risks were accurate.

CONCLUSIONS

The ADNEX model discriminates well between benign and malignant tumours and offers fair to excellent discrimination between four types of ovarian malignancy. The use of ADNEX has the potential to improve triage and management decisions and so reduce morbidity and mortality associated with adnexal pathology.