1988: use of a Bayesian statistical model for risk assessment in coronary artery surgery. Updated in 1995

Recursive and non-linear logistic regression: moving on from the original EuroSCORE and EuroSCORE II methodologies

OBJECTIVES

EuroSCORE II, despite improving on the original EuroSCORE system, has not solved all the calibration and predictability issues. Recursive, non-linear and mixed recursive and non-linear regression analysis were assessed with regard to sensitivity, specificity and predictability of the original EuroSCORE and EuroSCORE II systems.

METHODS

The original logistic EuroSCORE, EuroSCORE II and recursive, non-linear and mixed recursive and non-linear regression analyses of these risk models were assessed via receiver operator characteristic curves (ROC) and Hosmer-Lemeshow statistic analysis with regard to the accuracy of predicting in-hospital mortality. Analysis was performed for isolated coronary artery bypass grafts (CABGs) (n = 2913), aortic valve replacement (AVR) (n = 814), mitral valve surgery (n = 340), combined AVR and CABG (n = 517), aortic (n = 350), miscellaneous cases (n = 642), and combinations of the above cases (n = 5576).

RESULTS

The original EuroSCORE had an ROC below 0.7 for isolated AVR and combined AVR and CABG. None of the methods described increased the ROC above 0.7. The EuroSCORE II risk model had an ROC below 0.7 for isolated AVR only. Recursive regression, non-linear regression, and mixed recursive and non-linear regression all increased the ROC above 0.7 for isolated AVR. The original EuroSCORE had a Hosmer-Lemeshow statistic that was above 0.05 for all patients and the subgroups analysed. All of the techniques markedly increased the Hosmer-Lemeshow statistic. The EuroSCORE II risk model had a Hosmer-Lemeshow statistic that was significant for all patients (P < 0.0001), and very close to significant for isolated CABG (P = 0.05) and for isolated AVR (P = 0.06). Non-linear regression failed to improve on the original Hosmer-Lemeshow statistic. The mixed recursive and non-linear regression using the EuroSCORE II risk model was the only model that produced an ROC of 0.7 or above for all patients and procedures and had a Hosmer-Lemeshow statistic that was highly non-significant.

CONCLUSIONS

The original EuroSCORE and the EuroSCORE II risk models do not have adequate ROC and Hosmer-Lemeshow statistics to allow accurate assessment of cardiac surgeons in the modern era. A mixed recursive and non-linear regression model utilizing the EuroSCORE II risk model improves both the ROC and Hosmer-Lemeshow statistics.

Surgical process modelling: a review

PURPOSE

Surgery is continuously subject to technological and medical innovations that are transforming daily surgical routines. In order to gain a better understanding and description of surgeries, the field of surgical process modelling (SPM) has recently emerged. The challenge is to support surgery through the quantitative analysis and understanding of operating room activities. Related surgical process models can then be introduced into a new generation of computer-assisted surgery systems.

METHODS

In this paper, we present a review of the literature dealing with SPM. This methodological review was obtained from a search using Google Scholar on the specific keywords: "surgical process analysis", "surgical process model" and "surgical workflow analysis".

RESULTS

This paper gives an overview of current approaches in the field that study the procedural aspects of surgery. We propose a classification of the domain that helps to summarise and describe the most important components of each paper we have reviewed, i.e., acquisition, modelling, analysis, application and validation/evaluation. These five aspects are presented independently along with an exhaustive list of their possible instantiations taken from the studied publications.

CONCLUSION

This review allows a greater understanding of the SPM field to be gained and introduces future related prospects.

A molecular computational model improves the preoperative diagnosis of thyroid nodules

Background

Thyroid nodules with indeterminate cytological features on fine needle aspiration (FNA) cytology have a 20% risk of thyroid cancer. The aim of the current study was to determine the diagnostic utility of an 8-gene assay to distinguish benign from malignant thyroid neoplasm.

Methods

The mRNA expression level of 9 genes (KIT, SYNGR2, C21orf4, Hs.296031, DDI2, CDH1, LSM7, TC1, NATH) was analysed by quantitative PCR (q-PCR) in 93 FNA cytological samples. To evaluate the diagnostic utility of all the genes analysed, we assessed the area under the curve (AUC) for each gene individually and in combination. BRAF exon 15 status was determined by pyrosequencing. An 8-gene computational model (Neural Network Bayesian Classifier) was built and a multiple-variable analysis was then performed to assess the correlation between the markers.

Results

The AUC for each significant marker ranged between 0.625 and 0.900, thus all the significant markers, alone and in combination, can be used to distinguish between malignant and benign FNA samples. The classifier made up of KIT, CDH1, LSM7, C21orf4, DDI2, TC1, Hs.296031 and BRAF had a predictive power of 88.8%. It proved to be useful for risk stratification of the most critical cytological group of the indeterminate lesions for which there is the greatest need of accurate diagnostic markers.

Conclusion

The genetic classification obtained with this model is highly accurate at differentiating malignant from benign thyroid lesions and might be a useful adjunct in the preoperative management of patients with thyroid nodules.

Development of a Bayesian model to estimate health care outcomes in the severely wounded

BACKGROUND

Graphical probabilistic models have the ability to provide insights as to how clinical factors are conditionally related. These models can be used to help us understand factors influencing health care outcomes and resource utilization, and to estimate morbidity and clinical outcomes in trauma patient populations.

STUDY DESIGN

Thirty-two combat casualties with severe extremity injuries enrolled in a prospective observational study were analyzed using step-wise machine-learned Bayesian belief network (BBN) and step-wise logistic regression (LR). Models were evaluated using 10-fold cross-validation to calculate area-under-the-curve (AUC) from receiver operating characteristics (ROC) curves.

RESULTS

Our BBN showed important associations between various factors in our data set that could not be developed using standard regression methods. Cross-validated ROC curve analysis showed that our BBN model was a robust representation of our data domain and that LR models trained on these findings were also robust: hospital-acquired infection (AUC: LR, 0.81; BBN, 0.79), intensive care unit length of stay (AUC: LR, 0.97; BBN, 0.81), and wound healing (AUC: LR, 0.91; BBN, 0.72) showed strong AUC.

CONCLUSIONS

A BBN model can effectively represent clinical outcomes and biomarkers in patients hospitalized after severe wounding, and is confirmed by 10-fold cross-validation and further confirmed through logistic regression modeling. The method warrants further development and independent validation in other, more diverse patient populations.

Development of a clinical decision model for thyroid nodules

Background

Thyroid nodules represent a common problem brought to medical attention. Four to seven percent of the United States adult population (10–18 million people) has a palpable thyroid nodule, however the majority (>95%) of thyroid nodules are benign. While, fine needle aspiration remains the most cost effective and accurate diagnostic tool for thyroid nodules in current practice, over 20% of patients undergoing FNA of a thyroid nodule have indeterminate cytology (follicular neoplasm) with associated malignancy risk prevalence of 20–30%. These patients require thyroid lobectomy/isthmusectomy purely for the purpose of attaining a definitive diagnosis. Given that the majority (70–80%) of these patients have benign surgical pathology, thyroidectomy in these patients is conducted principally with diagnostic intent. Clinical models predictive of malignancy risk are needed to support treatment decisions in patients with thyroid nodules in order to reduce morbidity associated with unnecessary diagnostic surgery.

Methods

Data were analyzed from a completed prospective cohort trial conducted over a 4-year period involving 216 patients with thyroid nodules undergoing ultrasound (US), electrical impedance scanning (EIS) and fine needle aspiration cytology (FNA) prior to thyroidectomy. A Bayesian model was designed to predict malignancy in thyroid nodules based on multivariate dependence relationships between independent covariates. Ten-fold cross-validation was performed to estimate classifier error wherein the data set was randomized into ten separate and unique train and test sets consisting of a training set (90% of records) and a test set (10% of records). A receiver-operating-characteristics (ROC) curve of these predictions and area under the curve (AUC) were calculated to determine model robustness for predicting malignancy in thyroid nodules.

Results

Thyroid nodule size, FNA cytology, US and EIS characteristics were highly predictive of malignancy. Cross validation of the model created with Bayesian Network Analysis effectively predicted malignancy [AUC = 0.88 (95%CI: 0.82–0.94)] in thyroid nodules. The positive and negative predictive values of the model are 83% (95%CI: 76%–91%) and 79% (95%CI: 72%–86%), respectively.

Conclusion

An integrated predictive decision model using Bayesian inference incorporating readily obtainable thyroid nodule measures is clinically relevant, as it effectively predicts malignancy in thyroid nodules. This model warrants further validation testing in prospective clinical trials.

A comparative analysis of predictive models of morbidity in intensive care unit after cardiac surgery - part II: an illustrative example

BACKGROUND

Popular predictive models for estimating morbidity probability after heart surgery are compared critically in a unitary framework. The study is divided into two parts. In the first part modelling techniques and intrinsic strengths and weaknesses of different approaches were discussed from a theoretical point of view. In this second part the performances of the same models are evaluated in an illustrative example.

METHODS

Eight models were developed: Bayes linear and quadratic models, k-nearest neighbour model, logistic regression model, Higgins and direct scoring systems and two feed-forward artificial neural networks with one and two layers. Cardiovascular, respiratory, neurological, renal, infectious and hemorrhagic complications were defined as morbidity. Training and testing sets each of 545 cases were used. The optimal set of predictors was chosen among a collection of 78 preoperative, intraoperative and postoperative variables by a stepwise procedure. Discrimination and calibration were evaluated by the area under the receiver operating characteristic curve and Hosmer-Lemeshow goodness-of-fit test, respectively.

RESULTS

Scoring systems and the logistic regression model required the largest set of predictors, while Bayesian and k-nearest neighbour models were much more parsimonious. In testing data, all models showed acceptable discrimination capacities, however the Bayes quadratic model, using only three predictors, provided the best performance. All models showed satisfactory generalization ability: again the Bayes quadratic model exhibited the best generalization, while artificial neural networks and scoring systems gave the worst results. Finally, poor calibration was obtained when using scoring systems, k-nearest neighbour model and artificial neural networks, while Bayes (after recalibration) and logistic regression models gave adequate results.

CONCLUSION

Although all the predictive models showed acceptable discrimination performance in the example considered, the Bayes and logistic regression models seemed better than the others, because they also had good generalization and calibration. The Bayes quadratic model seemed to be a convincing alternative to the much more usual Bayes linear and logistic regression models. It showed its capacity to identify a minimum core of predictors generally recognized as essential to pragmatically evaluate the risk of developing morbidity after heart surgery.

A comparative analysis of predictive models of morbidity in intensive care unit after cardiac surgery - part I: model planning

Background

Different methods have recently been proposed for predicting morbidity in intensive care units (ICU). The aim of the present study was to critically review a number of approaches for developing models capable of estimating the probability of morbidity in ICU after heart surgery. The study is divided into two parts. In this first part, popular models used to estimate the probability of class membership are grouped into distinct categories according to their underlying mathematical principles. Modelling techniques and intrinsic strengths and weaknesses of each model are analysed and discussed from a theoretical point of view, in consideration of clinical applications.

Methods

Models based on Bayes rule, k-nearest neighbour algorithm, logistic regression, scoring systems and artificial neural networks are investigated. Key issues for model design are described. The mathematical treatment of some aspects of model structure is also included for readers interested in developing models, though a full understanding of mathematical relationships is not necessary if the reader is only interested in perceiving the practical meaning of model assumptions, weaknesses and strengths from a user point of view.

Results

Scoring systems are very attractive due to their simplicity of use, although this may undermine their predictive capacity. Logistic regression models are trustworthy tools, although they suffer from the principal limitations of most regression procedures. Bayesian models seem to be a good compromise between complexity and predictive performance, but model recalibration is generally necessary. k-nearest neighbour may be a valid non parametric technique, though computational cost and the need for large data storage are major weaknesses of this approach. Artificial neural networks have intrinsic advantages with respect to common statistical models, though the training process may be problematical.

Conclusion

Knowledge of model assumptions and the theoretical strengths and weaknesses of different approaches are fundamental for designing models for estimating the probability of morbidity after heart surgery. However, a rational choice also requires evaluation and comparison of actual performances of locally-developed competitive models in the clinical scenario to obtain satisfactory agreement between local needs and model response. In the second part of this study the above predictive models will therefore be tested on real data acquired in a specialized ICU.

A multivariate Bayesian model for assessing morbidity after coronary artery surgery

Introduction

Although most risk-stratification scores are derived from preoperative patient variables, there are several intraoperative and postoperative variables that can influence prognosis. Higgins and colleagues previously evaluated the contribution of preoperative, intraoperative and postoperative predictors to the outcome. We developed a Bayes linear model to discriminate morbidity risk after coronary artery bypass grafting and compared it with three different score models: the Higgins' original scoring system, derived from the patient's status on admission to the intensive care unit (ICU), and two models designed and customized to our patient population.

Methods

We analyzed 88 operative risk factors; 1,090 consecutive adult patients who underwent coronary artery bypass grafting were studied. Training and testing data sets of 740 patients and 350 patients, respectively, were used. A stepwise approach enabled selection of an optimal subset of predictor variables. Model discrimination was assessed by receiver operating characteristic (ROC) curves, whereas calibration was measured using the Hosmer-Lemeshow goodness-of-fit test.

Results

A set of 12 preoperative, intraoperative and postoperative predictor variables was identified for the Bayes linear model. Bayes and locally customized score models fitted according to the Hosmer-Lemeshow test. However, the comparison between the areas under the ROC curve proved that the Bayes linear classifier had a significantly higher discrimination capacity than the score models. Calibration and discrimination were both much worse with Higgins' original scoring system.

Conclusion

Most prediction rules use sequential numerical risk scoring to quantify prognosis and are an advanced form of audit. Score models are very attractive tools because their application in routine clinical practice is simple. If locally customized, they also predict patient morbidity in an acceptable manner. The Bayesian model seems to be a feasible alternative. It has better discrimination and can be tailored more easily to individual institutions.

Bayesian analysis: a new statistical paradigm for new technology

Full Bayesian analysis is an alternative statistical paradigm, as opposed to traditionally used methods, usually called frequentist statistics. Bayesian analysis is controversial because it requires assuming a prior distribution, which can be arbitrarily chosen; thus there is a subjective element, which is considered to be a major weakness. However, this could also be considered a strength since it provides a formal way of incorporating prior knowledge. Since it is flexible and permits repeated looks at evolving data, Bayesian analysis is particularly well suited to the evaluation of new medical technology. Bayesian analysis can refer to a range of things: from a simple, noncontroversial formula for inverting probabilities to an alternative approach to the philosophy of science. Its advantages include: (1) providing direct probability statements--which are what most people wrongly assume they are getting from conventional statistics; (2) formally incorporating previous information in statistical inference of a data set, a natural approach which we follow in everyday reasoning; and (3) flexible, adaptive research designs allowing multiple looks at accumulating study data. Its primary disadvantage is the element of subjectivity which some think is not scientific. We discuss and compare frequentist and Bayesian approaches and provide three examples of Bayesian analysis: (1) EKG interpretation, (2) a coin-tossing experiment, and (3) assessing the thromboembolic risk of a new mechanical heart valve.

Development of a risk adjustment mortality model using the American College of Cardiology-National Cardiovascular Data Registry (ACC-NCDR) experience: 1998-2000

OBJECTIVES

We sought to develop and evaluate a risk adjustment model for in-hospital mortality following percutaneous coronary intervention (PCI) procedures using data from a large, multi-center registry.

BACKGROUND

The 1998-2000 American College of Cardiology-National Cardiovascular Data Registry (ACC-NCDR) dataset was used to overcome limitations of prior risk-adjustment analyses.

METHODS

Data on 100,253 PCI procedures collected at the ACC-NCDR between January 1, 1998, and September 30, 2000, were analyzed. A training set/test set approach was used. Separate models were developed for presentation with and without acute myocardial infarction (MI) within 24 h.

RESULTS

Factors associated with increased risk of PCI mortality (with odds ratios in parentheses) included cardiogenic shock (8.49), increasing age (2.61 to 11.25), salvage (13.38) urgent (1.78) or emergent PCI (5.75), pre-procedure intra-aortic balloon pump insertion (1.68), decreasing left ventricular ejection fraction (0.87 to 3.93), presentation with acute MI (1.31), diabetes (1.41), renal failure (3.04), chronic lung disease (1.33); treatment approaches including thrombolytic therapy (1.39) and non-stent devices (1.64); and lesion characteristics including left main (2.04), proximal left anterior descending disease (1.97) and Society for Cardiac Angiography and Interventions lesion classification (1.64 to 2.11). Overall, excellent discrimination was achieved (C-index = 0.89) and application of the model to high-risk patient groups demonstrated C-indexes exceeding 0.80. Patient factors were more predictive in the MI model, while lesion and procedural factors were more predictive in the analysis of non-MI patients.

CONCLUSIONS

A risk adjustment model for in-hospital mortality after PCI was successfully developed using a contemporary multi-center registry. This model is an important tool for valid comparison of in-hospital mortality after PCI.

Prediction rules for complications in coronary bypass surgery: a comparison and methodological critique

BACKGROUND

Clinical prediction rules have been developed that use preoperative information to stratify patients according to risk of complications after cardiac surgery.

OBJECTIVES

To assess the methodological standards and performance of 7 models.

PARTICIPANTS

The validation portion of the Quality Measurement and Management Initiative (QMMI) cohort included a random sample of all adult patients (n = 3,261) who underwent coronary artery bypass grafting (CABG) surgery not involving valvular or other concomitant procedures at 12 medical centers from August 1993 to October 1995.

OUTCOME MEASURES

Methodological standards used for model comparison were adapted from published criteria. Model performance was assessed by receiver-operating characteristic (ROC) analysis, and calibration was evaluated with the Hosmer-Lemeshow (HL) statistic and observed-expected plots.

METHODS

We performed cross-validation by applying the published criteria for the development of each model to the validation subset of the QMMI cohort and by assessing the performance of each model in discriminating outcomes.

RESULTS

Wide variations existed in the methodologies used to develop and validate the 5 additive scores evaluated. Cross-validation of all 5 additive scores revealed degradation in their abilities to discriminate outcomes. The 2 logistic models examined performed similarly to the additive scores examined in predicting mortality.

CONCLUSIONS

Substantial variation existed both in the methodologies used to develop models and in the ability of the models to predict outcomes. Models developed at single institutions or using fewer patients may be less generalizable when applied to diverse clinical settings. Additive and logistic regression models performed similarly, as assessed by ROC and HL analyses.