The accuracy of complications documented in a prospective complication registry

Development and validation of artificial intelligence models for early detection of postoperative infections (PERISCOPE): a multicentre study using electronic health record data

Background

Postoperative infections significantly impact patient outcomes and costs, exacerbated by late diagnoses, yet early reliable predictors are scarce. Existing artificial intelligence (AI) models for postoperative infection prediction often lack external validation or perform poorly in local settings when validated. We aimed to develop locally valid models as part of the PERISCOPE AI system to enable early detection, safer discharge, and more timely treatment of patients.

Methods

We developed and validated XGBoost models to predict postoperative infections within 7 and 30 days of surgery. Using retrospective pre-operative and intra-operative electronic health record data from 2014 to 2023 across various surgical specialities, the models were developed at Hospital A and validated and updated at Hospitals B and C in the Netherlands and Belgium. Model performance was evaluated before and after updating using the two most recent years of data as temporal validation datasets. Main outcome measures were model discrimination (area under the receiver operating characteristic curve (AUROC)), calibration (slope, intercept, and plots), and clinical utility (decision curve analysis with net benefit).

Findings

The study included 253,010 surgical procedures with 23,903 infections within 30-days. Discriminative performance, calibration properties, and clinical utility significantly improved after updating. Final AUROCs after updating for Hospitals A, B, and C were 0.82 (95% confidence interval (CI) 0.81-0.83), 0.82 (95% CI 0.81-0.83), and 0.91 (95% CI 0.90-0.91) respectively for 30-day predictions on the temporal validation datasets (2022-2023). Calibration plots demonstrated adequate correspondence between observed outcomes and predicted risk. All local models were deemed clinically useful as the net benefit was higher than default strategies (treat all and treat none) over a wide range of clinically relevant decision thresholds.

Interpretation

PERISCOPE can accurately predict overall postoperative infections within 7- and 30-days post-surgery. The robust performance implies potential for improving clinical care in diverse clinical target populations. This study supports the need for approaches to local updating of AI models to account for domain shifts in patient populations and data distributions across different clinical settings.

Funding

This study was funded by a REACT EU grant from European Regional Development Fund (ERDF) and Kansen voor West.

Automated Identification of Postoperative Infections to Allow Prediction and Surveillance Based on Electronic Health Record Data: Scoping Review

BACKGROUND

Postoperative infections remain a crucial challenge in health care, resulting in high morbidity, mortality, and costs. Accurate identification and labeling of patients with postoperative bacterial infections is crucial for developing prediction models, validating biomarkers, and implementing surveillance systems in clinical practice.

OBJECTIVE

This scoping review aimed to explore methods for identifying patients with postoperative infections using electronic health record (EHR) data to go beyond the reference standard of manual chart review.

METHODS

We performed a systematic search strategy across PubMed, Embase, Web of Science (Core Collection), the Cochrane Library, and Emcare (Ovid), targeting studies addressing the prediction and fully automated surveillance (ie, without manual check) of diverse bacterial infections in the postoperative setting. For prediction modeling studies, we assessed the labeling methods used, categorizing them as either manual or automated. We evaluated the different types of EHR data needed for the surveillance and labeling of postoperative infections, as well as the performance of fully automated surveillance systems compared with manual chart review.

RESULTS

We identified 75 different methods and definitions used to identify patients with postoperative infections in studies published between 2003 and 2023. Manual labeling was the predominant method in prediction modeling research, 65% (49/75) of the identified methods use structured data, and 45% (34/75) use free text and clinical notes as one of their data sources. Fully automated surveillance systems should be used with caution because the reported positive predictive values are between 0.31 and 0.76.

CONCLUSIONS

There is currently no evidence to support fully automated labeling and identification of patients with infections based solely on structured EHR data. Future research should focus on defining uniform definitions, as well as prioritizing the development of more scalable, automated methods for infection detection using structured EHR data.

Investigating the clinical reasoning abilities of large language model GPT-4: an analysis of postoperative complications from renal surgeries

PURPOSE

Large language models, a subset of artificial intelligence, have immense potential to support human tasks. The role of these models in science and medicine is unclear, requiring strong critical thinking and analysis skills. The objective of our study was to evaluate GPT-4's abilities to assess postoperative complications after renal surgeries.

MATERIALS AND METHODS

Discharge summaries were compiled, and patient information was deidentified in a Python-based program. Prompts were engineered in GPT-4 to assess for the presence of postoperative complications. GPT-4 was further asked to interpret each complication's Clavien-Dindo classification and institutional-specific category. GPT-4's database was compared to a human-curated database. Discrepancies were manually reviewed to calculate match and accuracy rates.

RESULTS

Approximately 944 renal surgeries were conducted from August 2005 to March 2022. There was a 79.6% match rate between GPT-4 and human-curated data in detecting postoperative complications. Accuracy rates were 86.7% for GPT-4 and 92.9% for human-curated. A subgroup of 139 patients had a complication detected by both GPT-4 and human with available Clavien-Dindo classification and category information. There was a 37.4% overall match rate for Clavien-Dindo grade and 55.4% match rate for category.

CONCLUSIONS

GPT-4 was able to accurately detect if there were any postoperative complications. It struggled with the complex task of further analyzing complications, especially with Clavien-Dindo classification, which requires more critical thinking and interpretation. While GPT-4 is not yet ready for advanced postoperative complication analysis, it can still be used to support clinicians in this endeavor.

Patterns of neck metastasis and occult neck disease during recurrent laryngeal cancer

OBJECTIVE

Numerous factors are considered to impact on the rate of complications during salvage total laryngectomy procedures. Neck dissection could be one of these factors. This study analysed the pattern of lymph node metastasis and rate of occult neck disease during salvage total laryngectomy as well as the impact of neck dissection on survival and complication rates.

METHOD

This was a retrospective analysis of a prospectively maintained laryngectomy database in two large tertiary teaching hospitals.

RESULTS

The rate of occult neck disease was 11.1 per cent. Most cases with occult neck disease had rT₄ disease. Patients with complications, advanced tumour stage and positive margins had a significant decrease in overall survival. Patients receiving elective neck dissection did not have any survival benefit. Positron emission tomography-computed tomography showed a very high specificity and negative predictive value.

CONCLUSION

According to the low risk of occult neck disease when using contemporary imaging techniques as well as the lack of impact on survival, conservative management of the neck should be considered for crT₁-T₃ recurrence.

A systematic review and meta-analysis of the cerebrovascular event incidence after transcatheter aortic valve implantation

OBJECTIVE

Periinterventional stroke is one of the most feared potential complication, among patients treated with transcatheter aortic valve implantation (TAVI). The purpose of this review was to investigate the incidence of cerebrovascular events and the influence of postinterventional neurologic check-up in patients undergoing TAVI.

METHODS

A systematic review and meta-analysis were conducted according to the PRISMA guideline. Three separate electronic searches of the public domains Medline and Clinicaltrials.gov were performed to identify the 30-day incidence of stroke within randomized controlled trials (RCTs) and registries for patients undergoing a TAVI procedure. A meta-analysis was conducted to evaluate the 30-day incidence of stroke within RCTs. Furthermore, we pooled the RCTs in which a scheduled neurological check-up was conducted or not to investigate the effect of this intervention.

RESULTS

Twenty-three studies including 399,532,491 TAVI patients were included, 6370 from RCTs, 857,833 from cerebral-embolic protection device RCTs and 392,288 were adopted from registries. The mean 30-day incidence of stroke among all reviewed studies was 2.33%. In RCTs evaluating TAVI the pooled stroke incidence was 3.86%, among RCTs focused CEP the incidence was 6.4436% and in registries the incidence was 2.29%. Ten RCTs conducted scheduled neurological check-ups, the incidence in these was 4.03% and among the remaining RCTs it was 2.47%. In the meta-analysis, the pooled 30-day stroke incidence was 3.61% (95% CI 2.57-4.79%).

CONCLUSION

This systematic review demonstrates that the stroke incidences following TAVI differ strongly according to the study design and neurological follow-up. Intense neurological testing increases the incidence of a stroke after TAVI.

Complications after surgical management of proximal humeral fractures: a systematic review of event terms and definitions

BACKGROUND

The most frequently used surgical procedures for treating a proximal humeral fracture (PHF) are plate osteosynthesis, nail osteosynthesis and arthroplasty. Evidence-based recommendations for an appropriate surgical procedure after PHF requires transparent and valid safety data. We performed a systematic review to examine reported terms and definitions of complications after surgically-treated PHFs.

METHODS

A literature search was conducted on PubMed, Cochrane Library, EMBASE, Scopus and WorldCat to identify clinical articles and book chapters on complications of PHF published from 2010 to 2017. Complication terms and definitions were extracted from each selected article independently by two reviewers and grouped according to a predefined scheme.

RESULTS

From 1376 initial references, we selected 470 articles, of which 103 were reviewed in reverse chronological order until no further information was gained. Twelve book chapters were reviewed. We found 667 local event terms associated with complications after surgical treatment of PHFs. The most frequently used event terms were infection (52 references), nonunion (n = 42), malunion (n = 35), avascular necrosis (n = 27) and pain (n = 25). Overall, 345, 177, 257 and 102 local event terms were related to plating, nailing, arthroplasty and other surgical techniques, respectively. Radiological assessment was the basis for the majority of event terms and complication definitions. Thirty-six event definitions were extracted, mostly defining the terms "secondary fracture displacement", "screw perforation/cutout", "malunion", "delayed healing" and "notching".

CONCLUSION

Scientific literature on surgically-managed PHF uses different terms to describe complications and without approved definitions, which highlights a lack of agreement on adverse event terminology for PHFs. Defined event terms are mostly based on radiological observations. Consensus among shoulder surgeons on a core event set is indispensable to support the standardization of safety reporting for surgically-treated PHFs.

Validating a transnational fracture treatment registry using a standardized method

AIM

Subsequent to a three-month pilot phase, recruiting patients for the newly established BFCC (Baltic Fracture Competence Centre) transnational fracture registry, a validation of the data quality needed to be carried out, applying a standardized method.

METHOD

During the literature research, the method of "adaptive monitoring" fulfilled the requirements of the registry and was applied. It consisted of a three-step audit process; firstly, scoring of the overall data quality, followed by source data verification of a sample size, relative to the scoring result, and finally, feedback to the registry on measures to improve data quality. Statistical methods for scoring of data quality and visualisation of discrepancies between registry data and source data were developed and applied.

RESULTS

Initially, the data quality of the registry scored as medium. During source data verification, missing items in the registry, causing medium data quality, turned out to be absent in the source as well. A subsequent adaptation of the score evaluated the registry's data quality as good. It was suggested to add variables to some items in order to improve the accuracy of the registry.

DISCUSSION

The application of the method of adaptive monitoring has only been published by Jacke et al., with a similar improvement of the scoring result following the audit process. Displaying data from the registry in graphs helped to find missing items and discover issues with data formats. Graphically comparing the degree of agreement between the registry and source data allowed to discover systematic faults.

CONCLUSIONS

The method of adaptive monitoring gives a substantiated guideline for systematically evaluating and monitoring a registry's data quality and is currently second to none. The resulting transparency of the registry's data quality could be helpful in annual reports, as published by most major registries. As the method has been rarely applied, further successive applications in established registries would be desirable.

Complications, reoperations and revisions after proximal interphalangeal joint arthroplasty: a systematic review and meta-analysis

This systematic review and meta-analysis investigates the prevalence of complications, reoperations (surgeries without implant modifications) and revisions (surgeries with implant modifications) after proximal interphalangeal joint arthroplasty with pyrocarbon, metal-polyethylene and silicone implants. Thirty-four articles investigating 1868 proximal interphalangeal joints were included. Implant-related complications were associated with 14%, 10% and 11% of the pyrocarbon, metal-polyethylene and silicone implants, respectively, yet these rates were not significantly different from one another. Silicone implants showed more finger deviations (3%) and instabilities (2%) compared with the other implants. Reoperations were fewer for silicone arthroplasties (1%) compared with pyrocarbon (7%) and metal-polyethylene implants (10%). The revision rates of 4%, 3% and 2% were similar for pyrocarbon, metal-polyethylene and silicone implants. Our results indicate that silicone implants remain a valuable option for the treatment of stable proximal interphalangeal joints. Surface replacing implants might be better to correct unstable or deviated proximal interphalangeal joints, although they are associated with a higher risk of reoperations.

Towards standardised definitions of shoulder arthroplasty complications: a systematic review of terms and definitions

INTRODUCTION

A transparent, reliable and accurate reporting of complications is essential for an evidence-based evaluation of shoulder arthroplasty (SA). We systematically reviewed the literature for terms and definitions related to negative events associated with SA.

MATERIALS AND METHODS

Various biomedical databases were searched for reviews, clinical studies and case reports of complications associated with SA. Any general definition of a complication, classification system, all reported terms related to complications and negative events with their definitions were extracted. Terms were grouped and organised in a hierarchical structure. Definitions of negative events were tabulated and compared.

RESULTS

From 1086 initial references published between 2010 and 2014, 495 full-text papers were reviewed. Five reports provided a general definition of the term "surgical complication" and 29 used a classification system of complications. A total of 1399 extracted terms were grouped based on similarities and involved implant or anatomical parts. One hundred and six reports (21.4%) defined at least one negative event for 28 different terms. There were 64 definitions related to humeral or glenoid loosening, and 25 systems documenting periprosthetic radiolucency. Other definitions considered notching, stress shielding, implant failure and tuberosity malposition.

CONCLUSIONS

A clear standardised set of SA complication definitions is lacking. Few authors reported complications based on definitions mainly considering radiological criteria without clinical parameters. This review should initiate and support the development of a standardised SA complication core set.

Incidence and causes of silent and symptomatic stroke following surgical and transcatheter aortic valve replacement: a comprehensive review

Stroke associated with aortic valve replacement in calcific aortic stenosis, either via transcatheter implantation (TAVR) or via surgical replacement (SAVR), is one of the most devastating complications. However, data concerning the clinical impact and incidence of clinical and silent stroke complicating SAVR and TAVR are varying. This comprehensive review of the literature explores the genuine incidence of neurological events after these procedures. Additionally, potential factors responsible for the discrepancies in stroke rates in the current literature are analysed and a lack of uniform neurological definitions and standardized neurological assessments revealed. Current stroke rates after TAVR show a decline from 7 to 1.7-4.8% in recent studies. Randomized studies comparing TAVR with SAVR yielded initially a significantly higher stroke rate after TAVR procedures as opposed to SAVR. Recently published data showed opposite results with strokes being higher following SAVR. Current data concerning stroke after surgical valve replacement report significantly higher rates of clinical strokes (17%) than previously mentioned in the literature (≤4.9%). Silent cerebral lesions were detected in 68-93% after TAVR and 38-54% after SAVR. A broader application of cerebral protection devices may help to reduce embolic cerebral events.

Technical errors and complications in orthopaedic trauma surgery

INTRODUCTION

Adverse events and associated morbidity and subsequent costs receive increasing attention in clinical practice and research. As opposed to complications, errors are not described or analysed in literature on fracture surgery. The aim of this study was to provide a description of errors and complications in relation to fracture surgery, as well as the circumstances in which they occur, for example urgency, type of surgeon, and type of fracture.

METHODS

All errors and complications were recorded prospectively in our hospital's complication registry, which forms an integral part of the electronic medical patient file. All recorded errors and complications in the complication registry linked to fracture surgery between 1 January, 2000 and 31 December, 2010 were analysed.

RESULTS

During the study period 4310 osteosynthesis procedures were performed. In 78 (1.8 %) procedures an error in osteosynthesis was registered. The number of procedures in which an error occurred was significantly lower (OR = 0.53; p = 0.007) when an orthopaedic trauma surgeon was part of the operating team. Of all 3758 patients who were admitted to the surgical ward for osteosynthesis, 745 (19.8 %) had one or more postoperative complications registered. There was no significant difference in the number of postoperative complications after osteosynthesis procedures in which an orthopaedic trauma surgeon was present or absent (16.7 vs. 19.1 %; p = 0.088; OR 0.85).

DISCUSSION

In the present study the true error rate after osteosynthesis may have been higher than the rate found. Errors that had no significant consequence may be especially susceptible to underreporting.

CONCLUSION

The present study suggests that an osteosynthesis procedure performed by or actively assisted by an orthopaedic trauma surgeon decreases the probability of an error in osteosynthesis. Apart from errors in osteosynthesis, the involvement of an orthopaedic trauma surgeon did not lead to a significant reduction in the number of postoperative complications.

Complications Following Arthroscopic Rotator Cuff Tear Repair: A Systematic Review of Terms and Definitions With Focus on Shoulder Stiffness

Background

Valid comparison of outcomes after surgical procedures requires consensus on which instruments and parameters should be used, including the recording and evaluation of surgical complications. An international standard outlining the terminology and definitions of surgical complications in orthopaedics is lacking.

Purpose

This study systematically reviewed the literature for terms and definitions related to the occurrence of negative events or complications after arthroscopic rotator cuff repair (ARCR) with specific focus on shoulder stiffness.

Study Design

Systematic review; Level of evidence, 4.

Methods

PubMed, EMBASE, Cochrane Library, and Scopus databases were searched for reviews, clinical studies, and case reports of complications associated with ARCR. Reference lists of selected articles were also screened. The terminology of complications and their definitions were extracted from all relevant original articles by a single reviewer and verified by a second reviewer. Definitions of shoulder stiffness or equivalent terms were tabulated.

Results

Of 654 references published after 2007 and obtained from the search, 233 full-text papers (44 reviews, 155 studies, 31 case reports, and 3 surgical technique presentations) were reviewed. Twenty-two additional references cited for a definition were checked. One report defined the term surgical complication. There were 242 different terms used to describe local events and 64 to describe nonlocal events. Furthermore, 16 definitions of terms such as frozen shoulder, shoulder stiffness, or stiff painful shoulder were identified. Diagnosis criteria for shoulder stiffness differed widely; 12 various definitions for restriction in range of motion were noted. One definition included a gradation of stiffness severity, whereas another considered the patient’s subjective assessment of motion.

Conclusion

The literature does not consistently report on complications after ARCR, making valid comparison of the incidence of these events among published reports impossible. Specifically, the variation in criteria used to diagnose shoulder stiffness is problematic for valid and accurate reporting of this event. A standard for reporting this event and other complications after ARCR is needed.

Clinical Relevance

This review serves as the basis for the development of a uniform documentation process for shoulder stiffness and the standardization of complication definitions in ARCR following international consensus.

Which clinical scenarios do surgeons record as complications? A benchmarking study of seven hospitals

OBJECTIVES

To investigate agreement and potential differences in the application and interpretation of the definition among surgical departments of various hospitals.

DESIGN

24 cases were formulated including general, trauma, gastrointestinal and vascular surgery, and based on points of discussion about the definition and ambiguities regarding complication registration as encountered in daily practice. The cases were presented to the surgical staff and residents in seven Dutch hospitals, using the national registration system of complications and an electronic response system.

RESULTS

In total, 134 participants responded. Interpretation differences were particularly found regarding: (1) complications considered as logical consequences of a surgical procedure; (2) complications occurring after radiological interventions; (3) severity criteria such as when to consider a complication as a '(probably) permanent damage or function loss'; (4) registering a cancelled operation as a complication and (5) patients with serial complications during hospital stay.

CONCLUSIONS

The definition of surgical complications as currently applied in the Netherlands does not ensure a uniform complication registration. Improvement of this registration system is mandatory before benchmarking of these findings in the public domain is appropriate. Modifications of the current definition of a surgical complication, and improved consensus about specific clinical situations and training of surgeons might improve the quality of benchmarking.

[Quality of documentation of intraoperative and postoperative complications : improvement of documentation for a nationwide quality assurance program and comparison with routine data]

INTRODUCTION

Complications after cholecystectomy are continuously documented in a nationwide database in Germany. Recent studies demonstrated a lack of reliability of these data. The aim of the study was to evaluate the impact of a control algorithm on documentation quality and the use of routine diagnosis coding as an additional validation instrument.

METHODS

Completeness and correctness of the documentation of complications after cholecystectomy was compared over a time interval of 12 months before and after implementation of an algorithm for faster and more accurate documentation. Furthermore, the coding of all diagnoses was screened to identify intraoperative and postoperative complications.

RESULTS AND DISCUSSION

The sensitivity of the documentation for complications improved from 46 % to 70 % (p = 0.05, specificity 98 % in both time intervals). A prolonged time interval of more than 6 weeks between patient discharge and documentation was associated with inferior data quality (incorrect documentation in 1.5 % versus 15 %, p < 0.05). The rate of case documentation within the 6 weeks after hospital discharge was clearly improved after implementation of the control algorithm. Sensitivity and specificity of screening for complications by evaluating routine diagnoses coding were 70 % and 85 %, respectively. The quality of documentation was improved by implementation of a simple memory algorithm.

Developing a Registry for Complications in Otorhinolaryngologic Surgery: Tonsil Surgery as a Pilot Cohort

OBJECTIVE

To find a suitable method to prospectively register all tonsil surgery-related complications.

STUDY DESIGN

Prospective cohort study.

SETTING

Tertiary care center.

SUBJECTS AND METHODS

From September 2011 to February 2012, patients undergoing tonsillectomy or tonsillotomy were enrolled. A wide range of demographic and clinical data including incidents of postoperative complications was recorded prospectively, and patient records were reviewed 9 months after the end of study period. We evaluated the coverage of prospective data recording, analyzed the complication rates, and assessed the process of registration.

RESULTS

A total of 573 patients were recruited. The study registry including 57 variables required the completion of missing data before analysis. Of all 79 patients with a complication, 69.6% were captured prospectively at the emergency department, and the rest were found when reviewing the patient records. The proportion of prospectively captured complications was highest for the most common complications (eg, 81.1% for secondary hemorrhage). The overall complication rate was 13.8%. Secondary hemorrhage was the most common complication, with the incidence of 9.6%.

CONCLUSION

We have demonstrated the initial feasibility of a prospective complication registry for otorhinolaryngology procedures, and the results can be applied accordingly. We also present 5 practical recommendations when initiating a functional registry. Particular attention should be paid to recognition and registration of both rare and serious events. Regular analysis of the results is required in order to respond to possible changes in the incidence or nature of complications.

Stroke after aortic valve surgery: results from a prospective cohort

BACKGROUND

The incidence and impact of clinical stroke and silent radiographic cerebral infarction complicating open surgical aortic valve replacement (AVR) are poorly characterized.

METHODS AND RESULTS

We performed a prospective cohort study of subjects ≥65 years of age who were undergoing AVR for calcific aortic stenosis. Subjects were evaluated by neurologists preoperatively and postoperatively and underwent postoperative magnetic resonance imaging. Over a 4-year period, 196 subjects were enrolled at 2 sites (mean age, 75.8±6.2 years; 36% women; 6% nonwhite). Clinical strokes were detected in 17%, transient ischemic attack in 2%, and in-hospital mortality was 5%. The frequency of stroke in the Society for Thoracic Surgery database in this cohort was 7%. Most strokes were mild; the median National Institutes of Health Stroke Scale was 3 (interquartile range, 1-9). Clinical stroke was associated with increased length of stay (median, 12 versus 10 days; P=0.02). Moderate or severe stroke (National Institutes of Health Stroke Scale ≥10) occurred in 8 (4%) and was strongly associated with in-hospital mortality (38% versus 4%; P=0.005). Of the 109 stroke-free subjects with postoperative magnetic resonance imaging, silent infarct was identified in 59 (54%). Silent infarct was not associated with in-hospital mortality or increased length of stay.

CONCLUSIONS

Clinical stroke after AVR was more common than reported previously, more than double for this same cohort in the Society for Thoracic Surgery database, and silent cerebral infarctions were detected in more than half of the patients undergoing AVR. Clinical stroke complicating AVR is associated with increased length of stay and mortality.