APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support

Identification of Key Genes and Potential Therapeutic Targets in Sepsis-Associated Acute Kidney Injury Using Transformer and Machine Learning Approaches

Sepsis-associated acute kidney injury (SA-AKI) is a life-threatening complication of sepsis, characterized by high mortality and prolonged hospitalization. Early diagnosis and effective therapy remain difficult despite extensive investigation. To address this, we developed an AI-driven integrative framework that combines a Transformer-based deep learning model with established machine learning techniques (LASSO, SVM-RFE, Random Forest and neural networks) to uncover complex, nonlinear interactions among gene-expression biomarkers. Analysis of normalized microarray data from GEO (GSE95233 and GSE69063) identified differentially expressed genes (DEGs), and KEGG/GO enrichment via clusterProfiler revealed key pathways in immune response, protein synthesis, and antigen presentation. By integrating multiple transcriptomic cohorts, we pinpointed 617 SA-AKI-associated DEGs-21 of which overlapped between sepsis and AKI datasets. Our Transformer-based classifier ranked five genes (MYL12B, RPL10, PTBP1, PPIA, and TOMM7) as top diagnostic markers, with AUC values ranging from 0.9395 to 0.9996 (MYL12B yielding 0.9996). Drug-gene interaction mining using DGIdb (FDR < 0.05) nominated 19 candidate therapeutics for SA-AKI. Together, these findings demonstrate that melding deep learning with classical machine learning not only sharpens early SA-AKI detection but also systematically uncovers actionable drug targets, laying groundwork for precision intervention in critical care settings.

Implications of artificial intelligence in periodontal treatment maintenance: a scoping review

Gingivitis and periodontitis, are widespread conditions with diverse influence on oral and systemic health. Traditional diagnostic methods in periodontology often rely on subjective clinical assessments, which can lead to variability and inconsistencies in care. Imbibing artificial intelligence (AI) facilitates a significant solution by enhancing precision metrics, treatment planning, and personalized care. Studies published between 2018 and 2024 was conducted to evaluate AI applications in periodontal maintenance. Databases such as PubMed, Cochrane, Web of Science and Scopus were searched using keywords like "artificial intelligence," "machine learning," and "periodontitis." Studies employing AI for diagnosis, prognosis, or periodontal maintenance using clinical or radiographic data were included. Deep learning algorithms such as convolutional neural networks (CNNs) and segmentation techniques were analyzed for their diagnostic accuracy. AI demonstrated superior performance in detecting periodontal conditions, with accuracy rates surpassing 90% in some studies. Advanced models, such as Multi-Label U-Net, exhibited high precision in radiographic analyses, outperforming traditional methods. Additionally, AI facilitated predictive analytics for disease progression and personalized treatment strategies. AI has transformed periodontal care, offering accuracy, personalized care, and efficient workflow integration. Addressing challenges like standardization and ethical concerns is critical for its broader adoption.

Endoscopic size measurement of colorectal polyps: a systematic review of techniques

Accurate size measurement of colorectal polyps is critical for clinical decision making and patient management. This systematic review aimed to evaluate the current techniques used for colonic polyp measurement to improve the reliability of size estimations in routine practice.A comprehensive literature search was conducted across PubMed, EMBASE, and MEDLINE to identify studies relevant to size measurement techniques published between 1980 and March 2024. The primary outcome was the accuracy of polyp sizing techniques used during colonoscopy.61 studies were included with 34 focusing on unassisted and assisted endoscopic visual estimation and 27 on computer-based tools. There was significant variability in visual size estimation among endoscopists. The most accurate techniques identified were computer-based systems, such as virtual scale endoscopes (VSE) and artificial intelligence (AI)-based systems. The least accurate techniques were visual or snare-based polyp size estimation. VSE assists endoscopists by providing an adaptive scale for real-time, direct, in vivo polyp measurements, while AI systems offer size measurements independent of the endoscopist's subjective judgment.This review highlights the need for standardized, accurate, and accessible techniques to optimize sizing accuracy during endoscopic procedures. There is no consensus on a gold standard for measuring polyps during colonoscopy. While biopsy forceps, snare, and graduated devices can improve the accuracy of visual size estimation, their clinical implementation is limited by practical, time, and cost challenges. Computer-based techniques will likely offer improved accuracy of polyp sizing in the near future.

International visualization analysis of research hotspots and development trends in the study of clinical decision support systems utilizing CiteSpace

Objective

This study aims to elucidate the current status and trends in clinical decision support systems (CDSS). It will analyze the direction of research development in this field and provide valuable references for future research and the application of CDSS.

Methods

We conducted a search of the Web of Science Core Collection database from January 2014 to May 2024 to identify relevant literature on clinical decision support systems. CiteSpace (6.2. R4) software was utilized to visualize and analyze various aspects of the included literature, including publication volume, country of origin, authors, institutions, cited literature, keywords, and keyword clustering, and to generate corresponding graphs.

Results

A total of 2,668 articles were ultimately included in this study. The scholar with the highest number of publications is Professor Adam from the Department of Biomedical Information at Vanderbilt University in the United States. The top five countries contributing to this research are the United States, the United Kingdom, Germany, the Netherlands, and China. Based on an analysis of cited literature and keyword clustering, the research primarily focuses on predicting biochemical recurrence, cardiovascular disease, clinical guidelines, evidence-based computerized decision support systems, and intensive care units. The prominent topics in this field include artificial intelligence, natural language processing, venous thromboembolism, user-centered design, and emergency medicine.

Conclusion

Research on CDSS is demonstrating an upward trend and shows promising development prospects. Artificial intelligence, natural language processing, and user-centered design are the future trends.

Artificial intelligence tool development: what clinicians need to know?

Digital medicine and smart healthcare will not be realised without the cognizant participation of clinicians. Artificial intelligence (AI) today primarily involves computers or machines designed to simulate aspects of human intelligence using mathematically designed neural networks, although early AI systems relied on a variety of non-neural network techniques. With the increased complexity of the neural layers, deep machine learning (ML) can self-learn and augment many human tasks that require decision-making on the basis of multiple sources of data. Clinicians are important stakeholders in the use of AI and ML tools. The review questions are as follows: What is the typical process of AI tool development in the full cycle? What are the important concepts and technical aspects of each step? This review synthesises a targeted literature review and reports and summarises online structured materials to present a succinct explanation of the whole development process of AI tools. The development of AI tools in healthcare involves a series of cyclical processes: (1) identifying clinical problems suitable for AI solutions, (2) forming project teams or collaborating with experts, (3) organising and curating relevant data, (4) establishing robust physical and virtual infrastructure, and computer systems' architecture that support subsequent stages, (5) exploring AI neural networks on open access platforms before making a new decision, (6) validating AI/ML models, (7) registration, (8) clinical deployment and continuous performance monitoring and (9) improving the AI ecosystem ensures its adaptability to evolving clinical needs. A sound understanding of this would help clinicians appreciate the development of AI tools and engage in codesigning, evaluating and monitoring the tools. This would facilitate broader use and closer regulation of AI/ML tools in healthcare settings.

Human-centred AI for emergency cardiac care: Evaluating RAPIDx AI with PROLIFERATE_AI

BACKGROUND

Chest pain diagnosis in emergency care is hindered by overlapping cardiac and non-cardiac symptoms, causing diagnostic uncertainty. Artificial Intelligence, such as RAPIDx AI, aims to enhance accuracy through clinical and biochemical data integration, but its adoption relies on addressing usability, explainability, and seamless workflow integration without disrupting care.

OBJECTIVE

Evaluate RAPIDx AI's integration into clinical workflows, address usability barriers, and optimise its adoption in emergencies.

METHODS

The PROLIFERATE_AI framework was implemented across 12 EDs (July 2022-January 2024) with 39 participants: 15 experts co-designed a survey via Expert Knowledge Elicitation (EKE), applied to 24 ED clinicians to assess RAPIDx AI usability and adoption. Bayesian inference, using priors, estimated comprehension, emotional engagement, usage, and preference, while Monte Carlo simulations quantified uncertainty and variability, generating posterior means and 95% bootstrapped confidence intervals. Qualitative thematic analysis identified barriers and optimisation needs, with data triangulated through the PROLIFERATE_AI scoring system to rate RAPIDx AI's performance by user roles and demographics.

RESULTS

Registrars exhibited the highest comprehension (median: 0.466, 95 % CI: 0.41-0.51) and preference (median: 0.458, 95 % CI: 0.41-0.48), while residents/interns scored the lowest in comprehension (median: 0.198, 95 % CI: 0.17-0.26) and emotional engagement (median: 0.112, 95 % CI: 0.09-0.14). Registered nurses showed strong emotional engagement (median: 0.379, 95 % CI: 0.35-0.45). Novice users faced usability and workflow integration barriers, while experienced clinicians suggested automation and streamlined workflows. RAPIDx AI scored "Good Impact," excelling with trained users but requiring targeted refinements for novices.

CONCLUSION

RAPIDx AI enhances diagnostic accuracy and efficiency for experienced users, but usability challenges for novices highlight the need for targeted training and interface refinements. The PROLIFERATE_AI framework offers a robust methodology for evaluating and scaling AI solutions, addressing the evolving needs of sociotechnical systems.

Utility of disease probability scores to guide decision-making during screening for phaeochromocytoma and paraganglioma: a machine learning modelling cross sectional study

Background

Interpretation of plasma metanephrines and methoxytyramine to assess likelihood of phaeochromocytoma/paraganglioma (PPGL) during screening can be challenging. This study (study period: 2021-2023) introduces new methods to select machine-learning (ML) models and evaluate derived probability-scores to better interpret laboratory results.

Methods

ML models were trained and internally tested using data from 2046 patients with and without PPGL and according to several features: age, pre-test risk of PPGL, plasma metanephrines and methoxytyramine. External validation involved a second cohort of 1641 patients with and without PPGL. The study employed several processes to select and evaluate the best model: concordance of models with human intelligence; intra- and inter-laboratory variability in derived probability-scores; and comparison of scores of the selected model to predictions of ten clinical care specialists before and after provision of those scores.

Findings

External validation established equally excellent diagnostic performance for all five best ML models according to areas under ROC curves (0.988-0.994) and balanced accuracies (0.958-0.981). Probability-scores of models, however, varied widely and were poorly correlated. The additional selection processes indicated an artificial-network model as a superior and more robust model than others. Predictions of disease likelihood by specialists, according to six categories from disease highly unlikely to disease clear, varied widely for individual patients. Within each of the six predictive categories, median probability-scores of the artificial-network model were 70-, 175-, 59-, 15-, 3.5- and 1.7-fold higher (P < 0.0001) in patients with than without PPGL. This superiority of probability scores over variable predictions by specialists remained evident after specialists were tasked to modify their predictions according to those scores.

Interpretation

This study employed several novel processes to establish an ML model with probability-scores superior to predictions of disease likelihood by specialists. However, the negligible improvement in interpretations by specialists after provision of probability-scores indicates this alone is insufficient to improve decision-making.

Funding

Deutsche Forschungsgemeinschaft.

AI-PEDURO - Artificial intelligence in pediatric urology: Protocol for a living scoping review and online repository

BACKGROUND

Artificial intelligence (AI) and machine learning (ML) methods are increasingly being applied in pediatric urology across a growing number of settings, with more extensive databases and wider interest for use in clinical practice. More than 30 ML models have been published in the pediatric urology literature, but many lack items required by contemporary reporting frameworks to be high quality. For example, most studies lack multi-institution validation, validation over time, and validation within the clinical environment, resulting in a large discrepancy between the number of models developed versus the number of models deployed in a clinical setting, a phenomenon known as the AI chasm. Furthermore, pediatric urology is a unique subspecialty of urology with low frequency conditions and complex phenotypes where clinical management can rely on a lower quality of evidence.

OBJECTIVE

To establish the AI in PEDiatric UROlogy (AI-PEDURO) collaborative, which will carry out a living scoping review and create an online repository (www.aipeduro.com) for models in the field and facilitate an evidence synthesis of AI models in pediatric urology.

METHODS AND ANALYSIS

The scoping review will follow PRISMA-ScR guidelines. We will include ML models identified through standardized search methods of four databases, hand-search papers, and user-submitted models. Retrieved records will be included if they involve ML algorithms for prediction, classification, or risk stratification for pediatric urology conditions. The results will be tabulated and assessed for trends within the literature. Based on data availability, models will be divided into clinical disease sections (e.g. hydronephrosis, hypospadias, vesicoureteral reflux). A risk assessment will be performed using the APPRAISE-AI tool. The retrieved model cards (brief summary model characteristics in table form) will be uploaded to the online repository for open access to clinicians, patients, and data scientists, and will be linked to the Digital Object Identifier (DOI) for each article.

DISCUSSION AND CONCLUSION

We hope this living scoping review and online repository will offer a valuable reference for pediatric urologists to assess disease-specific ML models' scope, validity, and credibility to encourage opportunities for collaboration, external validation, clinical testing, and responsible deployment. In addition, the repository may aid in identifying areas in need of further research.

AI in radiological imaging of soft-tissue and bone tumours: a systematic review evaluating against CLAIM and FUTURE-AI guidelines

BACKGROUND

Soft-tissue and bone tumours (STBT) are rare, diagnostically challenging lesions with variable clinical behaviours and treatment approaches. This systematic review aims to provide an overview of Artificial Intelligence (AI) methods using radiological imaging for diagnosis and prognosis of these tumours, highlighting challenges in clinical translation, and evaluating study alignment with the Checklist for AI in Medical Imaging (CLAIM) and the FUTURE-AI international consensus guidelines for trustworthy and deployable AI to promote the clinical translation of AI methods.

METHODS

The systematic review identified literature from several bibliographic databases, covering papers published before 17/07/2024. Original research published in peer-reviewed journals, focused on radiology-based AI for diagnosis or prognosis of primary STBT was included. Exclusion criteria were animal, cadaveric, or laboratory studies, and non-English papers. Abstracts were screened by two of three independent reviewers to determine eligibility. Included papers were assessed against the two guidelines by one of three independent reviewers. The review protocol was registered with PROSPERO (CRD42023467970).

FINDINGS

The search identified 15,015 abstracts, from which 325 articles were included for evaluation. Most studies performed moderately on CLAIM, averaging a score of 28.9 ± 7.5 out of 53, but poorly on FUTURE-AI, averaging 5.1 ± 2.1 out of 30.

INTERPRETATION

Imaging-AI tools for STBT remain at the proof-of-concept stage, indicating significant room for improvement. Future efforts by AI developers should focus on design (e.g. defining unmet clinical need, intended clinical setting and how AI would be integrated in clinical workflow), development (e.g. building on previous work, training with data that reflect real-world usage, explainability), evaluation (e.g. ensuring biases are evaluated and addressed, evaluating AI against current best practices), and the awareness of data reproducibility and availability (making documented code and data publicly available). Following these recommendations could improve clinical translation of AI methods.

FUNDING

Hanarth Fonds, ICAI Lab, NIHR, EuCanImage.

Artificial Intelligence for Classification of Endoscopic Severity of Inflammatory Bowel Disease: A Systematic Review and Critical Appraisal

BACKGROUND

Endoscopic scoring indices for ulcerative colitis and Crohn's disease are subject to inter-endoscopist variability. There is increasing interest in the development of deep learning models to standardize endoscopic assessment of intestinal diseases. Here, we summarize and critically appraise the literature on artificial intelligence-assisted endoscopic characterization of inflammatory bowel disease severity.

METHODS

A systematic search of Ovid MEDLINE, EMBASE, Cochrane Central Register of Controlled Trials, and IEEE Xplore was performed to identify reports of AI systems used for endoscopic severity classification of IBD. Selected studies were critically appraised for methodological and reporting quality using APPRAISE-AI.

RESULTS

Thirty-one studies published between 2019 and 2024 were included. Of 31 studies, 28 studies examined endoscopic classification of ulcerative colitis and 3 examined Crohn's disease. Researchers sought to accomplish a wide range of classification tasks, including binary and multilevel classification, based on still images or full-length colonoscopy videos. Overall scores for study quality ranged from 41 (moderate quality) to 64 (high quality) out of 100, with 28 out of 31 studies within the moderate quality range. The highest-scoring domains were clinical relevance and reporting quality, while the lowest-scoring domains were robustness of results and reproducibility.

CONCLUSIONS

Multiple AI models have demonstrated the potential for clinical translation for ulcerative colitis. Research concerning the endoscopic severity assessment of Crohn's disease is limited and should be further explored. More rigorous external validation of AI models and increased transparency of data and codes are needed to improve the quality of AI studies.

Machine learning prediction of premature death from multimorbidity among people with inflammatory bowel disease: a population-based retrospective cohort study

BACKGROUND

Multimorbidity, the co-occurrence of 2 or more chronic conditions, is important in patients with inflammatory bowel disease (IBD) given its association with complex care plans, poor health outcomes, and excess mortality. Our objectives were to describe premature death (age < 75 yr) among people with IBD and to identify patterns between multimorbidity and premature death among decedents with IBD.

METHODS

Using the administrative health data of people with IBD who died between 2010 and 2020 in Ontario, Canada, we conducted a population-based, retrospective cohort study. We described the proportion of premature deaths among people with IBD. We developed statistical and machine learning models to predict premature death from the presence of 17 chronic conditions and the patients' age at diagnosis. We evaluated models using accuracy, positive predictive value, sensitivity, F1 scores, area under the receiver operating curve (AUC), calibration plots, and explainability plots.

RESULTS

All models showed strong performance (AUC 0.81-0.95). The best performing was the model that incorporated age at diagnosis for each chronic condition developed at or before age 60 years (AUC 0.95, 95% confidence interval 0.94-0.96). Salient features for predicting premature death were young ages of diagnosis for mood disorder, osteo-and other arthritis types, other mental health disorders, and hypertension, as well as male sex.

INTERPRETATION

By comparing results from multiple approaches modelling the impact of chronic conditions on premature death among people with IBD, we showed that conditions developed early in life (age ≤ 60 yr) and their age of onset were important for predicting their health trajectory. Clinically, our findings emphasize the need for models of care that ensure people with IBD have access to high-quality, multidisciplinary health care.

Artificial intelligence in clinical medicine: a state-of-the-art overview of systematic reviews with methodological recommendations for improved reporting

Medicine has become increasingly receptive to the use of artificial intelligence (AI). This overview of systematic reviews (SRs) aims to categorise current evidence about it and identify the current methodological state of the art in the field proposing a classification of AI model (CLASMOD-AI) to improve future reporting. PubMed/MEDLINE, Scopus, Cochrane library, EMBASE and Epistemonikos databases were screened by four blinded reviewers and all SRs that investigated AI tools in clinical medicine were included. 1923 articles were found, and of these, 360 articles were examined via the full-text and 161 SRs met the inclusion criteria. The search strategy, methodological, medical and risk of bias information were extracted. The CLASMOD-AI was based on input, model, data training, and performance metric of AI tools. A considerable increase in the number of SRs was observed in the last five years. The most covered field was oncology accounting for 13.9% of the SRs, with diagnosis as the predominant objective in 44.4% of the cases). The risk of bias was assessed in 49.1% of included SRs, yet only 39.2% of these used tools with specific items to assess AI metrics. This overview highlights the need for improved reporting on AI metrics, particularly regarding the training of AI models and dataset quality, as both are essential for a comprehensive quality assessment and for mitigating the risk of bias using specialized evaluation tools.

Implementation of artificial intelligence approaches in oncology clinical trials: A systematic review

INTRODUCTION

There is a growing interest in leveraging artificial intelligence (AI) technologies to enhance various aspects of clinical trials. The goal of this systematic review is to assess the impact of implementing AI approaches on different aspects of oncology clinical trials.

METHODS

Pertinent keywords were used to find relevant articles published in PubMed, Scopus, and Google Scholar databases, which described the clinical application of AI approaches. A quality evaluation utilizing a customized checklist specifically adapted was conducted. This study is registered with PROSPERO (CRD42024537153).

RESULTS

Out of the identified 2833 studies, 72 studies satisfied the inclusion criteria. Clinical Trial Enrollment & Eligibility were among the most commonly studied clinical trial aspects with 30 papers. The prediction of outcomes was covered in 25 studies of which 15 addressed the prediction of patients' survival and 10 addressed the prediction of drug outcomes. The trial design was studied in 10 articles. Three studies addressed each of the personalized treatments and decision-making, while one addressed data management. The results demonstrate using AI in cancer clinical trials has the potential to increase clinical trial enrollment, predict clinical outcomes, improve trial design, enhance personalized treatments, and increase concordance in decision-making. Additionally, automating some areas and tasks, clinical trials were made more efficient, and human error was minimized. Nevertheless, concerns and restrictions related to the application of AI in clinical studies are also noted.

CONCLUSION

AI tools have the potential to revolutionize the design, enrollment rate, and outcome prediction of oncology clinical trials.

The Heart of Transformation: Exploring Artificial Intelligence in Cardiovascular Disease

The application of artificial intelligence (AI) and machine learning (ML) in medicine and healthcare has been extensively explored across various areas. AI and ML can revolutionize cardiovascular disease management by significantly enhancing diagnostic accuracy, disease prediction, workflow optimization, and resource utilization. This review summarizes current advancements in AI and ML concerning cardiovascular disease, including their clinical investigation and use in primary cardiac imaging techniques, common cardiovascular disease categories, clinical research, patient care, and outcome prediction. We analyze and discuss commonly used AI and ML models, algorithms, and methodologies, highlighting their roles in improving clinical outcomes while addressing current limitations and future clinical applications. Furthermore, this review emphasizes the transformative potential of AI and ML in cardiovascular practice by improving clinical decision making, reducing human error, enhancing patient monitoring and support, and creating more efficient healthcare workflows for complex cardiovascular conditions.

A living scoping review and online repository of artificial intelligence models in pediatric urology: Results from the AI-PEDURO collaborative

INTRODUCTION

Artificial intelligence (AI) is increasingly being applied across pediatric urology. We provide a living scoping review and online repository developed by the AI in PEDiatric UROlogy (AI-PEDURO) collaborative that summarizes the current and emerging evidence on the AI models developed in pediatric urology.

MATERIAL AND METHODS

The protocol was published a priori, and Preferred Reporting Items for Systematic Review and Meta-analysis Scoping Review (PRISMA-ScR) guidelines were followed. We conducted a comprehensive search of four electronic databases and reviewed relevant data sources from inception until June 2024 to identify studies that have implemented AI for prediction, classification, or risk stratification for pediatric urology conditions. Model quality was assessed by the APPRAISE-AI tool.

RESULTS

Overall, 59 studies were included in this review from 1557 unique records. Of the 59 published studies, 44 studies (75 %) were published after 2019, with hydronephrosis and vesicoureteral reflux/urinary tract infection as the most common topics (17 studies, 28 % each). Studies originated from USA (22 studies, 37 %), Canada (10 studies, 17 %), China (8 studies, 14 %), and Turkey (7 studies, 12 %). Neural network (35 studies, 59 %), support-vector-machine (21 studies, 36 %), and tree-based models (19 studies, 32 %) were the most used machine learning algorithms, with 14 studies (24 %) providing useable repositories or applications. APPRAISE-AI assessed 12 studies (20 %) of studies as low quality, 39 studies (66 %) as moderate quality, and 8 studies (14 %) as high quality, with specific improvements noted in model robustness and reporting standards over time (p = 0.03). Findings were synthesized into an online repository (www.aipeduro.com).

DISCUSSION

There is an increasing pace of AI model development in pediatric urology. Model topics are broad, algorithm choice is diverse, and the overall quality of models are improving over time. While there is still a lack of clinical translation of the AI models in pediatric urology, the usage of online repositories and reporting frameworks can facilitate sharing, improvement, and clinical implementation of future models.

CONCLUSIONS

This living scoping review and online repository will highlight the current landscape of AI models in pediatric urology and facilitate their clinical translation and inform future research initiatives. From this work, we provide a summary of recommendations based on the current literature for future studies.

Guidelines and standard frameworks for artificial intelligence in medicine: a systematic review

Objectives

The continuous integration of artificial intelligence (AI) into clinical settings requires the development of up-to-date and robust guidelines and standard frameworks that consider the evolving challenges of AI implementation in medicine. This review evaluates the quality of these guideline and summarizes ethical frameworks, best practices, and recommendations.

Materials and Methods

The Appraisal of Guidelines, Research, and Evaluation II tool was used to assess the quality of guidelines based on 6 domains: scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence. The protocol of this review including the eligibility criteria, the search strategy data extraction sheet and methods, was published prior to the actual review with International Registered Report Identifier of DERR1-10.2196/47105.

Results

The initial search resulted in 4975 studies from 2 databases and 7 studies from manual search. Eleven articles were selected for data extraction based on the eligibility criteria. We found that while guidelines generally excel in scope, purpose, and editorial independence, there is significant variability in applicability and the rigor of guideline development. Well-established initiatives such as TRIPOD+AI, DECIDE-AI, SPIRIT-AI, and CONSORT-AI have shown high quality, particularly in terms of stakeholder involvement. However, applicability remains a prominent challenge among the guidelines. The result also showed that the reproducibility, ethical, and environmental aspects of AI in medicine still need attention from both medical and AI communities.

Discussion

Our work highlights the need for working toward the development of integrated and comprehensive reporting guidelines that adhere to the principles of Findability, Accessibility, Interoperability and Reusability. This alignment is essential for fostering a cultural shift toward transparency and open science, which are pivotal milestone for sustainable digital health research.

Conclusion

This review evaluates the current reporting guidelines, discussing their advantages as well as challenges and limitations.

Detection of periodontal bone loss and periodontitis from 2D dental radiographs via machine learning and deep learning: systematic review employing APPRAISE-AI and meta-analysis

OBJECTIVES

Periodontitis is a serious periodontal infection that damages the soft tissues and bone around teeth and is linked to systemic conditions. Accurate diagnosis and staging, complemented by radiographic evaluation, are vital. This systematic review (PROSPERO ID: CRD42023480552) explores artificial intelligence (AI) applications in assessing alveolar bone loss and periodontitis on dental panoramic and periapical radiographs.

METHODS

Five databases (Medline, Embase, Scopus, Web of Science, and Cochrane's Library) were searched from January 1990 to January 2024. Keywords related to "artificial intelligence", "Periodontal bone loss/Periodontitis", and "Dental radiographs" were used. Risk of bias and quality assessment of included papers were performed according to the APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support. Meta analysis was carried out via the "metaprop" command in R V3.6.1.

RESULTS

Thirty articles were included in the review, where 10 papers were eligible for meta-analysis. Based on quality scores from the APPRAISE-AI critical appraisal tool of the 30 papers, 1 (3.3%) were of very low quality (score < 40), 3 (10.0%) were of low quality (40 ≤ score < 50), 19 (63.3%) were of intermediate quality (50 ≤ score < 60), and 7 (23.3%) were of high quality (60 ≤ score < 80). No papers were of very high quality (score ≥ 80). Meta-analysis indicated that model performance was generally good, eg, sensitivity 87% (95% CI, 80%-93%), specificity 76% (95% CI, 69%-81%), and accuracy 84% (95% CI, 75%-91%).

CONCLUSION

Deep learning shows much promise in evaluating periodontal bone levels, although there was some variation in performance. AI studies can lack transparency and reporting standards could be improved. Our systematic review critically assesses the application of deep learning models in detecting alveolar bone loss on dental radiographs using the APPRAISE-AI tool, highlighting their efficacy and identifying areas for improvement, thus advancing the practice of clinical radiology.

Assessing the Reporting Quality of Machine Learning Algorithms in Head and Neck Oncology

OBJECTIVE

This study aimed to assess reporting quality of machine learning (ML) algorithms in the head and neck oncology literature using the TRIPOD-AI criteria.

DATA SOURCES

A comprehensive search was conducted using PubMed, Scopus, Embase, and Cochrane Database of Systematic Reviews, incorporating search terms related to "artificial intelligence," "machine learning," "deep learning," "neural network," and various head and neck neoplasms.

REVIEW METHODS

Two independent reviewers analyzed each published study for adherence to the 65-point TRIPOD-AI criteria. Items were classified as "Yes," "No," or "NA" for each publication. The proportion of studies satisfying each TRIPOD-AI criterion was calculated. Additionally, the evidence level for each study was evaluated independently by two reviewers using the Oxford Centre for Evidence-Based Medicine (OCEBM) Levels of Evidence. Discrepancies were reconciled through discussion until consensus was reached.

RESULTS

The study highlights the need for improvements in ML algorithm reporting in head and neck oncology. This includes more comprehensive descriptions of datasets, standardization of model performance reporting, and increased sharing of ML models, data, and code with the research community. Adoption of TRIPOD-AI is necessary for achieving standardized ML research reporting in head and neck oncology.

CONCLUSION

Current reporting of ML algorithms hinders clinical application, reproducibility, and understanding of the data used for model training. To overcome these limitations and improve patient and clinician trust, ML developers should provide open access to models, code, and source data, fostering iterative progress through community critique, thus enhancing model accuracy and mitigating biases.

LEVEL OF EVIDENCE

NA Laryngoscope, 135:687-694, 2025.

Deep Learning and Multidisciplinary Imaging in Pediatric Surgical Oncology: A Scoping Review

BACKGROUND

Medical images play an important role in diagnosis and treatment of pediatric solid tumors. The field of radiology, pathology, and other image-based diagnostics are getting increasingly important and advanced. This indicates a need for advanced image processing technology such as Deep Learning (DL).

AIM

Our review focused on the use of DL in multidisciplinary imaging in pediatric surgical oncology.

METHODS

A search was conducted within three databases (Pubmed, Embase, and Scopus), and 2056 articles were identified. Three separate screenings were performed for each identified subfield.

RESULTS

In total, we identified 36 articles, divided between radiology (n = 22), pathology (n = 9), and other image-based diagnostics (n = 5). Four types of tasks were identified in our review: classification, prediction, segmentation, and synthesis. General statements about the studies'' performance could not be made due to the inhomogeneity of the included studies. To implement DL in pediatric clinical practice, both technical validation and clinical validation are of uttermost importance.

CONCLUSION

In conclusion, our review provided an overview of all DL research in the field of pediatric surgical oncology. The more advanced status of DL in adults should be used as guide to move the field of DL in pediatric oncology further, to keep improving the outcomes of children with cancer.

Does machine learning improve prediction accuracy of the Endoscopic Third Ventriculostomy Success Score? A contemporary Hydrocephalus Clinical Research Network cohort study

PURPOSE

This Hydrocephalus Clinical Research Network (HCRN) study had two aims: (1) to compare the predictive performance of the original ETV Success Score (ETVSS) using logistic regression modeling with other newer machine learning models and (2) to assess whether inclusion of imaging variables improves prediction performance using machine learning models.

METHODS

We identified children undergoing first-time ETV for hydrocephalus that were enrolled prospectively at HCRN sites between 200 and 2020. The primary outcome was ETV success 6 months after index surgery. The cohort was randomly divided into training (70%) and testing (30%) datasets. The classic ETVSS variables were used for logistic regression and machine learning models. Predictive performance of each model was evaluated on the testing dataset using area under the receiver operating characteristic curve (AUROC).

RESULTS

There were 752 patients that underwent first time ETV, of which 185 patients (24.6%) experienced ETV failure within 6 months. For aim 1, using the classic ETVSS variables, machine learning models did not outperform logistic regression with AUROC 0.60 (95% CI: 0.52-0.69) for Naïve Bayes (highest machine learning model performance) and 0.68 (95% CI: 0.60-0.76) for logistic regression. After inclusion of imaging features (aim 2), machine learning model prediction improved but remained no better than the above logistic regression with the highest AUROC of 0.67 (95% CI: 0.59-0.75) attained using Naïve Bayes architecture compared to 0.68 (95% CI: 0.59-0.76) for logistic regression.

CONCLUSIONS

This contemporary multicenter observational cohort study demonstrated that machine learning modeling strategies did not improve performance of the ETVSS model over logistic regression.

Predictive models of sepsis-associated acute kidney injury based on machine learning: a scoping review

BACKGROUND

With the development of artificial intelligence, the application of machine learning to develop predictive models for sepsis-associated acute kidney injury has made potential breakthroughs in early identification, grading, diagnosis, and prognosis determination.

METHODS

Here, we conducted a systematic search of the PubMed, Cochrane Library, Embase (Ovid), Web of Science, and Scopus databases on April 28, 2023, and screened relevant literature. Then, we comprehensively extracted relevant data related to machine learning algorithms, predictors, and predicted objectives. We subsequently performed a critical evaluation of research quality, data aggregation, and analyses.

RESULTS

We screened 25 studies on predictive models for sepsis-associated acute kidney injury from a total of originally identified 2898 studies. The most commonly used machine learning algorithm is traditional logistic regression, followed by eXtreme gradient boosting. We categorized these predictive models into early identification models (60%), prognostic prediction models (32%), and subtype identification models (8%) according to their predictive purpose. The five most commonly used predictors were serum creatinine levels, lactate levels, age, blood urea nitrogen concentration, and diabetes mellitus. In addition, a single data source, insufficient assessment of clinical utility, lack of model bias assessment, and hyperparameter adjustment may be the main reasons for the low quality of the current research.

CONCLUSIONS

However, studies on the nondeath prognostic outcomes, the long-term clinical outcomes, and the subtype identification models are insufficient. Additionally, the poor quality of the research and the insufficient practicality of the model are problems that need to be addressed urgently.

Use of artificial intelligence algorithms to analyse systemic sclerosis-interstitial lung disease imaging features

The use of artificial intelligence (AI) in high-resolution computed tomography (HRCT) for diagnosing systemic sclerosis-associated interstitial lung disease (SSc-ILD) is relatively limited. This study aimed to analyse lung HRCT images of patients with systemic sclerosis with interstitial lung disease (SSc-ILD) using artificial intelligence (AI), conduct correlation analysis with clinical manifestations and prognosis, and explore the features and prognosis of SSc-ILD. Overall, 72 lung HRCT images and clinical data of 58 patients with SSC-ILD were collected. ILD lesion type, location, and volume on HRCT images were identified and evaluated using AI. The imaging characteristics of diffuse SSC (dSSc)-ILD and limited SSc-ILD (lSSc-ILD) were statistically analysed. Furthermore, the correlations between lesion type, clinical indicators, and prognosis were investigated. dSSc and lSSc were more prevalent in patients with a disease duration of < 1 and ≥ 5 years, respectively. SSc-ILD mainly comprises non-specific interstitial pneumonia (NSIP), usual interstitial pneumonia (UIP), and unclassifiable idiopathic interstitial pneumonia. HRCT reveals various lesion types in the early stages of the disease, with an increase in the number of lesion types as the disease progresses. Lesions appearing as grid, ground-glass, and nodular shadows were dispersed throughout both lungs, while those appearing as consolidation shadows and honeycomb were distributed across the lungs. Ground-glass opacity lesion type was absent on HRCT images of patients with SSc-ILD and pulmonary hypertension. This study showed that AI can efficiently analyse imaging characteristics of SSc-ILD, demonstrating its potential to learn from complex images with high generalisation ability.

Trial Factors Associated With Completion of Clinical Trials Evaluating AI: Retrospective Case-Control Study

BACKGROUND

Evaluation of artificial intelligence (AI) tools in clinical trials remains the gold standard for translation into clinical settings. However, design factors associated with successful trial completion and the common reasons for trial failure are unknown.

OBJECTIVE

This study aims to compare trial design factors of complete and incomplete clinical trials testing AI tools. We conducted a case-control study of complete (n=485) and incomplete (n=51) clinical trials that evaluated AI as an intervention of ClinicalTrials.gov.

METHODS

Trial design factors, including area of clinical application, intended use population, and intended role of AI, were extracted. Trials that did not evaluate AI as an intervention and active trials were excluded. The assessed trial design factors related to AI interventions included the domain of clinical application related to organ systems; intended use population for patients or health care providers; and the role of AI for different applications in patient-facing clinical workflows, such as diagnosis, screening, and treatment. In addition, we also assessed general trial design factors including study type, allocation, intervention model, masking, age, sex, funder, continent, length of time, sample size, number of enrollment sites, and study start year. The main outcome was the completion of the clinical trial. Odds ratio (OR) and 95% CI values were calculated for all trial design factors using propensity-matched, multivariable logistic regression.

RESULTS

We queried ClinicalTrials.gov on December 23, 2023, using AI keywords to identify complete and incomplete trials testing AI technologies as a primary intervention, yielding 485 complete and 51 incomplete trials for inclusion in this study. Our nested propensity-matched, case-control results suggest that trials conducted in Europe were significantly associated with trial completion when compared with North American trials (OR 2.85, 95% CI 1.14-7.10; P=.03), and the trial sample size was positively associated with trial completion (OR 1.00, 95% CI 1.00-1.00; P=.02).

CONCLUSIONS

Our case-control study is one of the first to identify trial design factors associated with completion of AI trials and catalog study-reported reasons for AI trial failure. We observed that trial design factors positively associated with trial completion include trials conducted in Europe and sample size. Given the promising clinical use of AI tools in health care, our results suggest that future translational research should prioritize addressing the design factors of AI clinical trials associated with trial incompletion and common reasons for study failure.

Artificial intelligence for predicting response to neoadjuvant chemotherapy for bladder cancer A comprehensive systematic review and meta-analysis

INTRODUCTION

Neoadjuvant cisplatin-based combination chemotherapy (NAC) followed by radical cystectomy is the standard of care for cisplatin-fit patients harboring muscle-invasive bladder cancer (MIBC). Prediction of response to NAC is essential for clinical decision-making regarding alternatives in case of non-response, and bladder-sparing in case of complete response. This research aimed to assess the performance of machine learning in predicting therapeutic response following NAC treatment in patients with MIBC.

METHODS

A systematic review adhering to the PRISMA guidelines was conducted until July 2023. The study integrated articles relating to artificial intelligence and NAC response in MIBC from various databases. The quality of articles was evaluated using the Quality Assessment Tool for Diagnostic Accuracy Studies 2 (QUADAS-2). A meta-analysis was subsequently performed on selected studies to determine the sensitivity and specificity of machine learning algorithms in predicting NAC response.

RESULTS

Of 655 articles identified, 12 studies comprising 1523 patients were included, and four studies were eligible for meta-analysis. The sensitivity and specificity of the studies were 0.62 (95% confidence interval [CI] 0.50-0.72) and 0.82 (95% CI 0.72-0.89), respectively, with a heterogeneity score (I2) of 38.5%. The machine learning algorithms used computed tomography, genetic, and anatomopathologic data as input and exhibited promising potential for predicting NAC response.

CONCLUSIONS

Machine-learning algorithms, especially those using computed tomography, genetic, and pathologic data, demonstrate significant potential for predicting NAC response in MIBC. Standardization of methodologic data analysis and response criteria are needed as validation studies.

Application of STREAM-URO and APPRAISE-AI reporting standards for artificial intelligence studies in pediatric urology: A case example with pediatric hydronephrosis

INTRODUCTION

Artificial intelligence (AI) and machine learning (ML) in pediatric urology is gaining increased popularity and credibility. However, the literature lacks standardization in reporting and there are areas for methodological improvement, which incurs difficulty in comparison between studies and may ultimately hurt clinical implementation of these models. The "STandardized REporting of Applications of Machine learning in UROlogy" (STREAM-URO) framework provides methodological instructions to improve transparent reporting in urology and APPRAISE-AI in a critical appraisal tool which provides quantitative measures for the quality of AI studies. The adoption of these will allow urologists and developers to ensure consistency in reporting, improve comparison, develop better models, and hopefully inspire clinical translation.

METHODS

In this article, we have applied STREAM-URO framework and APPRAISE-AI tool to the pediatric hydronephrosis literature. By doing this, we aim to describe best practices on ML reporting in urology with STREAM-URO and provide readers with a critical appraisal tool for ML quality with APPRAISE-AI. By applying these to the pediatric hydronephrosis literature, we provide some tutorial for other readers to employ these in developing and appraising ML models. We also present itemized recommendations for adequate reporting, and critically appraise the quality of ML in pediatric hydronephrosis insofar. We provide examples of strong reporting and highlight areas for improvement.

RESULTS

There were 8 ML models applied to pediatric hydronephrosis. The 26-item STREAM-URO framework is provided in Appendix A and 24-item APPRAISE-AI tool is provided in Appendix B. Across the 8 studies, the median compliance with STREAM-URO was 67 % and overall study quality was moderate. The highest scoring APPRAISE-AI domains in pediatric hydronephrosis were clinical relevance and reporting quality, while the worst were methodological conduct, robustness of results, and reproducibility.

CONCLUSIONS

If properly conducted and reported, ML has the potential to impact the care we provide to patients in pediatric urology. While AI is exciting, the paucity of strong evidence limits our ability to translate models to practice. The first step toward this goal is adequate reporting and ensuring high quality models, and STREAM-URO and APPRAISE-AI can facilitate better reporting and critical appraisal, respectively.

Predicting non-muscle invasive bladder cancer outcomes using artificial intelligence: a systematic review using APPRAISE-AI

Accurate prediction of recurrence and progression in non-muscle invasive bladder cancer (NMIBC) is essential to inform management and eligibility for clinical trials. Despite substantial interest in developing artificial intelligence (AI) applications in NMIBC, their clinical readiness remains unclear. This systematic review aimed to critically appraise AI studies predicting NMIBC outcomes, and to identify common methodological and reporting pitfalls. MEDLINE, EMBASE, Web of Science, and Scopus were searched from inception to February 5th, 2024 for AI studies predicting NMIBC recurrence or progression. APPRAISE-AI was used to assess methodological and reporting quality of these studies. Performance between AI and non-AI approaches included within these studies were compared. A total of 15 studies (five on recurrence, four on progression, and six on both) were included. All studies were retrospective, with a median follow-up of 71 months (IQR 32-93) and median cohort size of 125 (IQR 93-309). Most studies were low quality, with only one classified as high quality. While AI models generally outperformed non-AI approaches with respect to accuracy, c-index, sensitivity, and specificity, this margin of benefit varied with study quality (median absolute performance difference was 10 for low, 22 for moderate, and 4 for high quality studies). Common pitfalls included dataset limitations, heterogeneous outcome definitions, methodological flaws, suboptimal model evaluation, and reproducibility issues. Recommendations to address these challenges are proposed. These findings emphasise the need for collaborative efforts between urological and AI communities paired with rigorous methodologies to develop higher quality models, enabling AI to reach its potential in enhancing NMIBC care.

Question answering systems for health professionals at the point of care-a systematic review

OBJECTIVES

Question answering (QA) systems have the potential to improve the quality of clinical care by providing health professionals with the latest and most relevant evidence. However, QA systems have not been widely adopted. This systematic review aims to characterize current medical QA systems, assess their suitability for healthcare, and identify areas of improvement.

MATERIALS AND METHODS

We searched PubMed, IEEE Xplore, ACM Digital Library, ACL Anthology, and forward and backward citations on February 7, 2023. We included peer-reviewed journal and conference papers describing the design and evaluation of biomedical QA systems. Two reviewers screened titles, abstracts, and full-text articles. We conducted a narrative synthesis and risk of bias assessment for each study. We assessed the utility of biomedical QA systems.

RESULTS

We included 79 studies and identified themes, including question realism, answer reliability, answer utility, clinical specialism, systems, usability, and evaluation methods. Clinicians' questions used to train and evaluate QA systems were restricted to certain sources, types and complexity levels. No system communicated confidence levels in the answers or sources. Many studies suffered from high risks of bias and applicability concerns. Only 8 studies completely satisfied any criterion for clinical utility, and only 7 reported user evaluations. Most systems were built with limited input from clinicians.

DISCUSSION

While machine learning methods have led to increased accuracy, most studies imperfectly reflected real-world healthcare information needs. Key research priorities include developing more realistic healthcare QA datasets and considering the reliability of answer sources, rather than merely focusing on accuracy.

[Artificial intelligence in intensive care medicine]

The integration of artificial intelligence (AI) into intensive care medicine has made considerable progress in recent studies, particularly in the areas of predictive analytics, early detection of complications, and the development of decision support systems. The main challenges remain availability and quality of data, reduction of bias and the need for explainable results from algorithms and models. Methods to explain these systems are essential to increase trust, understanding, and ethical considerations among healthcare professionals and patients. Proper training of healthcare professionals in AI principles, terminology, ethical considerations, and practical application is crucial for the successful use of AI. Careful assessment of the impact of AI on patient autonomy and data protection is essential for its responsible use in intensive care medicine. A balance between ethical and practical considerations must be maintained to ensure patient-centered care while complying with data protection regulations. Synergistic collaboration between clinicians, AI engineers, and regulators is critical to realizing the full potential of AI in intensive care medicine and maximizing its positive impact on patient care. Future research and development efforts should focus on improving AI models for real-time predictions, increasing the accuracy and utility of AI-based closed-loop systems, and overcoming ethical, technical, and regulatory challenges, especially in generative AI systems.

Artificial intelligence in the risk prediction models of cardiovascular disease and development of an independent validation screening tool: a systematic review

BACKGROUND

A comprehensive overview of artificial intelligence (AI) for cardiovascular disease (CVD) prediction and a screening tool of AI models (AI-Ms) for independent external validation are lacking. This systematic review aims to identify, describe, and appraise AI-Ms of CVD prediction in the general and special populations and develop a new independent validation score (IVS) for AI-Ms replicability evaluation.

METHODS

PubMed, Web of Science, Embase, and IEEE library were searched up to July 2021. Data extraction and analysis were performed for the populations, distribution, predictors, algorithms, etc. The risk of bias was evaluated with the prediction risk of bias assessment tool (PROBAST). Subsequently, we designed IVS for model replicability evaluation with five steps in five items, including transparency of algorithms, performance of models, feasibility of reproduction, risk of reproduction, and clinical implication, respectively. The review is registered in PROSPERO (No. CRD42021271789).

RESULTS

In 20,887 screened references, 79 articles (82.5% in 2017-2021) were included, which contained 114 datasets (67 in Europe and North America, but 0 in Africa). We identified 486 AI-Ms, of which the majority were in development (n = 380), but none of them had undergone independent external validation. A total of 66 idiographic algorithms were found; however, 36.4% were used only once and only 39.4% over three times. A large number of different predictors (range 5-52,000, median 21) and large-span sample size (range 80-3,660,000, median 4466) were observed. All models were at high risk of bias according to PROBAST, primarily due to the incorrect use of statistical methods. IVS analysis confirmed only 10 models as "recommended"; however, 281 and 187 were "not recommended" and "warning," respectively.

CONCLUSION

AI has led the digital revolution in the field of CVD prediction, but is still in the early stage of development as the defects of research design, report, and evaluation systems. The IVS we developed may contribute to independent external validation and the development of this field.

Multitask learning to predict successful weaning in critically ill ventilated patients: A retrospective analysis of the MIMIC-IV database

Objective

Weaning is an essential issue in critical care. This study explores the efficacy of multitask learning models in predicting successful weaning in critically ill ventilated patients using the Medical Information Mart for Intensive Care (MIMIC) IV database.

Methods

We employed a multitask learning framework with a shared bottom network to facilitate common knowledge extraction across all tasks. We used the Shapley additive explanations (SHAP) plot and partial dependence plot (PDP) for model explainability. Furthermore, we conducted an error analysis to assess the strength and limitation of the model. Area under receiver operating characteristic curve (AUROC), calibration plot and decision curve analysis were used to determine the performance of the model.

Results

A total of 7758 critically ill patients were included in the analyses, and 78.5% of them were successfully weaned. Multitask learning combined with spontaneous breath trial achieved a higher performance to predict successful weaning compared with multitask learning combined with shock and mortality (area under receiver operating characteristic curve, AUROC, 0.820 ± 0.002 vs 0.817 ± 0.001, p < 0.001). We assessed the performance of the model using calibration and decision curve analyses and further interpreted the model through SHAP and PDP plots. The error analysis identified a relatively high error rate among those with low disease severities, including low mean airway pressure and high enteral feeding.

Conclusion

We demonstrated that multitask machine learning increased predictive accuracy for successful weaning through combining tasks with a high inter-task relationship. The model explainability and error analysis should enhance trust in the model.

Comprehensive Overview of Computational Modeling and Artificial Intelligence in Pediatric Neurosurgery

In this chapter, we give an overview of artificial intelligence tools and their use thus far in pediatric neurosurgery. We discuss different machine learning algorithms from a data-driven approach in order to guide clinicians and scientists as they apply them to real-world datasets. We provide examples of their successful application as well as evaluate limitations and pitfalls specific to clinical use. Finally, we explore future directions and exciting new opportunities to take advantage of these tools as they continue to advance and evolve.