Connecting Technological Innovation in Artificial Intelligence to Real-world Medical Practice through Rigorous Clinical Validation: What Peer-reviewed Medical Journals Could Do

Precision Psychiatry for Obsessive-Compulsive Disorder: Clinical Applications of Deep Learning Architectures

Obsessive-compulsive disorder (OCD) is a complex psychiatric condition characterized by significant heterogeneity in symptomatology and treatment response. Advances in neuroimaging, EEG, and other multimodal datasets have created opportunities to identify biomarkers and predict outcomes, yet traditional statistical methods often fall short in analyzing such high-dimensional data. Deep learning (DL) offers powerful tools for addressing these challenges by leveraging architectures capable of classification, prediction, and data generation. This brief review provides an overview of five key DL architectures-feedforward neural networks, convolutional neural networks, recurrent neural networks, generative adversarial networks, and transformers-and their applications in OCD research and clinical practice. We highlight how these models have been used to identify the neural predictors of treatment response, diagnose and classify OCD, and advance precision psychiatry. We conclude by discussing the clinical implementation of DL, summarizing its advances and promises in OCD, and underscoring key challenges for the field.

AI-ECG Supported Decision-Making for Coronary Angiography in Acute Chest Pain: The QCG-AID Study

This pilot study evaluates an artificial intelligence (AI)-assisted electrocardiography (ECG) analysis system, QCG, to enhance urgent coronary angiography (CAG) decision-making for acute chest pain in the emergency department (ED). We retrospectively analyzed 300 ED cases, categorized as non-coronary chest pain (Group 1), acute coronary syndrome (ACS) without occlusive coronary artery disease (CAD) (Group 2), and ACS with occlusive CAD (Group 3). Six clinicians made urgent CAG decision using a conventional approach (clinical data and ECG) and a QCG-assisted approach (including QCG scores). The QCG-assisted approach improved correct CAG decisions in Group 2 (36.0% vs. 45.3%, P = 0.003) and Group 3 (85.3% vs. 90.0%, P = 0.017), with minimal impact in Group 1 (92.7% vs. 95.0%, P = 0.125). Diagnostic accuracy for ACS improved from 77% to 81% with QCG assistance and reached 82% with QCG alone, supporting AI's potential to enhance urgent CAG decision-making for ED chest pain cases.

Clinical validation of an artificial intelligence algorithm for classifying tuberculosis and pulmonary findings in chest radiographs

Background

Chest X-ray (CXR) interpretation is critical in diagnosing various lung diseases. However, physicians, not specialists, are often the first ones to read them, frequently facing challenges in accurate interpretation. Artificial Intelligence (AI) algorithms could be of great help, but using real-world data is crucial to ensure their effectiveness in diverse healthcare settings. This study evaluates a deep learning algorithm designed for CXR interpretation, focusing on its utility for non-specialists in thoracic radiology physicians.

Purpose

To assess the performance of a Convolutional Neural Networks (CNNs)-based AI algorithm in interpreting CXRs and compare it with a team of physicians, including thoracic radiologists, who served as the gold-standard.

Methods

A retrospective study from January 2021 to July 2023 evaluated an algorithm with three independent models for Lung Abnormality, Radiological Findings, and Tuberculosis. The algorithm's performance was measured using accuracy, sensitivity, and specificity. Two groups of physicians validated the model: one with varying specialties and experience levels in interpreting chest radiographs (Group A) and another of board-certified thoracic radiologists (Group B). The study also assessed the agreement between the two groups on the algorithm's heatmap and its influence on their decisions.

Results

In the internal validation, the Lung Abnormality and Tuberculosis models achieved an AUC of 0.94, while the Radiological Findings model yielded a mean AUC of 0.84. During the external validation, utilizing the ground truth generated by board-certified thoracic radiologists, the algorithm achieved better sensitivity in 6 out of 11 classes than physicians with varying experience levels. Furthermore, Group A physicians demonstrated higher agreement with the algorithm in identifying markings in specific lung regions than Group B (37.56% Group A vs. 21.75% Group B). Additionally, physicians declared that the algorithm did not influence their decisions in 93% of the cases.

Conclusion

This retrospective clinical validation study assesses an AI algorithm's effectiveness in interpreting Chest X-rays (CXR). The results show the algorithm's performance is comparable to Group A physicians, using gold-standard analysis (Group B) as the reference. Notably, both Groups reported minimal influence of the algorithm on their decisions in most cases.

Differences in technical and clinical perspectives on AI validation in cancer imaging: mind the gap!

Good practices in artificial intelligence (AI) model validation are key for achieving trustworthy AI. Within the cancer imaging domain, attracting the attention of clinical and technical AI enthusiasts, this work discusses current gaps in AI validation strategies, examining existing practices that are common or variable across technical groups (TGs) and clinical groups (CGs). The work is based on a set of structured questions encompassing several AI validation topics, addressed to professionals working in AI for medical imaging. A total of 49 responses were obtained and analysed to identify trends and patterns. While TGs valued transparency and traceability the most, CGs pointed out the importance of explainability. Among the topics where TGs may benefit from further exposure are stability and robustness checks, and mitigation of fairness issues. On the other hand, CGs seemed more reluctant towards synthetic data for validation and would benefit from exposure to cross-validation techniques, or segmentation metrics. Topics emerging from the open questions were utility, capability, adoption and trustworthiness. These findings on current trends in AI validation strategies may guide the creation of guidelines necessary for training the next generation of professionals working with AI in healthcare and contribute to bridging any technical-clinical gap in AI validation. RELEVANCE STATEMENT: This study recognised current gaps in understanding and applying AI validation strategies in cancer imaging and helped promote trust and adoption for interdisciplinary teams of technical and clinical researchers. KEY POINTS: Clinical and technical researchers emphasise interpretability, external validation with diverse data, and bias awareness in AI validation for cancer imaging. In cancer imaging AI research, clinical researchers prioritise explainability, while technical researchers focus on transparency and traceability, and see potential in synthetic datasets. Researchers advocate for greater homogenisation of AI validation practices in cancer imaging.

Research Trends of Artificial Intelligence in Lung Cancer: A Combined Approach of Analysis With Latent Dirichlet Allocation and HJ-Biplot Statistical Methods

Lung cancer (LC) remains one of the leading causes of cancer-related mortality worldwide. With recent technological advances, artificial intelligence (AI) has begun to play a crucial role in improving diagnostic and treatment methods. It is crucial to understand how AI has integrated into LC research and to identify the main areas of focus. The aim of the study was to provide an updated insight into the role of AI in LC research, analyzing evolving topics, geographical distribution, and contributions to journals. The study explores research trends in AI applied to LC through a novel approach combining latent Dirichlet allocation (LDA) topic modeling with the HJ-Biplot statistical technique. A growing interest in AI applications in LC oncology was observed, reflected in a significant increase in publications, especially after 2017, coinciding with the availability of computing resources. Frontiers in Oncology leads in publishing AI-related LC research, reflecting rigorous investigation in the field. Geographically, China and the United States lead in contributions, attributed to significant investment in R&D and corporate sector involvement. LDA analysis highlights key research areas such as pulmonary nodule detection, patient prognosis prediction, and clinical decision support systems, demonstrating the impact of AI in improving LC outcomes. DL and AI emerge as prominent trends, focusing on radiomics and feature selection, promising better decision-making in LC care. The increase in AI-driven research covers various topics, including data analysis methodologies, tumor characterization, and predictive methods, indicating a concerted effort to advance LC research. HJ-Biplot visualization reveals thematic clustering, illustrating temporal and geographical associations and highlighting the influence of high-impact journals and countries with advanced research capabilities. This multivariate approach offers insights into global collaboration dynamics and specialization, emphasizing the evolving role of AI in LC research and diagnosis.

Data set terminology of deep learning in medicine: a historical review and recommendation

Medicine and deep learning-based artificial intelligence (AI) engineering represent two distinct fields each with decades of published history. The current rapid convergence of deep learning and medicine has led to significant advancements, yet it has also introduced ambiguity regarding data set terms common to both fields, potentially leading to miscommunication and methodological discrepancies. This narrative review aims to give historical context for these terms, accentuate the importance of clarity when these terms are used in medical deep learning contexts, and offer solutions to mitigate misunderstandings by readers from either field. Through an examination of historical documents, including articles, writing guidelines, and textbooks, this review traces the divergent evolution of terms for data sets and their impact. Initially, the discordant interpretations of the word 'validation' in medical and AI contexts are explored. We then show that in the medical field as well, terms traditionally used in the deep learning domain are becoming more common, with the data for creating models referred to as the 'training set', the data for tuning of parameters referred to as the 'validation (or tuning) set', and the data for the evaluation of models as the 'test set'. Additionally, the test sets used for model evaluation are classified into internal (random splitting, cross-validation, and leave-one-out) sets and external (temporal and geographic) sets. This review then identifies often misunderstood terms and proposes pragmatic solutions to mitigate terminological confusion in the field of deep learning in medicine. We support the accurate and standardized description of these data sets and the explicit definition of data set splitting terminologies in each publication. These are crucial methods for demonstrating the robustness and generalizability of deep learning applications in medicine. This review aspires to enhance the precision of communication, thereby fostering more effective and transparent research methodologies in this interdisciplinary field.

Artificial intelligence in musculoskeletal imaging: realistic clinical applications in the next decade

This article will provide a perspective review of the most extensively investigated deep learning (DL) applications for musculoskeletal disease detection that have the best potential to translate into routine clinical practice over the next decade. Deep learning methods for detecting fractures, estimating pediatric bone age, calculating bone measurements such as lower extremity alignment and Cobb angle, and grading osteoarthritis on radiographs have been shown to have high diagnostic performance with many of these applications now commercially available for use in clinical practice. Many studies have also documented the feasibility of using DL methods for detecting joint pathology and characterizing bone tumors on magnetic resonance imaging (MRI). However, musculoskeletal disease detection on MRI is difficult as it requires multi-task, multi-class detection of complex abnormalities on multiple image slices with different tissue contrasts. The generalizability of DL methods for musculoskeletal disease detection on MRI is also challenging due to fluctuations in image quality caused by the wide variety of scanners and pulse sequences used in routine MRI protocols. The diagnostic performance of current DL methods for musculoskeletal disease detection must be further evaluated in well-designed prospective studies using large image datasets acquired at different institutions with different imaging parameters and imaging hardware before they can be fully implemented in clinical practice. Future studies must also investigate the true clinical benefits of current DL methods and determine whether they could enhance quality, reduce error rates, improve workflow, and decrease radiologist fatigue and burnout with all of this weighed against the costs.

Image-based artificial intelligence for the prediction of pathological complete response to neoadjuvant chemoradiotherapy in patients with rectal cancer: a systematic review and meta-analysis

OBJECTIVE

Artificial intelligence (AI) holds enormous potential for noninvasively identifying patients with rectal cancer who could achieve pathological complete response (pCR) following neoadjuvant chemoradiotherapy (nCRT). We aimed to conduct a meta-analysis to summarize the diagnostic performance of image-based AI models for predicting pCR to nCRT in patients with rectal cancer.

METHODS

This study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. A literature search of PubMed, Embase, Cochrane Library, and Web of Science was performed from inception to July 29, 2023. Studies that developed or utilized AI models for predicting pCR to nCRT in rectal cancer from medical images were included. The Quality Assessment of Diagnostic Accuracy Studies-AI was used to appraise the methodological quality of the studies. The bivariate random-effects model was used to summarize the individual sensitivities, specificities, and areas-under-the-curve (AUCs). Subgroup and meta-regression analyses were conducted to identify potential sources of heterogeneity. Protocol for this study was registered with PROSPERO (CRD42022382374).

RESULTS

Thirty-four studies (9933 patients) were identified. Pooled estimates of sensitivity, specificity, and AUC of AI models for pCR prediction were 82% (95% CI: 76-87%), 84% (95% CI: 79-88%), and 90% (95% CI: 87-92%), respectively. Higher specificity was seen for the Asian population, low risk of bias, and deep-learning, compared with the non-Asian population, high risk of bias, and radiomics (all P < 0.05). Single-center had a higher sensitivity than multi-center (P = 0.001). The retrospective design had lower sensitivity (P = 0.012) but higher specificity (P < 0.001) than the prospective design. MRI showed higher sensitivity (P = 0.001) but lower specificity (P = 0.044) than non-MRI. The sensitivity and specificity of internal validation were higher than those of external validation (both P = 0.005).

CONCLUSIONS

Image-based AI models exhibited favorable performance for predicting pCR to nCRT in rectal cancer. However, further clinical trials are warranted to verify the findings.

Establishing a novel deep learning model for detecting peri-implantiti

BACKGROUND/PURPOSE

The diagnosis of peri-implantitis using periapical radiographs is crucial. Recently, artificial intelligence may apply in radiographic image analysis effectively. The aim of this study was to differentiate the degree of marginal bone loss of an implant, and also to classify the severity of peri-implantitis using a deep learning model.

MATERIALS AND METHODS

A dataset of 800 periapical radiographic images were divided into training (n = 600), validation (n = 100), and test (n = 100) datasets with implants used for deep learning. An object detection algorithm (YOLOv7) was used to identify peri-implantitis. The classification performance of this model was evaluated using metrics, including the specificity, precision, recall, and F1 score.

RESULTS

Considering the classification performance, the specificity was 100%, precision was 100%, recall was 94.44%, and F1 score was 97.10%.

CONCLUSION

Results of this study suggested that implants can be identified from periapical radiographic images using deep learning-based object detection. This identification system could help dentists and patients suffering from implant problems. However, more images of other implant systems are needed to increase the learning performance to apply this system in clinical practice.

Artificial intelligence performance in detecting lymphoma from medical imaging: a systematic review and meta-analysis

BACKGROUND

Accurate diagnosis and early treatment are essential in the fight against lymphatic cancer. The application of artificial intelligence (AI) in the field of medical imaging shows great potential, but the diagnostic accuracy of lymphoma is unclear. This study was done to systematically review and meta-analyse researches concerning the diagnostic performance of AI in detecting lymphoma using medical imaging for the first time.

METHODS

Searches were conducted in Medline, Embase, IEEE and Cochrane up to December 2023. Data extraction and assessment of the included study quality were independently conducted by two investigators. Studies that reported the diagnostic performance of an AI model/s for the early detection of lymphoma using medical imaging were included in the systemic review. We extracted the binary diagnostic accuracy data to obtain the outcomes of interest: sensitivity (SE), specificity (SP), and Area Under the Curve (AUC). The study was registered with the PROSPERO, CRD42022383386.

RESULTS

Thirty studies were included in the systematic review, sixteen of which were meta-analyzed with a pooled sensitivity of 87% (95%CI 83-91%), specificity of 94% (92-96%), and AUC of 97% (95-98%). Satisfactory diagnostic performance was observed in subgroup analyses based on algorithms types (machine learning versus deep learning, and whether transfer learning was applied), sample size (≤ 200 or >  200), clinicians versus AI models and geographical distribution of institutions (Asia versus non-Asia).

CONCLUSIONS

Even if possible overestimation and further studies with a better standards for application of AI algorithms in lymphoma detection are needed, we suggest the AI may be useful in lymphoma diagnosis.

AI-Generated Information for Vascular Patients: Assessing the Standard of Procedure-Specific Information Provided by the ChatGPT AI-Language Model

Introduction Ensuring access to high-quality information is paramount to facilitating informed surgical decision-making. The use of the internet to access health-related information is increasing, along with the growing prevalence of AI language models such as ChatGPT. We aim to assess the standard of AI-generated patient-facing information through a qualitative analysis of its readability and quality. Materials and methods We performed a retrospective qualitative analysis of information regarding three common vascular procedures: endovascular aortic repair (EVAR), endovenous laser ablation (EVLA), and femoro-popliteal bypass (FPBP). The ChatGPT responses were compared to patient information leaflets provided by the vascular charity, Circulation Foundation UK. Readability was assessed using four readability scores: the Flesch-Kincaid reading ease (FKRE) score, the Flesch-Kincaid grade level (FKGL), the Gunning fog score (GFS), and the simple measure of gobbledygook (SMOG) index. Quality was assessed using the DISCERN tool by two independent assessors. Results The mean FKRE score was 33.3, compared to 59.1 for the information provided by the Circulation Foundation (SD=14.5, p=0.025) indicating poor readability of AI-generated information. The FFKGL indicated that the expected grade of students likely to read and understand ChatGPT responses was consistently higher than compared to information leaflets at 12.7 vs. 9.4 (SD=1.9, p=0.002). Two metrics measure readability in terms of the number of years of education required to understand a piece of writing: the GFS and SMOG. Both scores indicated that AI-generated answers were less accessible. The GFS for ChatGPT-provided information was 16.7 years versus 12.8 years for the leaflets (SD=2.2, p=0.002) and the SMOG index scores were 12.2 and 9.4 years for ChatGPT and the patient information leaflets, respectively (SD=1.7, p=0.001). The DISCERN scores were consistently higher in human-generated patient information leaflets compared to AI-generated information across all procedures; the mean score for the information provided by ChatGPT was 50.3 vs. 56.0 for the Circulation Foundation information leaflets (SD=3.38, p<0.001). Conclusion We concluded that AI-generated information about vascular surgical procedures is currently poor in both the readability of text and the quality of information. Patients should be directed to reputable, human-generated information sources from trusted professional bodies to supplement direct education from the clinician during the pre-procedure consultation process.

Validation of Artificial Intelligence Containing Products Across the Regulated Healthcare Industries

PURPOSE

The introduction of artificial intelligence / machine learning (AI/ML) products to the regulated fields of pharmaceutical research and development (R&D) and drug manufacture, and medical devices (MD) and in vitro diagnostics (IVD), poses new regulatory problems: a lack of a common terminology and understanding leads to confusion, delays and product failures. Validation as a key step in product development, common to each of these sectors including computerized systems and AI/ML development, offers an opportune point of comparison for aligning people and processes for cross-sectoral product development.

METHODS

A comparative approach, built upon workshops and a subsequent written sequence of exchanges, is summarized in a look-up table suitable for mixed-teams work.

RESULTS

1. A bottom-up, definitions led, approach which leads to a distinction between broad vs narrow validation, and their relationship to regulatory regimes. 2. Common basis introduction to the primary methodologies for software validation, including AI-containing software validation. 3. Pharmaceutical drug development and MD/IVD-specific perspectives on compliant AI software development, as a basis for collaboration.

CONCLUSIONS

Alignment of the terms and methodologies used in validation of software products containing artificial intelligence/machine learning (AI/ML) components across the regulated industries of human health is a vital first step in streamlining processes and improving workflows.

Are current clinical studies on artificial intelligence-based medical devices comprehensive enough to support a full health technology assessment? A systematic review

INTRODUCTION

Artificial Intelligence-based Medical Devices (AI-based MDs) are experiencing exponential growth in healthcare. This study aimed to investigate whether current studies assessing AI contain the information required for health technology assessment (HTA) by HTA bodies.

METHODS

We conducted a systematic literature review based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses methodology to extract articles published between 2016 and 2021 related to the assessment of AI-based MDs. Data extraction focused on study characteristics, technology, algorithms, comparators, and results. AI quality assessment and HTA scores were calculated to evaluate whether the items present in the included studies were concordant with the HTA requirements. We performed a linear regression for the HTA and AI scores with the explanatory variables of the impact factor, publication date, and medical specialty. We conducted a univariate analysis of the HTA score and a multivariate analysis of the AI score with an alpha risk of 5 %.

RESULTS

Of 5578 retrieved records, 56 were included. The mean AI quality assessment score was 67 %; 32 % of articles had an AI quality score ≥ 70 %, 50 % had a score between 50 % and 70 %, and 18 % had a score under 50 %. The highest quality scores were observed for the study design (82 %) and optimisation (69 %) categories, whereas the scores were lowest in the clinical practice category (23 %). The mean HTA score was 52 % for all seven domains. 100 % of the studies assessed clinical effectiveness, whereas only 9 % evaluated safety, and 20 % evaluated economic issues. There was a statistically significant relationship between the impact factor and the HTA and AI scores (both p = 0.046).

DISCUSSION

Clinical studies on AI-based MDs have limitations and often lack adapted, robust, and complete evidence. High-quality datasets are also required because the output data can only be trusted if the inputs are reliable. The existing assessment frameworks are not specifically designed to assess AI-based MDs. From the perspective of regulatory authorities, we suggest that these frameworks should be adapted to assess the interpretability, explainability, cybersecurity, and safety of ongoing updates. From the perspective of HTA agencies, we highlight that transparency, professional and patient acceptance, ethical issues, and organizational changes are required for the implementation of these devices. Economic assessments of AI should rely on a robust methodology (business impact or health economic models) to provide decision-makers with more reliable evidence.

CONCLUSION

Currently, AI studies are insufficient to cover HTA prerequisites. HTA processes also need to be adapted because they do not consider the important specificities of AI-based MDs. Specific HTA workflows and accurate assessment tools should be designed to standardise evaluations, generate reliable evidence, and create confidence.

Construction and Analysis of Emotion Recognition and Psychotherapy System of College Students under Convolutional Neural Network and Interactive Technology

This study's aim is to effectively establish a psychological intervention and treatment system for college students and discover and correct their psychological problems encountered in a timely manner. From the perspectives of pedagogy and psychology, the college students majoring in physical education are selected as the research objects, and an interactive college student emotion recognition and psychological intervention system is established based on convolutional neural network (CNN). The system takes face recognition as the data source, adopts feature recognition algorithms to effectively classify the different students, and designs a psychological intervention platform based on interactive technology, and it is compared with existing systems and models to further verify its effectiveness. The results show that the deep learning CNN has better ability to recognize student emotions than backpropagation neural network (BPNN) and decision tree (DT) algorithm. The recognition accuracy (ACC) can be as high as 89.32%. Support vector machine (SVM) algorithm is adopted to classify the emotions, and the recognition ACC is increased by 20%. When the system's K value is 5 and d value is 8, the ACC of the model can reach 92.35%. The use of this system for psychotherapy has a significant effect, and 45% of the students are very satisfied with the human-computer interaction of the system. This study aims to guess the psychology of students through emotion recognition and reduce human participation based on the human-computer interaction, which can provide a new research idea for college psychotherapy. At present, the mental health problems of college students cannot be ignored; especially every year, there will be news reports of college students' extreme behaviors due to depression and other psychological problems. An interactive college student emotion recognition and psychological intervention system based on convolutional neural network (CNN) is established. This system uses face recognition as the basic support technology and uses feature recognition algorithms to effectively classify different students. An interaction technology-based psychological intervention platform is designed and compared with existing systems and models to further verify the effectiveness of the proposed system. The results show that deep learning has better student emotion recognition ability than backpropagation neural network (BPNN) and decision tree algorithm. The recognition accuracy is up to 89.32%. Support vector machine algorithm is employed to classify emotions, and the recognition acceptability rate increases by 20%. When K is 5 and d is 8, the acceptability rate of the model can reach 92.35%. The effect of this system in psychotherapy is remarkable, and 45% of students are very satisfied with the human-computer interaction of this system. This work aims to speculate students' psychology through emotion recognition, reduce people's participation via human-computer interaction, and provide a new research idea for university psychotherapy.

Diagnostic accuracy and potential covariates of artificial intelligence for diagnosing orthopedic fractures: a systematic literature review and meta-analysis

OBJECTIVES

To systematically quantify the diagnostic accuracy and identify potential covariates affecting the performance of artificial intelligence (AI) in diagnosing orthopedic fractures.

METHODS

PubMed, Embase, Web of Science, and Cochrane Library were systematically searched for studies on AI applications in diagnosing orthopedic fractures from inception to September 29, 2021. Pooled sensitivity and specificity and the area under the receiver operating characteristic curves (AUC) were obtained. This study was registered in the PROSPERO database prior to initiation (CRD 42021254618).

RESULTS

Thirty-nine were eligible for quantitative analysis. The overall pooled AUC, sensitivity, and specificity were 0.96 (95% CI 0.94-0.98), 90% (95% CI 87-92%), and 92% (95% CI 90-94%), respectively. In subgroup analyses, multicenter designed studies yielded higher sensitivity (92% vs. 88%) and specificity (94% vs. 91%) than single-center studies. AI demonstrated higher sensitivity with transfer learning (with vs. without: 92% vs. 87%) or data augmentation (with vs. without: 92% vs. 87%), compared to those without. Utilizing plain X-rays as input images for AI achieved results comparable to CT (AUC 0.96 vs. 0.96). Moreover, AI achieved comparable results to humans (AUC 0.97 vs. 0.97) and better results than non-expert human readers (AUC 0.98 vs. 0.96; sensitivity 95% vs. 88%).

CONCLUSIONS

AI demonstrated high accuracy in diagnosing orthopedic fractures from medical images. Larger-scale studies with higher design quality are needed to validate our findings.

KEY POINTS

• Multicenter study design, application of transfer learning, and data augmentation are closely related to improving the performance of artificial intelligence models in diagnosing orthopedic fractures. • Utilizing plain X-rays as input images for AI to diagnose fractures achieved results comparable to CT (AUC 0.96 vs. 0.96). • AI achieved comparable results to humans (AUC 0.97 vs. 0.97) but was superior to non-expert human readers (AUC 0.98 vs. 0.96, sensitivity 95% vs. 88%) in diagnosing fractures.

An artificial intelligence-based risk prediction model of myocardial infarction

BACKGROUND

Myocardial infarction can lead to malignant arrhythmia, heart failure, and sudden death. Clinical studies have shown that early identification of and timely intervention for acute MI can significantly reduce mortality. The traditional MI risk assessment models are subjective, and the data that go into them are difficult to obtain. Generally, the assessment is only conducted among high-risk patient groups.

OBJECTIVE

To construct an artificial intelligence-based risk prediction model of myocardial infarction (MI) for continuous and active monitoring of inpatients, especially those in noncardiovascular departments, and early warning of MI.

METHODS

The imbalanced data contain 59 features, which were constructed into a specific dataset through proportional division, upsampling, downsampling, easy ensemble, and w-easy ensemble. Then, the dataset was traversed using supervised machine learning, with recursive feature elimination as the top-layer algorithm and random forest, gradient boosting decision tree (GBDT), logistic regression, and support vector machine as the bottom-layer algorithms, to select the best model out of many through a variety of evaluation indices.

RESULTS

GBDT was the best bottom-layer algorithm, and downsampling was the best dataset construction method. In the validation set, the F1 score and accuracy of the 24-feature downsampling GBDT model were both 0.84. In the test set, the F1 score and accuracy of the 24-feature downsampling GBDT model were both 0.83, and the area under the curve was 0.91.

CONCLUSION

Compared with traditional models, artificial intelligence-based machine learning models have better accuracy and real-time performance and can reduce the occurrence of in-hospital MI from a data-driven perspective, thereby increasing the cure rate of patients and improving their prognosis.

Evaluating Prediction Models of Sleep Apnea From Smartphone-Recorded Sleep Breathing Sounds

Importance

Breathing sounds during sleep are an important characteristic feature of obstructive sleep apnea (OSA) and have been regarded as a potential biomarker. Breathing sounds during sleep can be easily recorded using a microphone, which is found in most smartphone devices. Therefore, it may be easy to implement an evaluation tool for prescreening purposes.

Objective

To evaluate OSA prediction models using smartphone-recorded sounds and identify optimal settings with regard to noise processing and sound feature selection.

Design, Setting, and Participants

A cross-sectional study was performed among patients who visited the sleep center of Seoul National University Bundang Hospital for snoring or sleep apnea from August 2015 to August 2019. Audio recordings during sleep were performed using a smartphone during routine, full-night, in-laboratory polysomnography. Using a random forest algorithm, binary classifications were separately conducted for 3 different threshold criteria according to an apnea hypopnea index (AHI) threshold of 5, 15, or 30 events/h. Four regression models were created according to noise reduction and feature selection from the input sound to predict actual AHI: (1) noise reduction without feature selection, (2) noise reduction with feature selection, (3) neither noise reduction nor feature selection, and (4) feature selection without noise reduction. Clinical and polysomnographic parameters that may have been associated with errors were assessed. Data were analyzed from September 2019 to September 2020.

Main Outcomes and Measures

Accuracy of OSA prediction models.

Results

A total of 423 patients (mean [SD] age, 48.1 [12.8] years; 356 [84.1%] male) were analyzed. Data were split into training (n = 256 [60.5%]) and test data sets (n = 167 [39.5%]). Accuracies were 88.2%, 82.3%, and 81.7%, and the areas under curve were 0.90, 0.89, and 0.90 for an AHI threshold of 5, 15, and 30 events/h, respectively. In the regression analysis, using recorded sounds that had not been denoised and had only selected attributes resulted in the highest correlation coefficient (r = 0.78; 95% CI, 0.69-0.88). The AHI (β = 0.33; 95% CI, 0.24-0.42) and sleep efficiency (β = -0.20; 95% CI, -0.35 to -0.05) were found to be associated with estimation error.

Conclusions and Relevance

In this cross-sectional study, recorded sleep breathing sounds using a smartphone were used to create reasonably accurate OSA prediction models. Future research should focus on real-life recordings using various smartphone devices.

Considerations for artificial intelligence clinical impact in oncologic imaging: an AI4HI position paper

To achieve clinical impact in daily oncological practice, emerging AI-based cancer imaging research needs to have clearly defined medical focus, AI methods, and outcomes to be estimated. AI-supported cancer imaging should predict major relevant clinical endpoints, aiming to extract associations and draw inferences in a fair, robust, and trustworthy way. AI-assisted solutions as medical devices, developed using multicenter heterogeneous datasets, should be targeted to have an impact on the clinical care pathway. When designing an AI-based research study in oncologic imaging, ensuring clinical impact in AI solutions requires careful consideration of key aspects, including target population selection, sample size definition, standards, and common data elements utilization, balanced dataset splitting, appropriate validation methodology, adequate ground truth, and careful selection of clinical endpoints. Endpoints may be pathology hallmarks, disease behavior, treatment response, or patient prognosis. Ensuring ethical, safety, and privacy considerations are also mandatory before clinical validation is performed. The Artificial Intelligence for Health Imaging (AI4HI) Clinical Working Group has discussed and present in this paper some indicative Machine Learning (ML) enabled decision-support solutions currently under research in the AI4HI projects, as well as the main considerations and requirements that AI solutions should have from a clinical perspective, which can be adopted into clinical practice. If effectively designed, implemented, and validated, cancer imaging AI-supported tools will have the potential to revolutionize the field of precision medicine in oncology.

Zozus M, Jiang X, Brimhall BB, Windham AK, Schmidt S, Visweswaran S, Ye Y, Goodrum H, Ling Y, Barapatre S, Becich MJ. Artificial intelligence in clinical and translational science: Successes, challenges and opportunities

Artificial intelligence (AI) is transforming many domains, including finance, agriculture, defense, and biomedicine. In this paper, we focus on the role of AI in clinical and translational research (CTR), including preclinical research (T1), clinical research (T2), clinical implementation (T3), and public (or population) health (T4). Given the rapid evolution of AI in CTR, we present three complementary perspectives: (1) scoping literature review, (2) survey, and (3) analysis of federally funded projects. For each CTR phase, we addressed challenges, successes, failures, and opportunities for AI. We surveyed Clinical and Translational Science Award (CTSA) hubs regarding AI projects at their institutions. Nineteen of 63 CTSA hubs (30%) responded to the survey. The most common funding source (48.5%) was the federal government. The most common translational phase was T2 (clinical research, 40.2%). Clinicians were the intended users in 44.6% of projects and researchers in 32.3% of projects. The most common computational approaches were supervised machine learning (38.6%) and deep learning (34.2%). The number of projects steadily increased from 2012 to 2020. Finally, we analyzed 2604 AI projects at CTSA hubs using the National Institutes of Health Research Portfolio Online Reporting Tools (RePORTER) database for 2011-2019. We mapped available abstracts to medical subject headings and found that nervous system (16.3%) and mental disorders (16.2) were the most common topics addressed. From a computational perspective, big data (32.3%) and deep learning (30.0%) were most common. This work represents a snapshot in time of the role of AI in the CTSA program.

A pilot study of a deep learning approach to detect marginal bone loss around implants

Background

Recently, there has been considerable innovation in artificial intelligence (AI) for healthcare. Convolutional neural networks (CNNs) show excellent object detection and classification performance. This study assessed the accuracy of an artificial intelligence (AI) application for the detection of marginal bone loss on periapical radiographs.

Methods

A Faster region-based convolutional neural network (R-CNN) was trained. Overall, 1670 periapical radiographic images were divided into training (n = 1370), validation (n = 150), and test (n = 150) datasets. The system was evaluated in terms of sensitivity, specificity, the mistake diagnostic rate, the omission diagnostic rate, and the positive predictive value. Kappa (κ) statistics were compared between the system and dental clinicians.

Results

Evaluation metrics of AI system is equal to resident dentist. The agreement between the AI system and expert is moderate to substantial (κ = 0.547 and 0.568 for bone loss sites and bone loss implants, respectively) for detecting marginal bone loss around dental implants.

Conclusions

This AI system based on Faster R-CNN analysis of periapical radiographs is a highly promising auxiliary diagnostic tool for peri-implant bone loss detection.

Current applications of artificial intelligence in vascular surgery

Basic foundations of artificial intelligence (AI) include analyzing large amounts of data, recognizing patterns, and predicting outcomes. At the core of AI are well-defined areas, such as machine learning, natural language processing, artificial neural networks, and computer vision. Although research and development of AI in health care is being conducted in many medical subspecialties, only a few applications have been implemented in clinical practice. This is true in vascular surgery, where applications are mostly in the translational research stage. These AI applications are being evaluated in the realms of vascular diagnostics, perioperative medicine, risk stratification, and outcome prediction, among others. Apart from the technical challenges of AI and research outcomes on safe and beneficial use in patient care, ethical issues and policy surrounding AI will present future challenges for its successful implementation. This review will give a brief overview and a basic understanding of AI and summarize the currently available and used clinical AI applications in vascular surgery.

A deep learning-based automatic analysis of cardiovascular borders on chest radiographs of valvular heart disease: development/external validation

OBJECTIVES

Cardiovascular border (CB) analysis is the primary method for detecting and quantifying the severity of cardiovascular disease using posterior-anterior chest radiographs (CXRs). This study aimed to develop and validate a deep learning-based automatic CXR CB analysis algorithm (CB_auto) for diagnosing and quantitatively evaluating valvular heart disease (VHD).

METHODS

We developed CB_auto using 816 normal and 798 VHD CXRs. For validation, 640 normal and 542 VHD CXRs from three different hospitals and 132 CXRs from a public dataset were assigned. The reliability of the CB parameters determined by CB_auto was evaluated. To evaluate the differences between parameters determined by CB_auto and manual CB drawing (CB_hand), the absolute percentage measurement error (APE) was calculated. Pearson correlation coefficients were calculated between CB_hand and echocardiographic measurements.

RESULTS

CB parameters determined by CB_auto yielded excellent reliability (intraclass correlation coefficient > 0.98). The 95% limits of agreement for the cardiothoracic ratio were 0.00 ± 0.04% without systemic bias. The differences between parameters determined by CB_auto and CB_hand as defined by the APE were < 10% for all parameters except for carinal angle and left atrial appendage. In the public dataset, all CB parameters were successfully drawn in 124 of 132 CXRs (93.9%). All CB parameters were significantly greater in VHD than in normal controls (all p < 0.05). All CB parameters showed significant correlations (p < 0.05) with echocardiographic measurements.

CONCLUSIONS

The CB_auto system empowered by deep learning algorithm provided highly reliable CB measurements that could be useful not only in daily clinical practice but also for research purposes.

KEY POINTS

• A deep learning-based automatic CB analysis algorithm for diagnosing and quantitatively evaluating VHD using posterior-anterior chest radiographs was developed and validated. • Our algorithm (CB_auto) yielded comparable reliability to manual CB drawing (CB_hand) in terms of various CB measurement variables, as confirmed by external validation with datasets from three different hospitals and a public dataset. • All CB parameters were significantly different between VHD and normal control measurements, and echocardiographic measurements were significantly correlated with CB parameters measured from normal control and VHD CXRs.

A framework for validating AI in precision medicine: considerations from the European ITFoC consortium

BACKGROUND

Artificial intelligence (AI) has the potential to transform our healthcare systems significantly. New AI technologies based on machine learning approaches should play a key role in clinical decision-making in the future. However, their implementation in health care settings remains limited, mostly due to a lack of robust validation procedures. There is a need to develop reliable assessment frameworks for the clinical validation of AI. We present here an approach for assessing AI for predicting treatment response in triple-negative breast cancer (TNBC), using real-world data and molecular -omics data from clinical data warehouses and biobanks.

METHODS

The European "ITFoC (Information Technology for the Future Of Cancer)" consortium designed a framework for the clinical validation of AI technologies for predicting treatment response in oncology.

RESULTS

This framework is based on seven key steps specifying: (1) the intended use of AI, (2) the target population, (3) the timing of AI evaluation, (4) the datasets used for evaluation, (5) the procedures used for ensuring data safety (including data quality, privacy and security), (6) the metrics used for measuring performance, and (7) the procedures used to ensure that the AI is explainable. This framework forms the basis of a validation platform that we are building for the "ITFoC Challenge". This community-wide competition will make it possible to assess and compare AI algorithms for predicting the response to TNBC treatments with external real-world datasets.

CONCLUSIONS

The predictive performance and safety of AI technologies must be assessed in a robust, unbiased and transparent manner before their implementation in healthcare settings. We believe that the consideration of the ITFoC consortium will contribute to the safe transfer and implementation of AI in clinical settings, in the context of precision oncology and personalized care.

Utilizing a responsive web portal for studying disc tracing agreement in retinal images

Glaucoma is a leading cause of blindness worldwide whose detection is based on multiple factors, including measuring the cup to disc ratio, retinal nerve fiber layer and visual field defects. Advances in image processing and machine learning have allowed the development of automated approached for segmenting objects from fundus images. However, to build a robust system, a reliable ground truth dataset is required for proper training and validation of the model. In this study, we investigate the level of agreement in properly detecting the retinal disc in fundus images using an online portal built for such purposes. Two Doctors of Optometry independently traced the discs for 159 fundus images obtained from publicly available datasets using a purpose-built online portal. Additionally, we studied the effectiveness of ellipse fitting in handling misalignments in tracing. We measured tracing precision, interobserver variability, and average boundary distance between the results provided by ophthalmologists, and optometrist tracing. We also studied whether ellipse fitting has a positive or negative impact on properly detecting disc boundaries. The overall agreement between the optometrists in terms of locating the disc region in these images was 0.87. However, we found that there was a fair agreement on the disc border with kappa = 0.21. Disagreements were mainly in fundus images obtained from glaucomatous patients. The resulting dataset was deemed to be an acceptable ground truth dataset for training a validation of models for automatic detection of objects in fundus images.

Deep Learning-Based Artificial Intelligence for Mammography

During the past decade, researchers have investigated the use of computer-aided mammography interpretation. With the application of deep learning technology, artificial intelligence (AI)-based algorithms for mammography have shown promising results in the quantitative assessment of parenchymal density, detection and diagnosis of breast cancer, and prediction of breast cancer risk, enabling more precise patient management. AI-based algorithms may also enhance the efficiency of the interpretation workflow by reducing both the workload and interpretation time. However, more in-depth investigation is required to conclusively prove the effectiveness of AI-based algorithms. This review article discusses how AI algorithms can be applied to mammography interpretation as well as the current challenges in its implementation in real-world practice.

Brain metastasis detection using machine learning: a systematic review and meta-analysis

BACKGROUND

Accurate detection of brain metastasis (BM) is important for cancer patients. We aimed to systematically review the performance and quality of machine-learning-based BM detection on MRI in the relevant literature.

METHODS

A systematic literature search was performed for relevant studies reported before April 27, 2020. We assessed the quality of the studies using modified tailored questionnaires of the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) criteria and the Checklist for Artificial Intelligence in Medical Imaging (CLAIM). Pooled detectability was calculated using an inverse-variance weighting model.

RESULTS

A total of 12 studies were included, which showed a clear transition from classical machine learning (cML) to deep learning (DL) after 2018. The studies on DL used a larger sample size than those on cML. The cML and DL groups also differed in the composition of the dataset, and technical details such as data augmentation. The pooled proportions of detectability of BM were 88.7% (95% CI, 84-93%) and 90.1% (95% CI, 84-95%) in the cML and DL groups, respectively. The false-positive rate per person was lower in the DL group than the cML group (10 vs 135, P < 0.001). In the patient selection domain of QUADAS-2, three studies (25%) were designated as high risk due to non-consecutive enrollment and arbitrary exclusion of nodules.

CONCLUSION

A comparable detectability of BM with a low false-positive rate per person was found in the DL group compared with the cML group. Improvements are required in terms of quality and study design.

Applications of machine learning and deep learning to thyroid imaging: where do we stand?

Ultrasonography (US) is the primary diagnostic tool used to assess the risk of malignancy and to inform decision-making regarding the use of fine-needle aspiration (FNA) and postFNA management in patients with thyroid nodules. However, since US image interpretation is operator-dependent and interobserver variability is moderate to substantial, unnecessary FNA and/or diagnostic surgery are common in practice. Artificial intelligence (AI)-based computeraided diagnosis (CAD) systems have been introduced to help with the accurate and consistent interpretation of US features, ultimately leading to a decrease in unnecessary FNA. This review provides a developmental overview of the AI-based CAD systems currently used for thyroid nodules and describes the future developmental directions of these systems for the personalized and optimized management of thyroid nodules.

Deep learning in spatiotemporal cardiac imaging: A review of methodologies and clinical usability

The use of different cardiac imaging modalities such as MRI, CT or ultrasound enables the visualization and interpretation of altered morphological structures and function of the heart. In recent years, there has been an increasing interest in AI and deep learning that take into account spatial and temporal information in medical image analysis. In particular, deep learning tools using temporal information in image processing have not yet found their way into daily clinical practice, despite its presumed high diagnostic and prognostic value. This review aims to synthesize the most relevant deep learning methods and discuss their clinical usability in dynamic cardiac imaging using for example the complete spatiotemporal image information of the heart cycle. Selected articles were categorized according to the following indicators: clinical applications, quality of datasets, preprocessing and annotation, learning methods and training strategy, and test performance. Clinical usability was evaluated based on these criteria by classifying the selected papers into (i) clinical level, (ii) robust candidate and (iii) proof of concept applications. Interestingly, not a single one of the reviewed papers was classified as a "clinical level" study. Almost 39% of the articles achieved a "robust candidate" and as many as 61% a "proof of concept" status. In summary, deep learning in spatiotemporal cardiac imaging is still strongly research-oriented and its implementation in clinical application still requires considerable efforts. Challenges that need to be addressed are the quality of datasets together with clinical verification and validation of the performance achieved by the used method.

Computer-Aided Diagnostic System for Thyroid Nodules on Ultrasonography: Diagnostic Performance Based on the Thyroid Imaging Reporting and Data System Classification and Dichotomous Outcomes

BACKGROUND AND PURPOSE

Artificial intelligence-based computer-aided diagnostic systems have been introduced for thyroid cancer diagnosis. Our aim was to compare the diagnostic performance of a commercially available computer-aided diagnostic system and radiologist-based assessment for the detection of thyroid cancer based on the Thyroid Imaging Reporting and Data Systems (TIRADS) and dichotomous outcomes.

MATERIALS AND METHODS

In total, 372 consecutive patients with 454 thyroid nodules were enrolled. The computer-aided diagnostic system was set up to render a possible diagnosis in 2 formats, the Korean Society of Thyroid Radiology (K)-TIRADS and the American Thyroid Association (ATA)-TIRADS-classifications, and dichotomous outcomes (possibly benign or possibly malignant).

RESULTS

The diagnostic sensitivity, specificity, positive predictive value, negative predictive value, and accuracy of the computer-aided diagnostic system for thyroid cancer were, respectively, 97.6%, 21.6%, 42.0%, 93.9%, and 49.6% for K-TIRADS; 94.6%, 29.6%, 43.9%, 90.4%, and 53.5% for ATA-TIRADS; and 81.4%, 81.9%, 72.3%, 88.3%, and 81.7% for dichotomous outcomes. The sensitivities of the computer-aided diagnostic system did not differ significantly from those of the radiologist (all P > .05); the specificities and accuracies were significantly lower than those of the radiologist (all P < .001). Unnecessary fine-needle aspiration rates were lower for the dichotomous outcome characterizations, particularly for those performed by the radiologist. The interobserver agreement for the description of K-TIRADS and ATA-TIRADS classifications was fair-to-moderate, but the dichotomous outcomes were in substantial agreement.

CONCLUSIONS

The diagnostic performance of the computer-aided diagnostic system varies in terms of TIRADS classification and dichotomous outcomes and relative to radiologist-based assessments. Clinicians should know about the strengths and weaknesses associated with the diagnosis of thyroid cancer using computer-aided diagnostic systems.

Apprehensions about Excessive Belief in Digital Therapeutics: Points of Concern Excluding Merits

Digital therapeutics (DTx), like drugs or medical devices, 1) must prove their effectiveness and safety through clinical trials; 2) are provided to patients through prescriptions from doctors; and 3) may require the approval of regulatory agencies, though this might not be mandatory. Although DTx will play an important role in the medical field in the near future, some merits of DTx have been exaggerated at this crucial juncture. In the medical field, where safety and effectiveness are important, merely reducing the development time and costs of DTx is not advantageous. The adverse effects of DTx are not yet well-known, and will be identified eventually, with the passage of time. DTx is beneficial for the collection and analysis of real-world data (RWD); however, they require new and distinct work to collect and analyze high-quality RWD. Naturally, whether this is possible must be independently ascertained through scientific methods. Depending on the type of disease, it is not recommended that DTx be prescribed, even if the patient rejects conventional treatment. Prescription of conventional pharmacotherapy is often necessary, and if the prescription of DTx is inadequate, the critical time for initial treatment may be missed. There is no basis for continuing DTx use by patients. Rather, the rate of continuity of DTx use is extremely low. While many conventional pharmacotherapies have undergone numerous verification and safety tests over a long time, barriers to the application of DTx in the medical field are lower than those for conventional pharmacotherapies. Considering these reasons, except for certain special cases, an approach to DTx is needed that complements the prescription of conventional pharmacotherapy by the medical staff. When DTx are prescribed by doctors who clearly know their advantages and disadvantages, the doctors' expertise may undergo further refinement, and the quality of medical care is expected to improve.

Inconsistency in the use of the term "validation" in studies reporting the performance of deep learning algorithms in providing diagnosis from medical imaging

BACKGROUND

The development of deep learning (DL) algorithms is a three-step process-training, tuning, and testing. Studies are inconsistent in the use of the term "validation", with some using it to refer to tuning and others testing, which hinders accurate delivery of information and may inadvertently exaggerate the performance of DL algorithms. We investigated the extent of inconsistency in usage of the term "validation" in studies on the accuracy of DL algorithms in providing diagnosis from medical imaging.

METHODS AND FINDINGS

We analyzed the full texts of research papers cited in two recent systematic reviews. The papers were categorized according to whether the term "validation" was used to refer to tuning alone, both tuning and testing, or testing alone. We analyzed whether paper characteristics (i.e., journal category, field of study, year of print publication, journal impact factor [JIF], and nature of test data) were associated with the usage of the terminology using multivariable logistic regression analysis with generalized estimating equations. Of 201 papers published in 125 journals, 118 (58.7%), 9 (4.5%), and 74 (36.8%) used the term to refer to tuning alone, both tuning and testing, and testing alone, respectively. A weak association was noted between higher JIF and using the term to refer to testing (i.e., testing alone or both tuning and testing) instead of tuning alone (vs. JIF <5; JIF 5 to 10: adjusted odds ratio 2.11, P = 0.042; JIF >10: adjusted odds ratio 2.41, P = 0.089). Journal category, field of study, year of print publication, and nature of test data were not significantly associated with the terminology usage.

CONCLUSIONS

Existing literature has a significant degree of inconsistency in using the term "validation" when referring to the steps in DL algorithm development. Efforts are needed to improve the accuracy and clarity in the terminology usage.

Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers

Objective

To evaluate the design characteristics of studies that evaluated the performance of artificial intelligence (AI) algorithms for the diagnostic analysis of medical images.

Materials and Methods

PubMed MEDLINE and Embase databases were searched to identify original research articles published between January 1, 2018 and August 17, 2018 that investigated the performance of AI algorithms that analyze medical images to provide diagnostic decisions. Eligible articles were evaluated to determine 1) whether the study used external validation rather than internal validation, and in case of external validation, whether the data for validation were collected, 2) with diagnostic cohort design instead of diagnostic case-control design, 3) from multiple institutions, and 4) in a prospective manner. These are fundamental methodologic features recommended for clinical validation of AI performance in real-world practice. The studies that fulfilled the above criteria were identified. We classified the publishing journals into medical vs. non-medical journal groups. Then, the results were compared between medical and non-medical journals.

Results

Of 516 eligible published studies, only 6% (31 studies) performed external validation. None of the 31 studies adopted all three design features: diagnostic cohort design, the inclusion of multiple institutions, and prospective data collection for external validation. No significant difference was found between medical and non-medical journals.

Conclusion

Nearly all of the studies published in the study period that evaluated the performance of AI algorithms for diagnostic analysis of medical images were designed as proof-of-concept technical feasibility studies and did not have the design features that are recommended for robust validation of the real-world clinical performance of AI algorithms.

Real-World Performance of Computer-Aided Diagnosis System for Thyroid Nodules Using Ultrasonography

This study evaluated the diagnostic performance of a commercially available computer-aided diagnosis (CAD) system (S-Detect 1 and S-Detect 2 for thyroid) for detecting thyroid cancers. Among 218 thyroid nodules in 106 patients, the sensitivity, specificity, positive predictive value, negative predictive value and accuracy of the CAD systems were 80.2%, 82.6%, 75.0%, 86.3% and 81.7%, respectively, for the S-Detect 1 and 81.4%, 68.2%, 62.5%, 84.9% and 73.4%, respectively, for the S-Detect 2. The inter-observer agreement between the CAD system and radiologist for the description of calcifications was fair (kappa = 0.336), while the final diagnosis and each ultrasonographic descriptor showed moderate to substantial agreement for the S-Detect 2. To conclude, the current CAD systems had limited specificity in the diagnosis of thyroid cancer. One of the main limitations of the S-Detect 2 was its inaccuracy in recognizing calcifications, which meant that differentiation had to be undertaken by the radiologist.

Age of Data in Contemporary Research Articles Published in Representative General Radiology Journals

Objective

To analyze and compare the age of data in contemporary research articles published in representative general radiology journals.

Materials and Methods

We searched for articles reporting original research studies analyzing patient data that were published in the print issues of the Korean Journal of Radiology (KJR), European Radiology (ER), and Radiology in 2017. Eligible articles were reviewed to extract data collection period (time from first patient recruitment to last patient follow-up) and age of data (time between data collection end and publication). The journals were compared in terms of the proportion of articles reporting the data collection period to the level of calendar month and regarding the age of data.

Results

There were 50, 492, and 254 eligible articles in KJR, ER, and Radiology, respectively. Of these, 44 (88%; 95% confidence interval [CI]: 75.8-94.8%), 359 (73%; 95% CI: 68.9-76.7%), and 211 (83.1%; 95% CI: 78-87.2%) articles, respectively, provided enough details of data collection period, revealing a significant difference between ER and Radiology (p = 0.002). The age of data was significantly greater in KJR (median age: 826 days; range: 299-2843 days) than in ER (median age: 570 days; range: 56-4742 days; p < 0.001) and Radiology (median age: 618; range: 75-4271 days; p < 0.001).

Conclusion

Korean Journal of Radiology did not fall behind ER or Radiology in reporting of data collection period, but showed a significantly greater age of data than ER and Radiology, suggesting that KJR should take measures to improve the timeliness of its data.

A Novel Fundus Image Reading Tool for Efficient Generation of a Multi-dimensional Categorical Image Database for Machine Learning Algorithm Training

BACKGROUND

We described a novel multi-step retinal fundus image reading system for providing high-quality large data for machine learning algorithms, and assessed the grader variability in the large-scale dataset generated with this system.

METHODS

A 5-step retinal fundus image reading tool was developed that rates image quality, presence of abnormality, findings with location information, diagnoses, and clinical significance. Each image was evaluated by 3 different graders. Agreements among graders for each decision were evaluated.

RESULTS

The 234,242 readings of 79,458 images were collected from 55 licensed ophthalmologists during 6 months. The 34,364 images were graded as abnormal by at-least one rater. Of these, all three raters agreed in 46.6% in abnormality, while 69.9% of the images were rated as abnormal by two or more raters. Agreement rate of at-least two raters on a certain finding was 26.7%-65.2%, and complete agreement rate of all-three raters was 5.7%-43.3%. As for diagnoses, agreement of at-least two raters was 35.6%-65.6%, and complete agreement rate was 11.0%-40.0%. Agreement of findings and diagnoses were higher when restricted to images with prior complete agreement on abnormality. Retinal/glaucoma specialists showed higher agreements on findings and diagnoses of their corresponding subspecialties.

CONCLUSION

This novel reading tool for retinal fundus images generated a large-scale dataset with high level of information, which can be utilized in future development of machine learning-based algorithms for automated identification of abnormal conditions and clinical decision supporting system. These results emphasize the importance of addressing grader variability in algorithm developments.