Siamese neural networks for continuous disease severity evaluation and change detection in medical imaging

Unlocking the potential of artificial intelligence in electrocardiogram biometrics: age-related changes, anomaly detection, and data authenticity in mobile health platforms

Aims

Mobile devices such as smartphones and watches can now record single-lead electrocardiograms (ECGs), making wearables a potential screening tool for cardiac and wellness monitoring outside of healthcare settings. Because friends and family often share their smart phones and devices, confirmation that a sample is from a given patient is important before it is added to the electronic health record.

Methods and results

We sought to determine whether the application of Siamese neural network would permit the diagnostic ECG sample to serve as both a medical test and biometric identifier. When using similarity scores to discriminate whether a pair of ECGs came from the same patient or different patients, inputs of single-lead and 12-lead medians produced an area under the curve of 0.94 and 0.97, respectively.

Conclusion

The similar performance of the single-lead and 12-lead configurations underscores the potential use of mobile devices to monitor cardiac health.

Image Analysis-Based Machine Learning for the Diagnosis of Retinopathy of Prematurity: A Meta-analysis and Systematic Review

TOPIC

To evaluate the performance of machine learning (ML) in the diagnosis of retinopathy of prematurity (ROP) and to assess whether it can be an effective automated diagnostic tool for clinical applications.

CLINICAL RELEVANCE

Early detection of ROP is crucial for preventing tractional retinal detachment and blindness in preterm infants, which has significant clinical relevance.

METHODS

Web of Science, PubMed, Embase, IEEE Xplore, and Cochrane Library were searched for published studies on image-based ML for diagnosis of ROP or classification of clinical subtypes from inception to October 1, 2022. The quality assessment tool for artificial intelligence-centered diagnostic test accuracy studies was used to determine the risk of bias (RoB) of the included original studies. A bivariate mixed effects model was used for quantitative analysis of the data, and the Deek's test was used for calculating publication bias. Quality of evidence was assessed using Grading of Recommendations Assessment, Development and Evaluation.

RESULTS

Twenty-two studies were included in the systematic review; 4 studies had high or unclear RoB. In the area of indicator test items, only 2 studies had high or unclear RoB because they did not establish predefined thresholds. In the area of reference standards, 3 studies had high or unclear RoB. Regarding applicability, only 1 study was considered to have high or unclear applicability in terms of patient selection. The sensitivity and specificity of image-based ML for the diagnosis of ROP were 93% (95% confidence interval [CI]: 0.90-0.94) and 95% (95% CI: 0.94-0.97), respectively. The area under the receiver operating characteristic curve (AUC) was 0.98 (95% CI: 0.97-0.99). For the classification of clinical subtypes of ROP, the sensitivity and specificity were 93% (95% CI: 0.89-0.96) and 93% (95% CI: 0.89-0.95), respectively, and the AUC was 0.97 (95% CI: 0.96-0.98). The classification results were highly similar to those of clinical experts (Spearman's R = 0.879).

CONCLUSIONS

Machine learning algorithms are no less accurate than human experts and hold considerable potential as automated diagnostic tools for ROP. However, given the quality and high heterogeneity of the available evidence, these algorithms should be considered as supplementary tools to assist clinicians in diagnosing ROP.

FINANCIAL DISCLOSURE(S)

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

CaMeL-Net: Centroid-aware metric learning for efficient multi-class cancer classification in pathology images

BACKGROUND AND OBJECTIVE

Cancer grading in pathology image analysis is a major task due to its importance in patient care, treatment, and management. The recent developments in artificial neural networks for computational pathology have demonstrated great potential to improve the accuracy and quality of cancer diagnosis. These improvements are generally ascribable to the advance in the architecture of the networks, often leading to increase in the computation and resources. In this work, we propose an efficient convolutional neural network that is designed to conduct multi-class cancer classification in an accurate and robust manner via metric learning.

METHODS

We propose a centroid-aware metric learning network for an improved cancer grading in pathology images. The proposed network utilizes centroids of different classes within the feature embedding space to optimize the relative distances between pathology images, which manifest the innate similarities/dissimilarities between them. For improved optimization, we introduce a new loss function and a training strategy that are tailored to the proposed network and metric learning.

RESULTS

We evaluated the proposed approach on multiple datasets of colorectal and gastric cancers. For the colorectal cancer, two different datasets were employed that were collected from different acquisition settings. the proposed method achieved an accuracy, F1-score, quadratic weighted kappa of 88.7%, 0.849, and 0.946 for the first dataset and 83.3%, 0.764, and 0.907 for the second dataset, respectively. For the gastric cancer, the proposed method obtained an accuracy of 85.9%, F1-score of 0.793, and quadratic weighted kappa of 0.939. We also found that the proposed method outperforms other competing models and is computationally efficient.

CONCLUSIONS

The experimental results demonstrate that the prediction results by the proposed network are both accurate and reliable. The proposed network not only outperformed other related methods in cancer classification but also achieved superior computational efficiency during training and inference. The future study will entail further development of the proposed method and the application of the method to other problems and domains.

Automated detection of genetic relatedness from fundus photographs using Siamese Neural Networks

Heritability of common eye diseases and ocular traits are relatively high. Here, we develop an automated algorithm to detect genetic relatedness from color fundus photographs (FPs). We estimated the degree of shared ancestry amongst individuals in the UK Biobank using KING software. A convolutional Siamese neural network-based algorithm was trained to output a measure of genetic relatedness using 7224 pairs (3612 related and 3612 unrelated) of FPs. The model achieved high performance for prediction of genetic relatedness; when computed Euclidean distances were used to determine probability of relatedness, the area under the receiver operating characteristic curve (AUROC) for identifying related FPs reached 0.926. We performed external validation of our model using FPs from the LIFE-Adult study and achieved an AUROC of 0.69. An occlusion map indicates that the optic nerve and its surrounding area may be the most predictive of genetic relatedness. We demonstrate that genetic relatedness can be captured from FP features. This approach may be used to uncover novel biomarkers for common ocular diseases.

The Objective Dementia Severity Scale Based on MRI with Contrastive Learning: A Whole Brain Neuroimaging Perspective

In the clinical treatment of Alzheimer's disease, one of the most important tasks is evaluating its severity for diagnosis and therapy. However, traditional testing methods are deficient, such as their susceptibility to subjective factors, incomplete evaluation, low accuracy, or insufficient granularity, resulting in unreliable evaluation scores. To address these issues, we propose an objective dementia severity scale based on MRI (ODSS-MRI) using contrastive learning to automatically evaluate the neurological function of patients. The approach utilizes a deep learning framework and a contrastive learning strategy to mine relevant information from structural magnetic resonance images to obtain the patient's neurological function level score. Given that the model is driven by the patient's whole brain imaging data, but without any possible biased manual intervention or instruction from the physician or patient, it provides a comprehensive and objective evaluation of the patient's neurological function. We conducted experiments on the Alzheimer's disease Neuroimaging Initiative (ADNI) dataset, and the results showed that the proposed ODSS-MRI was correlated with the stages of AD 88.55% better than all existing methods. This demonstrates its efficacy to describe the neurological function changes of patients during AD progression. It also outperformed traditional psychiatric rating scales in discriminating different stages of AD, which is indicative of its superiority for neurological function evaluation.

Modeling inter-individual differences in ambulatory-based multimodal signals via metric learning: a case study of personalized well-being estimation of healthcare workers

Introduction

Intelligent ambulatory tracking can assist in the automatic detection of psychological and emotional states relevant to the mental health changes of professionals with high-stakes job responsibilities, such as healthcare workers. However, well-known differences in the variability of ambulatory data across individuals challenge many existing automated approaches seeking to learn a generalizable means of well-being estimation. This paper proposes a novel metric learning technique that improves the accuracy and generalizability of automated well-being estimation by reducing inter-individual variability while preserving the variability pertaining to the behavioral construct.

Methods

The metric learning technique implemented in this paper entails learning a transformed multimodal feature space from pairwise similarity information between (dis)similar samples per participant via a Siamese neural network. Improved accuracy via personalization is further achieved by considering the trait characteristics of each individual as additional input to the metric learning models, as well as individual trait base cluster criteria to group participants followed by training a metric learning model for each group.

Results

The outcomes of the proposed models demonstrate significant improvement over the other inter-individual variability reduction and deep neural baseline methods for stress, anxiety, positive affect, and negative affect.

Discussion

This study lays the foundation for accurate estimation of psychological and emotional states in realistic and ambulatory environments leading to early diagnosis of mental health changes and enabling just-in-time adaptive interventions.

Chirality Analysis of Complex Microparticles using Deep Learning on Realistic Sets of Microscopy Images

Nanoscale chirality is an actively growing research field spurred by the giant chiroptical activity, enantioselective biological activity, and asymmetric catalytic activity of chiral nanostructures. Compared to chiral molecules, the handedness of chiral nano- and microstructures can be directly established via electron microscopy, which can be utilized for the automatic analysis of chiral nanostructures and prediction of their properties. However, chirality in complex materials may have multiple geometric forms and scales. Computational identification of chirality from electron microscopy images rather than optical measurements is convenient but is fundamentally challenging, too, because (1) image features differentiating left- and right-handed particles can be ambiguous and (2) three-dimensional structure essential for chirality is 'flattened' into two-dimensional projections. Here, we show that deep learning algorithms can identify twisted bowtie-shaped microparticles with nearly 100% accuracy and classify them as left- and right-handed with as high as 99% accuracy. Importantly, such accuracy was achieved with as few as 30 original electron microscopy images of bowties. Furthermore, after training on bowtie particles with complex nanostructured features, the model can recognize other chiral shapes with different geometries without retraining for their specific chiral geometry with 93% accuracy, indicating the true learning abilities of the employed neural networks. These findings indicate that our algorithm trained on a practically feasible set of experimental data enables automated analysis of microscopy data for the accelerated discovery of chiral particles and their complex systems for multiple applications.

Highly Performing Automatic Detection of Structural Chromosomal Abnormalities Using Siamese Architecture

The detection of structural chromosomal abnormalities (SCA) is crucial for diagnosis, prognosis and management of many genetic diseases and cancers. This detection, done by highly qualified medical experts, is tedious and time-consuming. We propose a highly performing and intelligent method to assist cytogeneticists to screen for SCA. Each chromosome is present in two copies that make up a pair of chromosomes. Usually, SCA are present in only one copy of the pair. Convolutional neural networks (CNN) with Siamese architecture are particularly relevant for evaluating similarities between two images, which is why we used this method to detect abnormalities between both chromosomes of a given pair. As a proof-of-concept, we first focused on a deletion occurring on chromosome 5 (del(5q)) observed in hematological malignancies. Using our dataset, we conducted several experiments without and with data augmentation on seven popular CNN models. Overall, performances obtained were very relevant for detecting deletions, particularly with Xception and InceptionResNetV2 models achieving 97.50% and 97.01% of F₁-score, respectively. We additionally demonstrated that these models successfully recognized another SCA, inversion inv(3), which is one of the most difficult SCA to detect. The performance improved when the training was applied on inversion inv(3) dataset, achieving 94.82% of F₁-score. The technique that we propose in this paper is the first highly performing method based on Siamese architecture that allows the detection of SCA. Our code is publicly available at: https://github.com/MEABECHAR/ChromosomeSiameseAD.

Few-shot learning using explainable Siamese twin network for the automated classification of blood cells

Automated classification of blood cells from microscopic images is an interesting research area owing to advancements of efficient neural network models. The existing deep learning methods rely on large data for network training and generating such large data could be time-consuming. Further, explainability is required via class activation mapping for better understanding of the model predictions. Therefore, we developed a Siamese twin network (STN) model based on contrastive learning that trains on relatively few images for the classification of healthy peripheral blood cells using EfficientNet-B3 as the base model. Hence, in this study, a total of 17,092 publicly accessible cell histology images were analyzed from which 6% were used for STN training, 6% for few-shot validation, and the rest 88% for few-shot testing. The proposed architecture demonstrates percent accuracies of 97.00, 98.78, 94.59, 95.70, 98.86, 97.09, 99.71, and 96.30 during 8-way 5-shot testing for the classification of basophils, eosinophils, immature granulocytes, erythroblasts, lymphocytes, monocytes, platelets, and neutrophils, respectively. Further, we propose a novel class activation mapping scheme that highlights the important regions in the test image for the STN model interpretability. Overall, the proposed framework could be used for a fully automated self-exploratory classification of healthy peripheral blood cells. The whole proposed framework demonstrates the Siamese twin network training and 8-way k-shot testing. The values indicate the amount of dissimilarity.

Adversarial Evolving Neural Network for Longitudinal Knee Osteoarthritis Prediction

Knee osteoarthritis (KOA) as a disabling joint disease has doubled in prevalence since the mid-20th century. Early diagnosis for the longitudinal KOA grades has been increasingly important for effective monitoring and intervention. Although recent studies have achieved promising performance for baseline KOA grading, longitudinal KOA grading has been seldom studied and the KOA domain knowledge has not been well explored yet. In this paper, a novel deep learning architecture, namely adversarial evolving neural network (A-ENN), is proposed for longitudinal grading of KOA severity. As the disease progresses from mild to severe level, ENN involves the progression patterns for accurately characterizing the disease by comparing an input image it to the template images of different KL grades using convolution and deconvolution computations. In addition, an adversarial training scheme with a discriminator is developed to obtain the evolution traces. Thus, the evolution traces as fine-grained domain knowledge are further fused with the general convolutional image representations for longitudinal grading. Note that ENN can be applied to other learning tasks together with existing deep architectures, in which the responses characterize progressive representations. Comprehensive experiments on the Osteoarthritis Initiative (OAI) dataset were conducted to evaluate the proposed method. An overall accuracy was achieved as 62.7%, with the baseline, 12-month, 24-month, 36-month, and 48-month accuracy as 64.6%, 63.9%, 63.2%, 61.8% and 60.2%, respectively.

Multi-population generalizability of a deep learning-based chest radiograph severity score for COVID-19

To tune and test the generalizability of a deep learning-based model for assessment of COVID-19 lung disease severity on chest radiographs (CXRs) from different patient populations. A published convolutional Siamese neural network-based model previously trained on hospitalized patients with COVID-19 was tuned using 250 outpatient CXRs. This model produces a quantitative measure of COVID-19 lung disease severity (pulmonary x-ray severity (PXS) score). The model was evaluated on CXRs from 4 test sets, including 3 from the United States (patients hospitalized at an academic medical center (N = 154), patients hospitalized at a community hospital (N = 113), and outpatients (N = 108)) and 1 from Brazil (patients at an academic medical center emergency department (N = 303)). Radiologists from both countries independently assigned reference standard CXR severity scores, which were correlated with the PXS scores as a measure of model performance (Pearson R). The Uniform Manifold Approximation and Projection (UMAP) technique was used to visualize the neural network results. Tuning the deep learning model with outpatient data showed high model performance in 2 United States hospitalized patient datasets (R = 0.88 and R = 0.90, compared to baseline R = 0.86). Model performance was similar, though slightly lower, when tested on the United States outpatient and Brazil emergency department datasets (R = 0.86 and R = 0.85, respectively). UMAP showed that the model learned disease severity information that generalized across test sets. A deep learning model that extracts a COVID-19 severity score on CXRs showed generalizable performance across multiple populations from 2 continents, including outpatients and hospitalized patients.

External COVID-19 Deep Learning Model Validation on ACR AI-LAB: It's a Brave New World

PURPOSE

Deploying external artificial intelligence (AI) models locally can be logistically challenging. We aimed to use the ACR AI-LAB software platform for local testing of a chest radiograph (CXR) algorithm for COVID-19 lung disease severity assessment.

METHODS

An externally developed deep learning model for COVID-19 radiographic lung disease severity assessment was loaded into the AI-LAB platform at an independent academic medical center, which was separate from the institution in which the model was trained. The data set consisted of CXR images from 141 patients with reverse transcription-polymerase chain reaction-confirmed COVID-19, which were routed to AI-LAB for model inference. The model calculated a Pulmonary X-ray Severity (PXS) score for each image. This score was correlated with the average of a radiologist-based assessment of severity, the modified Radiographic Assessment of Lung Edema score, independently interpreted by three radiologists. The associations between the PXS score and patient admission and intubation or death were assessed.

RESULTS

The PXS score deployed in AI-LAB correlated with the radiologist-determined modified Radiographic Assessment of Lung Edema score (r = 0.80). PXS score was significantly higher in patients who were admitted (4.0 versus 1.3, P < .001) or intubated or died within 3 days (5.5 versus 3.3, P = .001).

CONCLUSIONS

AI-LAB was successfully used to test an external COVID-19 CXR AI algorithm on local data with relative ease, showing generalizability of the PXS score model. For AI models to scale and be clinically useful, software tools that facilitate the local testing process, like the freely available AI-LAB, will be important to cross the AI implementation gap in health care systems.

Bilateral Analysis Boosts the Performance of Mammography-based Deep Learning Models in Breast Cancer Risk Prediction

Breast cancer is one of the leading causes of death among women. Early prediction of breast cancer can significantly improve the survival rates. Breast density was proven as a reliable risk factor. Deep learning models can learn subtle cues in the mammogram images. CNN models were recently shown to improve the risk discrimination in full-field mammograms. This study aims to improve risk prediction models using bilateral analysis. Bilateral analysis is the process of comparing two breasts to verify presence of anomalies. We developed a Siamese neural network to leverage the bilateral information and asymmetries between the two mammograms of the same patient. We tested our model on 271 patients and compared the results of our Siamese model against the traditional unilateral CNN model. Our results showed AUCs of 0.75 and 0.70 respectively (p = 0.0056). The Siamese model also exhibits higher sensitivity, specificity, precision, and false positive rate with values of 0.68, 0.69, 0.71, 0.31 respectively. While the CNN values were 0.61, 0.66, 0.67, 0.34 respectively. We merged both models by two techniques using pre-trained weights and weighted voting ensemble. The merging technique boosted the AUC to 0.78. The results suggest that bilateral analysis can significantly improve the risk discrimination.

Feature fusion siamese network for breast cancer detection comparing current and prior mammograms

PURPOSE

Automatic detection of very small and non-mass abnormalities from mammogram images has remained challenging. In clinical practice for each patient, radiologists commonly not only screen the mammogram images obtained during the examination, but also compare them with previous mammogram images to make a clinical decision. To design an AI system to mimic radiologists for better cancer detection, in this work we proposed an end-to-end enhanced Siamese convolutional neural network to detect breast cancer using previous year and current year mammogram images.

METHODS

The proposed Siamese based network uses high resolution mammogram images and fuses features of pairs of previous year and current year mammogram images to predict cancer probabilities. The proposed approach is developed based on the concept of one-shot learning that learns the abnormal differences between current and prior images instead of abnormal objects, and as a result can perform better with small sample size data sets. We developed two variants of the proposed network. In the first model, to fuse the features of current and previous images, we designed an enhanced distance learning network that considers not only the overall distance, but also the pixel-wise distances between the features. In the other model, we concatenated the features of current and previous images to fuse them.

RESULTS

We compared the performance of the proposed models with those of some baseline models that use current images only (ResNet and VGG) and also use current and prior images (LSTM and vanilla Siamese) in terms of accuracy, sensitivity, precision, F1 score and AUC. Results show that the proposed models outperform the baseline models and the proposed model with the distance learning network performs the best (accuracy: 0.92, sensitivity: 0.93, precision: 0.91, specificity: 0.91, F1: 0.92 and AUC: 0.95).

CONCLUSIONS

Integrating prior mammogram images improves automatic cancer classification, specially for very small and non-mass abnormalities. For classification models that integrate current and prior mammogram images, using an enhanced and effective distance learning network can advance the performance of the models. This article is protected by copyright. All rights reserved.

AI musculoskeletal clinical applications: how can AI increase my day-to-day efficiency?

Artificial intelligence (AI) is expected to bring greater efficiency in radiology by performing tasks that would otherwise require human intelligence, also at a much faster rate than human performance. In recent years, milestone deep learning models with unprecedented low error rates and high computational efficiency have shown remarkable performance for lesion detection, classification, and segmentation tasks. However, the growing field of AI has significant implications for radiology that are not limited to visual tasks. These are essential applications for optimizing imaging workflows and improving noninterpretive tasks. This article offers an overview of the recent literature on AI, focusing on the musculoskeletal imaging chain, including initial patient scheduling, optimized protocoling, magnetic resonance imaging reconstruction, image enhancement, medical image-to-image translation, and AI-aided image interpretation. The substantial developments of advanced algorithms, the emergence of massive quantities of medical data, and the interest of researchers and clinicians reveal the potential for the growing applications of AI to augment the day-to-day efficiency of musculoskeletal radiologists.

Artificial intelligence applied to musculoskeletal oncology: a systematic review

Developments in artificial intelligence have the potential to improve the care of patients with musculoskeletal tumors. We performed a systematic review of the published scientific literature to identify the current state of the art of artificial intelligence applied to musculoskeletal oncology, including both primary and metastatic tumors, and across the radiology, nuclear medicine, pathology, clinical research, and molecular biology literature. Through this search, we identified 252 primary research articles, of which 58 used deep learning and 194 used other machine learning techniques. Articles involving deep learning have mostly involved bone scintigraphy, histopathology, and radiologic imaging. Articles involving other machine learning techniques have mostly involved transcriptomic analyses, radiomics, and clinical outcome prediction models using medical records. These articles predominantly present proof-of-concept work, other than the automated bone scan index for bone metastasis quantification, which has translated to clinical workflows in some regions. We systematically review and discuss this literature, highlight opportunities for multidisciplinary collaboration, and identify potentially clinically useful topics with a relative paucity of research attention. Musculoskeletal oncology is an inherently multidisciplinary field, and future research will need to integrate and synthesize noisy siloed data from across clinical, imaging, and molecular datasets. Building the data infrastructure for collaboration will help to accelerate progress towards making artificial intelligence truly useful in musculoskeletal oncology.

Deep Learning Classification of Breast Cancer Tissue from Terahertz Imaging Through Wavelet Synchro-Squeezed Transformation and Transfer Learning

Terahertz imaging and spectroscopy is an exciting technology that has the potential to provide insights in medical imaging. Prior research has leveraged statistical inference to classify tissue regions from terahertz images. To date, these approaches have shown that the segmentation problem is challenging for images of fresh tissue and for tumors that have invaded muscular regions. Artificial intelligence, particularly machine learning and deep learning, has been shown to improve performance in some medical imaging challenges. This paper builds on that literature by modifying a set of deep learning approaches to the challenge of classifying tissue regions of images captured by terahertz imaging and spectroscopy of freshly excised murine xenograft tissue. Our approach is to preprocess the images through a wavelet synchronous-squeezed transformation (WSST) to convert time-sequential terahertz data of each THz pixel to a spectrogram. Spectrograms are used as input tensors to a deep convolution neural network for pixel-wise classification. Based on the classification result of each pixel, a cancer tissue segmentation map is achieved. In experimentation, we adopt leave-one-sample-out cross-validation strategy, and evaluate our chosen networks and results using multiple metrics such as accuracy, precision, intersection, and size. The results from this experimentation demonstrate improvement in classification accuracy compared to statistical methods, an improvement to segmentation between muscle and cancerous regions in xenograft tumors, and identify areas to improve the imaging and classification methodology.

An update on the knee osteoarthritis severity grading using wide residual learning

BACKGROUND

Knee Osteoarthritis (KOA) is the most common type of Osteoarthritis (OA) and it is diagnosed by physicians using a standard 0 -4 Kellgren Lawrence (KL) grading system which sets the KOA on a spectrum of 5 grades; starting from normal (0) to Severe OA (4).

OBJECTIVES

In this paper, we propose a transfer learning approach of a very deep wide residual learning-based network (WRN-50-2) which is fine-tuned using X-ray plain radiographs from the Osteoarthritis Initiative (OAI) dataset to learn the KL severity grading of KOA.

METHODS

We propose a data augmentation approach of OAI data to avoid data imbalance and reduce overfitting by applying it only to certain KL grades depending on their number of plain radiographs. Then we conduct experiments to test the model based on an independent testing data of original plain radiographs acquired from the OAI dataset.

RESULTS

Experimental results showed good generalization power in predicting the KL grade of knee X-rays with an accuracy of 72% and Precision 74%. Moreover, using Grad-Cam, we also observed that network selected some distinctive features that describe the prediction of a KL grade of a knee radiograph.

CONCLUSION

This study demonstrates that our proposed new model outperforms several other related works, and it can be further improved to be used to help radiologists make more accurate and precise diagnosis of KOA in future clinical practice.

Emergence of Deep Learning in Knee Osteoarthritis Diagnosis

Osteoarthritis (OA), especially knee OA, is the most common form of arthritis, causing significant disability in patients worldwide. Manual diagnosis, segmentation, and annotations of knee joints remain as the popular method to diagnose OA in clinical practices, although they are tedious and greatly subject to user variation. Therefore, to overcome the limitations of the commonly used method as above, numerous deep learning approaches, especially the convolutional neural network (CNN), have been developed to improve the clinical workflow efficiency. Medical imaging processes, especially those that produce 3-dimensional (3D) images such as MRI, possess ability to reveal hidden structures in a volumetric view. Acknowledging that changes in a knee joint is a 3D complexity, 3D CNN has been employed to analyse the joint problem for a more accurate diagnosis in the recent years. In this review, we provide a broad overview on the current 2D and 3D CNN approaches in the OA research field. We reviewed 74 studies related to classification and segmentation of knee osteoarthritis from the Web of Science database and discussed the various state-of-the-art deep learning approaches proposed. We highlighted the potential and possibility of 3D CNN in the knee osteoarthritis field. We concluded by discussing the possible challenges faced as well as the potential advancements in adopting 3D CNNs in this field.

Differentiation of COVID-19 conditions in planar chest radiographs using optimized convolutional neural networks

In this study, an attempt has been made to differentiate Novel Coronavirus-2019 (COVID-19) conditions from healthy subjects in Chest radiographs using a simplified end-to-end Convolutional Neural Network (CNN) model and occlusion sensitivity maps. Early detection and faster automated screening of the COVID-19 patients is essential. For this, the images are considered from publicly available datasets. Significant biomarkers representing critical image features are extracted from CNN by experimentally investigating on cross-validation methods and hyperparameter settings. The performance of the network is evaluated using standard metrics. Perturbation based occlusion sensitivity maps are employed on the features obtained from the classification model to visualise the localization of abnormal areas. Results demonstrate that the simplified CNN model with optimised parameters is able to extract significant features with a sensitivity of 97.35% and F-measure of 96.71% to detect COVID-19 images. The algorithm achieves an Area Under the Curve-Receiver Operating Characteristic score of 99.4% with Matthews correlation coefficient of 0.93. High value of Diagnostic odds ratio is also obtained. Occlusion sensitivity maps provide precise localization of abnormal regions by identifying COVID-19 conditions. As early detection through chest radiographic images are useful for automated screening of the disease, this method appears to be clinically relevant in providing a visual diagnostic solution using a simplified and efficient model.

Two Eyes Are Better Than One: Exploiting Binocular Correlation for Diabetic Retinopathy Severity Grading

Diabetic retinopathy (DR) is one of the most common eye conditions among diabetic patients. However, vision loss occurs primarily in the late stages of DR, and the symptoms of visual impairment, ranging from mild to severe, can vary greatly, adding to the burden of diagnosis and treatment in clinical practice. Deep learning methods based on retinal images have achieved remarkable success in automatic DR grading, but most of them neglect that the presence of diabetes usually affects both eyes, and ophthalmologists usually compare both eyes concurrently for DR diagnosis, leaving correlations between left and right eyes unexploited. In this study, simulating the diagnostic process, we propose a two-stream binocular network to capture the subtle correlations between left and right eyes, in which, paired images of eyes are fed into two identical subnetworks separately during training. We design a contrastive grading loss to learn binocular correlation for five-class DR detection, which maximizes inter-class dissimilarity while minimizing the intra-class difference. Experimental results on the EyePACS dataset show the superiority of the proposed binocular model, outperforming monocular methods by a large margin.Clinical relevance- Compared to conventional DR grading methods based on monocular images, our approach can provide more accurate predictions and extract graphical patterns from retinal images of both eyes for clinical reference.

Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging

PURPOSE

To evaluate the trustworthiness of saliency maps for abnormality localization in medical imaging.

MATERIALS AND METHODS

Using two large publicly available radiology datasets (Society for Imaging Informatics in Medicine-American College of Radiology Pneumothorax Segmentation dataset and Radiological Society of North America Pneumonia Detection Challenge dataset), the performance of eight commonly used saliency map techniques were quantified in regard to (a) localization utility (segmentation and detection), (b) sensitivity to model weight randomization, (c) repeatability, and (d) reproducibility. Their performances versus baseline methods and localization network architectures were compared, using area under the precision-recall curve (AUPRC) and structural similarity index measure (SSIM) as metrics.

RESULTS

All eight saliency map techniques failed at least one of the criteria and were inferior in performance compared with localization networks. For pneumothorax segmentation, the AUPRC ranged from 0.024 to 0.224, while a U-Net achieved a significantly superior AUPRC of 0.404 (P < .005). For pneumonia detection, the AUPRC ranged from 0.160 to 0.519, while a RetinaNet achieved a significantly superior AUPRC of 0.596 (P <.005). Five and two saliency methods (of eight) failed the model randomization test on the segmentation and detection datasets, respectively, suggesting that these methods are not sensitive to changes in model parameters. The repeatability and reproducibility of the majority of the saliency methods were worse than localization networks for both the segmentation and detection datasets.

CONCLUSION

The use of saliency maps in the high-risk domain of medical imaging warrants additional scrutiny and recommend that detection or segmentation models be used if localization is the desired output of the network.Keywords: Technology Assessment, Technical Aspects, Feature Detection, Convolutional Neural Network (CNN) Supplemental material is available for this article. © RSNA, 2021.

IFSS-Net: Interactive Few-Shot Siamese Network for Faster Muscle Segmentation and Propagation in Volumetric Ultrasound

We present an accurate, fast and efficient method for segmentation and muscle mask propagation in 3D freehand ultrasound data, towards accurate volume quantification. A deep Siamese 3D Encoder-Decoder network that captures the evolution of the muscle appearance and shape for contiguous slices is deployed. We use it to propagate a reference mask annotated by a clinical expert. To handle longer changes of the muscle shape over the entire volume and to provide an accurate propagation, we devise a Bidirectional Long Short Term Memory module. Also, to train our model with a minimal amount of training samples, we propose a strategy combining learning from few annotated 2D ultrasound slices with sequential pseudo-labeling of the unannotated slices. We introduce a decremental update of the objective function to guide the model convergence in the absence of large amounts of annotated data. After training with a few volumes, the decremental update strategy switches from a weak supervised training to a few-shot setting. Finally, to handle the class-imbalance between foreground and background muscle pixels, we propose a parametric Tversky loss function that learns to penalize adaptively the false positives and the false negatives. We validate our approach for the segmentation, label propagation, and volume computation of the three low-limb muscles on a dataset of 61600 images from 44 subjects. We achieve a Dice score coefficient of over 95% and a volumetric error of 1.6035 ± 0.587%.

Machine learning techniques demonstrating individual movement patterns of the vertebral column: the fingerprint of spinal motion

Surface topography systems enable the capture of spinal dynamic movement; however, it is unclear whether vertebral dynamics are unique enough to identify individuals. Therefore, in this study, we investigated whether the identification of individuals is possible based on dynamic spinal data. Three different data representations were compared (automated extracted features using contrastive loss and triplet loss functions, as well as simple descriptive statistics). High accuracies indicated the possible existence of a personal spinal 'fingerprint', therefore enabling subject recognition. The present work forms the basis for an objective comparison of subjects and the transfer of the method to clinical use cases.

Intelligent System for Railway Wheelset Press-Fit Inspection Using Deep Learning

Railway wheelsets are the key to ensuring the safe operation of trains. To achieve zero-defect production, railway equipment manufacturers must strictly control every link in the wheelset production process. The press-fit curve output by the wheelset assembly machine is an essential indicator of the wheelset’s assembly quality. The operators will still need to manually and individually recheck press-fit curves in our practical case. However, there are many uncertainties in the manual inspection. For example, subjective judgment can easily cause inconsistent judgment results between different inspectors, or the probability of human misinterpretation can increase as the working hours increase. Therefore, this study proposes an intelligent railway wheelset inspection system based on deep learning, which improves the reliability and efficiency of manual inspection of wheelset assembly quality. To solve the severe imbalance in the number of collected images, this study establishes a predicted model of press-fit quality based on a deep Siamese network. Our experimental results show that the precision measurement is outstanding for the testing dataset contained 3863 qualified images and 28 unqualified images of press-fit curves. The proposed system will serve as a successful case of a paradigm shift from traditional manufacturing to digital manufacturing.

Representation Learning for Fine-Grained Change Detection

Fine-grained change detection in sensor data is very challenging for artificial intelligence though it is critically important in practice. It is the process of identifying differences in the state of an object or phenomenon where the differences are class-specific and are difficult to generalise. As a result, many recent technologies that leverage big data and deep learning struggle with this task. This review focuses on the state-of-the-art methods, applications, and challenges of representation learning for fine-grained change detection. Our research focuses on methods of harnessing the latent metric space of representation learning techniques as an interim output for hybrid human-machine intelligence. We review methods for transforming and projecting embedding space such that significant changes can be communicated more effectively and a more comprehensive interpretation of underlying relationships in sensor data is facilitated. We conduct this research in our work towards developing a method for aligning the axes of latent embedding space with meaningful real-world metrics so that the reasoning behind the detection of change in relation to past observations may be revealed and adjusted. This is an important topic in many fields concerned with producing more meaningful and explainable outputs from deep learning and also for providing means for knowledge injection and model calibration in order to maintain user confidence.

Deep learning prediction of mild cognitive impairment conversion to Alzheimer's disease at 3 years after diagnosis using longitudinal and whole-brain 3D MRI

BACKGROUND

While there is no cure for Alzheimer's disease (AD), early diagnosis and accurate prognosis of AD may enable or encourage lifestyle changes, neurocognitive enrichment, and interventions to slow the rate of cognitive decline. The goal of our study was to develop and evaluate a novel deep learning algorithm to predict mild cognitive impairment (MCI) to AD conversion at three years after diagnosis using longitudinal and whole-brain 3D MRI.

METHODS

This retrospective study consisted of 320 normal cognition (NC), 554 MCI, and 237 AD patients. Longitudinal data include T1-weighted 3D MRI obtained at initial presentation with diagnosis of MCI and at 12-month follow up. Whole-brain 3D MRI volumes were used without a priori segmentation of regional structural volumes or cortical thicknesses. MRIs of the AD and NC cohort were used to train a deep learning classification model to obtain weights to be applied via transfer learning for prediction of MCI patient conversion to AD at three years post-diagnosis. Two (zero-shot and fine tuning) transfer learning methods were evaluated. Three different convolutional neural network (CNN) architectures (sequential, residual bottleneck, and wide residual) were compared. Data were split into 75% and 25% for training and testing, respectively, with 4-fold cross validation. Prediction accuracy was evaluated using balanced accuracy. Heatmaps were generated.

RESULTS

The sequential convolutional approach yielded slightly better performance than the residual-based architecture, the zero-shot transfer learning approach yielded better performance than fine tuning, and CNN using longitudinal data performed better than CNN using a single timepoint MRI in predicting MCI conversion to AD. The best CNN model for predicting MCI conversion to AD at three years after diagnosis yielded a balanced accuracy of 0.793. Heatmaps of the prediction model showed regions most relevant to the network including the lateral ventricles, periventricular white matter and cortical gray matter.

CONCLUSIONS

This is the first convolutional neural network model using longitudinal and whole-brain 3D MRIs without extracting regional brain volumes or cortical thicknesses to predict future MCI to AD conversion at 3 years after diagnosis. This approach could lead to early prediction of patients who are likely to progress to AD and thus may lead to better management of the disease.

Multi-Radiologist User Study for Artificial Intelligence-Guided Grading of COVID-19 Lung Disease Severity on Chest Radiographs

RATIONALE AND OBJECTIVES

Radiographic findings of COVID-19 pneumonia can be used for patient risk stratification; however, radiologist reporting of disease severity is inconsistent on chest radiographs (CXRs). We aimed to see if an artificial intelligence (AI) system could help improve radiologist interrater agreement.

MATERIALS AND METHODS

We performed a retrospective multi-radiologist user study to evaluate the impact of an AI system, the PXS score model, on the grading of categorical COVID-19 lung disease severity on 154 chest radiographs into four ordinal grades (normal/minimal, mild, moderate, and severe). Four radiologists (two thoracic and two emergency radiologists) independently interpreted 154 CXRs from 154 unique patients with COVID-19 hospitalized at a large academic center, before and after using the AI system (median washout time interval was 16 days). Three different thoracic radiologists assessed the same 154 CXRs using an updated version of the AI system trained on more imaging data. Radiologist interrater agreement was evaluated using Cohen and Fleiss kappa where appropriate. The lung disease severity categories were associated with clinical outcomes using a previously published outcomes dataset using Fisher's exact test and Chi-square test for trend.

RESULTS

Use of the AI system improved radiologist interrater agreement (Fleiss κ = 0.40 to 0.66, before and after use of the system). The Fleiss κ for three radiologists using the updated AI system was 0.74. Severity categories were significantly associated with subsequent intubation or death within 3 days.

CONCLUSION

An AI system used at the time of CXR study interpretation can improve the interrater agreement of radiologists.

Predicting treatment response from longitudinal images using multi-task deep learning

Radiographic imaging is routinely used to evaluate treatment response in solid tumors. Current imaging response metrics do not reliably predict the underlying biological response. Here, we present a multi-task deep learning approach that allows simultaneous tumor segmentation and response prediction. We design two Siamese subnetworks that are joined at multiple layers, which enables integration of multi-scale feature representations and in-depth comparison of pre-treatment and post-treatment images. The network is trained using 2568 magnetic resonance imaging scans of 321 rectal cancer patients for predicting pathologic complete response after neoadjuvant chemoradiotherapy. In multi-institution validation, the imaging-based model achieves AUC of 0.95 (95% confidence interval: 0.91–0.98) and 0.92 (0.87–0.96) in two independent cohorts of 160 and 141 patients, respectively. When combined with blood-based tumor markers, the integrated model further improves prediction accuracy with AUC 0.97 (0.93–0.99). Our approach to capturing dynamic information in longitudinal images may be broadly used for screening, treatment response evaluation, disease monitoring, and surveillance.

Radiographic imaging is routinely used to evaluate treatment response in solid tumors. Here, the authors present a multi-task deep learning approach that allows simultaneous tumor segmentation and response prediction from longitudinal images in a multi-center study on rectal cancer.

Short-Term Lesion Change Detection for Melanoma Screening With Novel Siamese Neural Network

Short-term monitoring of lesion changes has been a widely accepted clinical guideline for melanoma screening. When there is a significant change of a melanocytic lesion at three months, the lesion will be excised to exclude melanoma. However, the decision on change or no-change heavily depends on the experience and bias of individual clinicians, which is subjective. For the first time, a novel deep learning based method is developed in this paper for automatically detecting short-term lesion changes in melanoma screening. The lesion change detection is formulated as a task measuring the similarity between two dermoscopy images taken for a lesion in a short time-frame, and a novel Siamese structure based deep network is proposed to produce the decision: changed (i.e. not similar) or unchanged (i.e. similar enough). Under the Siamese framework, a novel structure, namely Tensorial Regression Process, is proposed to extract the global features of lesion images, in addition to deep convolutional features. In order to mimic the decision-making process of clinicians who often focus more on regions with specific patterns when comparing a pair of lesion images, a segmentation loss (SegLoss) is further devised and incorporated into the proposed network as a regularization term. To evaluate the proposed method, an in-house dataset with 1,000 pairs of lesion images taken in a short time-frame at a clinical melanoma centre was established. Experimental results on this first-of-a-kind large dataset indicate that the proposed model is promising in detecting the short-term lesion change for objective melanoma screening.

Osteoarthritis year in review 2020: imaging

This narrative "Year in Review" highlights a selection of articles published between January 2019 and April 2020, to be presented at the OARSI World Congress 2020 within the field of osteoarthritis (OA) imaging. Articles were obtained from a PubMed search covering the above period, utilizing a variety of relevant search terms. We then selected original and review studies on OA-related imaging in humans, particularly those with direct clinical relevance, with a focus on the knee. Topics selected encompassed clinically relevant models of early OA, particularly imaging applications on cruciate ligament rupture, as these are of direct clinical interest and provide potential opportunity to evaluate preventive therapy. Further, imaging applications on structural modification of articular tissues in patients with established OA, by non-pharmacological, pharmacological and surgical interventions are summarized. Finally, novel deep learning approaches to imaging are reviewed, as these facilitate implementation and scaling of quantitative imaging application in clinical trials and clinical practice. Methodological or observational studies outside these key focus areas were not included. Studies focused on biology, biomechanics, biomarkers, genetics and epigenetics, and clinical studies that did not contain an imaging component are covered in other articles within the OARSI "Year in Review" series. In conclusion, exciting progress has been made in clinically validating human models of early OA, and the field of automated articular tissue segmentation. Most importantly though, it has been shown that structure modification of articular cartilage is possible, and future research should focus on the translation of these structural findings to clinical benefit.

Multi-Institutional Assessment and Crowdsourcing Evaluation of Deep Learning for Automated Classification of Breast Density

OBJECTIVE

We developed deep learning algorithms to automatically assess BI-RADS breast density.

METHODS

Using a large multi-institution patient cohort of 108,230 digital screening mammograms from the Digital Mammographic Imaging Screening Trial, we investigated the effect of data, model, and training parameters on overall model performance and provided crowdsourcing evaluation from the attendees of the ACR 2019 Annual Meeting.

RESULTS

Our best-performing algorithm achieved good agreement with radiologists who were qualified interpreters of mammograms, with a four-class κ of 0.667. When training was performed with randomly sampled images from the data set versus sampling equal number of images from each density category, the model predictions were biased away from the low-prevalence categories such as extremely dense breasts. The net result was an increase in sensitivity and a decrease in specificity for predicting dense breasts for equal class compared with random sampling. We also found that the performance of the model degrades when we evaluate on digital mammography data formats that differ from the one that we trained on, emphasizing the importance of multi-institutional training sets. Lastly, we showed that crowdsourced annotations, including those from attendees who routinely read mammograms, had higher agreement with our algorithm than with the original interpreting radiologists.

CONCLUSION

We demonstrated the possible parameters that can influence the performance of the model and how crowdsourcing can be used for evaluation. This study was performed in tandem with the development of the ACR AI-LAB, a platform for democratizing artificial intelligence.

Automated Assessment and Tracking of COVID-19 Pulmonary Disease Severity on Chest Radiographs using Convolutional Siamese Neural Networks

PURPOSE

To develop an automated measure of COVID-19 pulmonary disease severity on chest radiographs (CXRs), for longitudinal disease tracking and outcome prediction.

MATERIALS AND METHODS

A convolutional Siamese neural network-based algorithm was trained to output a measure of pulmonary disease severity on CXRs (pulmonary x-ray severity (PXS) score), using weakly-supervised pretraining on ∼160,000 anterior-posterior images from CheXpert and transfer learning on 314 frontal CXRs from COVID-19 patients. The algorithm was evaluated on internal and external test sets from different hospitals (154 and 113 CXRs respectively). PXS scores were correlated with radiographic severity scores independently assigned by two thoracic radiologists and one in-training radiologist (Pearson r). For 92 internal test set patients with follow-up CXRs, PXS score change was compared to radiologist assessments of change (Spearman ρ). The association between PXS score and subsequent intubation or death was assessed. Bootstrap 95% confidence intervals (CI) were calculated.

RESULTS

PXS scores correlated with radiographic pulmonary disease severity scores assigned to CXRs in the internal and external test sets (r=0.86 (95%CI 0.80-0.90) and r=0.86 (95%CI 0.79-0.90) respectively). The direction of change in PXS score in follow-up CXRs agreed with radiologist assessment (ρ=0.74 (95%CI 0.63-0.81)). In patients not intubated on the admission CXR, the PXS score predicted subsequent intubation or death within three days of hospital admission (area under the receiver operating characteristic curve=0.80 (95%CI 0.75-0.85)).

CONCLUSION

A Siamese neural network-based severity score automatically measures radiographic COVID-19 pulmonary disease severity, which can be used to track disease change and predict subsequent intubation or death.

Automated assessment of COVID-19 pulmonary disease severity on chest radiographs using convolutional Siamese neural networks

Purpose To develop an automated measure of COVID-19 pulmonary disease severity on chest radiographs (CXRs), for longitudinal disease evaluation and clinical risk stratification. Materials and Methods A convolutional Siamese neural network-based algorithm was trained to output a measure of pulmonary disease severity on anterior-posterior CXRs (pulmonary x-ray severity (PXS) score), using weakly-supervised pretraining on ~160,000 images from CheXpert and transfer learning on 314 CXRs from patients with COVID-19. The algorithm was evaluated on internal and external test sets from different hospitals, containing 154 and 113 CXRs respectively. The PXS score was correlated with a radiographic severity score independently assigned by two thoracic radiologists and one in-training radiologist. For 92 internal test set patients with follow-up CXRs, the change in PXS score was compared to radiologist assessments of change. The association between PXS score and subsequent intubation or death was assessed. Results The PXS score correlated with the radiographic pulmonary disease severity score assigned to CXRs in the COVID-19 internal and external test sets (ρ=0.84 and ρ=0.78 respectively). The direction of change in PXS score in follow-up CXRs agreed with radiologist assessment (ρ=0.74). In patients not intubated on the admission CXR, the PXS score predicted subsequent intubation or death within three days of hospital admission (area under the receiver operator characteristic curve=0.80 (95%CI 0.75-0.85)). Conclusion A Siamese neural network-based severity score automatically measures COVID-19 pulmonary disease severity in chest radiographs, which can be scaled and rapidly deployed for clinical triage and workflow optimization.