Self-supervised learning for medical image classification: a systematic review and implementation guidelines

Using deep feature distances for evaluating the perceptual quality of MR image reconstructions

PURPOSE

Commonly used MR image quality (IQ) metrics have poor concordance with radiologist-perceived diagnostic IQ. Here, we develop and explore deep feature distances (DFDs)-distances computed in a lower-dimensional feature space encoded by a convolutional neural network (CNN)-as improved perceptual IQ metrics for MR image reconstruction. We further explore the impact of distribution shifts between images in the DFD CNN encoder training data and the IQ metric evaluation.

METHODS

We compare commonly used IQ metrics (PSNR and SSIM) to two "out-of-domain" DFDs with encoders trained on natural images, an "in-domain" DFD trained on MR images alone, and two domain-adjacent DFDs trained on large medical imaging datasets. We additionally compare these with several state-of-the-art but less commonly reported IQ metrics, visual information fidelity (VIF), noise quality metric (NQM), and the high-frequency error norm (HFEN). IQ metric performance is assessed via correlations with five expert radiologist reader scores of perceived diagnostic IQ of various accelerated MR image reconstructions. We characterize the behavior of these IQ metrics under common distortions expected during image acquisition, including their sensitivity to acquisition noise.

RESULTS

All DFDs and HFEN correlate more strongly with radiologist-perceived diagnostic IQ than SSIM, PSNR, and other state-of-the-art metrics, with correlations being comparable to radiologist inter-reader variability. Surprisingly, out-of-domain DFDs perform comparably to in-domain and domain-adjacent DFDs.

CONCLUSION

A suite of IQ metrics, including DFDs and HFEN, should be used alongside commonly-reported IQ metrics for a more holistic evaluation of MR image reconstruction perceptual quality. We also observe that general vision encoders are capable of assessing visual IQ even for MR images.

NaMA-Mamba: Foundation model for generalizable nasal disease detection using masked autoencoder with Mamba on endoscopic images

Artificial intelligence (AI) has shown great promise in analyzing nasal endoscopic images for disease detection. However, current AI systems require extensive expert-labeled data for each specific medical condition, limiting their applications. In this work, the challenge is addressed through two key innovations, the creation of the first large-scale pre-training dataset of nasal endoscopic images, and the development of a novel self-learning AI system specifically designed for nasal endoscopy, named NaMA-Mamba. In the proposed NaMA-Mamba model, two key technologies are utilized, which are the nasal endoscopic state space model (NE-SSM) for analyzing sequences of images and an enhanced learning mechanism (CoMAE) for capturing fine details in nasal tissues. These innovations enable the system to learn effectively from unlabeled images while maintaining high accuracy across different diagnostic tasks. In extensive testing, NaMA-Mamba achieved remarkable results using minimal labeled data, matching the performance of traditional systems that require full expert labeling while needing only 1% of the labeled data for tasks such as detecting nasal polyps and identifying nasopharyngeal cancer. These results demonstrate the potential of NaMA-Mamba to significantly improve the efficiency and accessibility of AI-assisted nasal disease diagnosis in clinical practice.

Automated radiography assessment of ankle joint instability using deep learning

This study developed and evaluated a deep learning (DL)-based system for automatically measuring talar tilt and anterior talar translation on weight-bearing ankle radiographs, which are key parameters in diagnosing ankle joint instability. The system was trained and tested using a dataset comprising of 1,452 anteroposterior radiographs (mean age ± standard deviation [SD]: 43.70 ± 22.60 years; age range: 6-87 years; males: 733, females: 719) and 2,984 lateral radiographs (mean age ± SD: 44.37 ± 22.72 years; age range: 6-92 years; male: 1,533, female: 1,451) from a total of 4,000 patients, provided by the National Information Society Agency. Patients who underwent joint fusion, bone grafting, or joint replacement were excluded. Statistical analyses, including correlation coefficient analysis and Bland-Altman plots, were conducted to assess the agreement and consistency between the DL-calculated and clinician-assessed measurements. The system demonstrated high accuracy, with strong correlations for talar tilt (Pearson correlation coefficient [r] = 0.798 (p < .001); intraclass correlation coefficient [ICC] = 0.797 [95% CI 0.74, 0.82]; concordance correlation coefficient [CCC] = 0.796 [95% CI 0.69, 0.85]; mean absolute error [MAE] = 1.088° [95% CI 0.06°, 1.14°]; mean square error [MSE] = 1.780° [95% CI 1.69°, 2.73°]; root mean square error [RMSE] = 1.374° [95% CI 1.31°, 1.44°]; 95% limit of agreement [LoA], 2.0° to - 2.3°) and anterior talar translation (r = .862 (p < .001); ICC = 0.861 [95% CI 0.84, 0.89]; CCC = 0.861 [95% CI 0.86, 0.89]; MAE = 0.468 mm [95% CI 0.42 mm, 0.51 mm]; MSE = 0.551 mm [95% CI 0.49 mm, 0.61 mm]; RMSE = 0.742 mm [95% CI 0.69 mm, 0.79 mm]; 95% LoA, 1.5 mm to - 1.3 mm). These results demonstrate the system's capability to provide objective and reproducible measurements, supporting clinical interpretation of ankle instability in routine radiographic practice.

Overcoming barriers in the use of artificial intelligence in point of care ultrasound

Point-of-care ultrasound is a portable, low-cost imaging technology focused on answering specific clinical questions in real time. Artificial intelligence amplifies its capabilities by aiding clinicians in the acquisition and interpretation of the images; however, there are growing concerns on its effectiveness and trustworthiness. Here, we address key issues such as population bias, explainability and training of artificial intelligence in this field and propose approaches to ensure clinical effectiveness.

Tailored self-supervised pretraining improves brain MRI diagnostic models

Self-supervised learning has shown potential in enhancing deep learning methods, yet its application in brain magnetic resonance imaging (MRI) analysis remains underexplored. This study seeks to leverage large-scale, unlabeled public brain MRI datasets to improve the performance of deep learning models in various downstream tasks for the development of clinical decision support systems. To enhance training efficiency, data filtering methods based on image entropy and slice positions were developed, condensing a combined dataset of approximately 2 million images from fastMRI-brain, OASIS-3, IXI, and BraTS21 into a more focused set of 250 K images enriched with brain features. The Momentum Contrast (MoCo) v3 algorithm was then employed to learn these image features, resulting in robustly pretrained models specifically tailored to brain MRI. The pretrained models were subsequently evaluated in tumor classification, lesion detection, hippocampal segmentation, and image reconstruction tasks. The results demonstrate that our brain MRI-oriented pretraining outperformed both ImageNet pretraining and pretraining on larger multi-organ, multi-modality medical datasets, achieving a ∼2.8 % increase in 4-class tumor classification accuracy, a ∼0.9 % improvement in tumor detection mean average precision, a ∼3.6 % gain in adult hippocampal segmentation Dice score, and a ∼0.1 PSNR improvement in reconstruction at 2-fold acceleration. This study underscores the potential of self-supervised learning for brain MRI using large-scale, tailored datasets derived from public sources.

Self-HER2Net: A generative self-supervised framework for HER2 classification in IHC histopathology of breast cancer

Breast cancer is a significant global health concern, where precise identification of proteins like Human Epidermal Growth Factor Receptor 2 (HER2) in cancer cells via Immunohistochemistry (IHC) is pivotal for treatment decisions. HER2 overexpression is evaluated through HER2 scoring on a scale from 0 to 3 + based on staining patterns and intensity. Recent efforts have been made to automate HER2 scoring using image processing and AI techniques. However, existing methods require large manually annotated datasets as these follow supervised learning paradigms. Therefore, we proposed a generative self-supervised learning (SSL) framework "Self-HER2Net" for the classification of HER2 scoring, to reduce dependence on large manually annotated data by leveraging one of best performing four novel generative self-supervised tasks, that we proposed. The first two SSL tasks HER2hsl and HER2hsv are domain-agnostic and the other two HER2dab and HER2hae are domain-specific SSL tasks focusing on domain-agnostic and domain-specific staining patterns and intensity representation. Our approach is evaluated under different budget scenarios (2 %, 15 %, & 100 % labeled datasets) and also out distribution test. For tile-level assessment, HER2hsv achieved the best performance with AUC-ROC of 0.965 ± 0.037. Our self-supervised learning approach shows potential for application in scenarios with limited annotated data for HER2 analysis.

Benefit from public unlabeled data: A Frangi filter-based pretraining network for 3D cerebrovascular segmentation

Precise cerebrovascular segmentation in Time-of-Flight Magnetic Resonance Angiography (TOF-MRA) data is crucial for computer-aided clinical diagnosis. The sparse distribution of cerebrovascular structures within TOF-MRA images often results in high costs for manual data labeling. Leveraging unlabeled TOF-MRA data can significantly enhance model performance. In this study, we have constructed the largest preprocessed unlabeled TOF-MRA dataset to date, comprising 1510 subjects. Additionally, we provide manually annotated segmentation masks for 113 subjects based on existing external image datasets to facilitate evaluation. We propose a simple yet effective pretraining strategy utilizing the Frangi filter, known for its capability to enhance vessel-like structures, to optimize the use of the unlabeled data for 3D cerebrovascular segmentation. This involves a Frangi filter-based preprocessing workflow tailored for large-scale unlabeled datasets and a multi-task pretraining strategy to efficiently utilize the preprocessed data. This approach ensures maximal extraction of useful knowledge from the unlabeled data. The efficacy of the pretrained model is assessed across four cerebrovascular segmentation datasets, where it demonstrates superior performance, improving the clDice metric by approximately 2%-3% compared to the latest semi- and self-supervised methods. Additionally, ablation studies validate the generalizability and effectiveness of our pretraining method across various backbone structures. The code and data have been open source at: https://github.com/shigen-StoneRoot/FFPN.

Self-supervised learning for graph-structured data in healthcare applications: A comprehensive review

The increasing complexity and interconnectedness of healthcare data present numerous opportunities to improve prediction, diagnosis, and treatment. Graph-structured data, which represents entities and their relationships, is well-suited for modeling these complex connections. However, effectively utilizing this data often requires strong and efficient learning algorithms, especially when dealing with limited labeled data. Self-supervised learning (SSL) has emerged as a powerful paradigm for leveraging unlabeled data to learn effective representations. This paper presents a comprehensive review of SSL approaches specifically designed for graph-structured data in healthcare applications. We explore the challenges and opportunities associated with healthcare data and assess the effectiveness of SSL techniques in real-world healthcare applications. Our discussion encompasses various healthcare settings, such as disease prediction, medical image analysis, and drug discovery. We critically evaluate the performance of different SSL methods across these tasks, highlighting their strengths, limitations, and potential future research directions. To the best of our knowledge, this is the first comprehensive review of SSL applied to graph data in healthcare, providing valuable guidance for researchers and practitioners looking to leverage these techniques to enhance outcomes and drive progress in the field.

A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings

Large foundation models show promise in biomedicine but face challenges in clinical use due to performance gaps, accessibility, cost, and lack of scalable evaluation. Here we show that open-source small multimodal models can bridge these gaps in radiology by generating free-text findings from chest X-ray images. Our data-centric approach leverages 697K curated radiology image-text pairs to train a specialized, domain-adapted chest X-ray encoder. We integrate this encoder with pre-trained language models via a lightweight adapter that aligns image and text modalities. To enable robust, clinically relevant evaluation, we develop and validate CheXprompt, a GPT-4-based metric for assessing factual accuracy aligned with radiologists' evaluations. Benchmarked with CheXprompt and other standard factuality metrics, LLaVA-Rad (7B) achieves state-of-the-art performance, outperforming much larger models like GPT-4V and Med-PaLM M (84B). While not immediately ready for real-time clinical deployment, LLaVA-Rad is a scalable, privacy-preserving and cost-effective step towards clinically adaptable multimodal AI for radiology.

Accurate prediction of disease-risk factors from volumetric medical scans by a deep vision model pre-trained with 2D scans

The application of machine learning to tasks involving volumetric biomedical imaging is constrained by the limited availability of annotated datasets of three-dimensional (3D) scans for model training. Here we report a deep-learning model pre-trained on 2D scans (for which annotated data are relatively abundant) that accurately predicts disease-risk factors from 3D medical-scan modalities. The model, which we named SLIViT (for 'slice integration by vision transformer'), preprocesses a given volumetric scan into 2D images, extracts their feature map and integrates it into a single prediction. We evaluated the model in eight different learning tasks, including classification and regression for six datasets involving four volumetric imaging modalities (computed tomography, magnetic resonance imaging, optical coherence tomography and ultrasound). SLIViT consistently outperformed domain-specific state-of-the-art models and was typically as accurate as clinical specialists who had spent considerable time manually annotating the analysed scans. Automating diagnosis tasks involving volumetric scans may save valuable clinician hours, reduce data acquisition costs and duration, and help expedite medical research and clinical applications.

A vision-language foundation model for the generation of realistic chest X-ray images

The paucity of high-quality medical imaging datasets could be mitigated by machine learning models that generate compositionally diverse images that faithfully represent medical concepts and pathologies. However, large vision-language models are trained on natural images, and the diversity distribution of the generated images substantially differs from that of medical images. Moreover, medical language involves specific and semantically rich vocabulary. Here we describe a domain-adaptation strategy for large vision-language models that overcomes distributional shifts. Specifically, by leveraging publicly available datasets of chest X-ray images and the corresponding radiology reports, we adapted a latent diffusion model pre-trained on pairs of natural images and text descriptors to generate diverse and visually plausible synthetic chest X-ray images (as confirmed by board-certified radiologists) whose appearance can be controlled with free-form medical text prompts. The domain-adaptation strategy for the text-conditioned synthesis of medical images can be used to augment training datasets and is a viable alternative to the sharing of real medical images for model training and fine-tuning.

Automated detection of retinal artery occlusion in fundus photography via self-supervised deep learning and multimodal interpretability using a multimodal AI chatbot

Retinal artery occlusion (RAO) is a sight-threatening condition that requires prompt diagnosis to prevent irreversible vision loss. This study presents an innovative AI-driven approach for RAO detection from fundus images, marking the first application of deep learning for this purpose. Using a self-supervised learning (SSL) framework with SimCLR, our model addresses the challenge of limited labeled RAO data. The ResNet50 model pretrained with SimCLR demonstrated high diagnostic accuracy, achieving areas under the receiver operating characteristic curve (AUC) of 0.924 and 0.988 on two external validation datasets, highlighting its robustness and generalizability in RAO detection. To enhance transparency in clinical AI, we incorporated a multimodal interpretability approach using a ChatGPT-4-based AI chatbot. This chatbot, combined with Grad-CAM visualizations, provides detailed clinical explanations of the model's predictions, emphasizing key RAO features such as retinal whitening and cherry-red spots. This multimodal interpretability framework improves clinicians' understanding of the model's decision-making process, facilitating clinical adoption and trust. By automating RAO detection, this AI model serves as a valuable tool for the early identification of ocular and systemic vascular risks, enabling timely intervention. These findings highlight the potential of fundus imaging for RAO detection and broader cardiovascular risk assessment, advancing AI's role in predictive healthcare.

Self-Supervised Multi-Task Learning for the Detection and Classification of RHD-Induced Valvular Pathology

Rheumatic heart disease (RHD) poses a significant global health challenge, necessitating improved diagnostic tools. This study investigated the use of self-supervised multi-task learning for automated echocardiographic analysis, aiming to predict echocardiographic views, diagnose RHD conditions, and determine severity. We compared two prominent self-supervised learning (SSL) methods: DINOv2, a vision-transformer-based approach known for capturing implicit features, and simple contrastive learning representation (SimCLR), a ResNet-based contrastive learning method recognised for its simplicity and effectiveness. Both models were pre-trained on a large, unlabelled echocardiogram dataset and fine-tuned on a smaller, labelled subset. DINOv2 achieved accuracies of 92% for view classification, 98% for condition detection, and 99% for severity assessment. SimCLR demonstrated good performance as well, achieving accuracies of 99% for view classification, 92% for condition detection, and 96% for severity assessment. Embedding visualisations, using both Uniform Manifold Approximation Projection (UMAP) and t-distributed Stochastic Neighbor Embedding (t-SNE), revealed distinct clusters for all tasks in both models, indicating the effective capture of the discriminative features of the echocardiograms. This study demonstrates the potential of using self-supervised multi-task learning for automated echocardiogram analysis, offering a scalable and efficient approach to improving RHD diagnosis, especially in resource-limited settings.

Automated Fast Prediction of Bone Mineral Density From Low-dose Computed Tomography

BACKGROUND

Low-dose chest CT (LDCT) is commonly employed for the early screening of lung cancer. However, it has rarely been utilized in the assessment of volumetric bone mineral density (vBMD) and the diagnosis of osteoporosis (OP).

PURPOSE

This study investigated the feasibility of using deep learning to establish a system for vBMD prediction and OP classification based on LDCT scans.

METHODS

This study included 551 subjects who underwent both LDCT and QCT examinations. First, the U-net was developed to automatically segment lumbar vertebrae from single 2D LDCT slices near the mid-vertebral level. Then, a prediction model was proposed to estimate vBMD, which was subsequently employed for detecting OP and osteopenia (OA). Specifically, two input modalities were constructed for the prediction model. The performance metrics of the models were calculated and evaluated.

RESULTS

The segmentation model exhibited a strong correlation with manual segmentation, achieving a mean Dice similarity coefficient (DSC) of 0.974, sensitivity of 0.964, positive predictive value (PPV) of 0.985, and Hausdorff distance of 3.261 in the test set. Linear regression and Bland-Altman analysis demonstrated strong agreement between the predicted vBMD from two-channel inputs and QCT-derived vBMD, with a root mean square error of 8.958 mg/mm3 and an R2 of 0.944. The areas under the curve for detecting OP and OA were 0.800 and 0.878, respectively, with an overall accuracy of 94.2%. The average processing time for this system was 1.5 s.

CONCLUSION

This prediction system could automatically estimate vBMD and detect OP and OA on LDCT scans, providing great potential for the osteoporosis screening.

DDEvENet: Evidence-based ensemble learning for uncertainty-aware brain parcellation using diffusion MRI

In this study, we developed an Evidential Ensemble Neural Network based on Deep learning and Diffusion MRI, namely DDEvENet, for anatomical brain parcellation. The key innovation of DDEvENet is the design of an evidential deep learning framework to quantify predictive uncertainty at each voxel during a single inference. To do so, we design an evidence-based ensemble learning framework for uncertainty-aware parcellation to leverage the multiple dMRI parameters derived from diffusion MRI. Using DDEvENet, we obtained accurate parcellation and uncertainty estimates across different datasets from healthy and clinical populations and with different imaging acquisitions. The overall network includes five parallel subnetworks, where each is dedicated to learning the FreeSurfer parcellation for a certain diffusion MRI parameter. An evidence-based ensemble methodology is then proposed to fuse the individual outputs. We perform experimental evaluations on large-scale datasets from multiple imaging sources, including high-quality diffusion MRI data from healthy adults and clinically diffusion MRI data from participants with various brain diseases (schizophrenia, bipolar disorder, attention-deficit/hyperactivity disorder, Parkinson's disease, cerebral small vessel disease, and neurosurgical patients with brain tumors). Compared to several state-of-the-art methods, our experimental results demonstrate highly improved parcellation accuracy across the multiple testing datasets despite the differences in dMRI acquisition protocols and health conditions. Furthermore, thanks to the uncertainty estimation, our DDEvENet approach demonstrates a good ability to detect abnormal brain regions in patients with lesions that are consistent with expert-drawn results, enhancing the interpretability and reliability of the segmentation results.

Clinical Application of Artificial Intelligence in Digital Breast Tomosynthesis

Digital breast tomosynthesis (DBT) provides improved cancer detection and lower recall rates when compared with full-field digital mammography (DM) and has been widely adopted for breast cancer screening. However, adopting DBT presents new challenges such as an increased number of acquired images resulting in longer interpretation times. Artificial intelligence (AI) offers numerous opportunities to enhance the advantages of DBT and mitigate its shortcomings. Research in the DBT AI domain has grown significantly and AI algorithms play a key role in the screening and diagnostic phases of breast cancer detection and characterization. The application of AI may streamline the workflow and reduce the time required for radiologists to interpret images. In addition, AI can minimize radiation exposure and enhance lesion visibility in synthetic two-dimensional DM images. This review provides an overview of AI technology in DBT, its clinical applications, and future considerations.

Breast cancer classification based on breast tissue structures using the Jigsaw puzzle task in self-supervised learning

Self-supervised learning (SSL) has gained attention in the medical field as a deep learning approach utilizing unlabeled data. The Jigsaw puzzle task in SSL enables models to learn both features of images and the positional relationships within images. In breast cancer diagnosis, radiologists evaluate not only lesion-specific features but also the surrounding breast structures. However, deep learning models that adopt a diagnostic approach similar to human radiologists are still limited. This study aims to evaluate the effectiveness of the Jigsaw puzzle task in characterizing breast tissue structures for breast cancer classification on mammographic images. Using the Chinese Mammography Database (CMMD), we compared four pre-training pipelines: (1) IN-Jig, pre-trained with both the ImageNet classification task and the Jigsaw puzzle task, (2) Scratch-Jig, pre-trained only with the Jigsaw puzzle task, (3) IN, pre-trained only with the ImageNet classification task, and (4) Scratch, that is trained from random initialization without any pre-training tasks. All pipelines were fine-tuned using binary classification to distinguish between the presence or absence of breast cancer. Performance was evaluated based on the area under the receiver operating characteristic curve (AUC), sensitivity, and specificity. Additionally, detailed analysis was conducted for performance across different radiological findings, breast density, and regions of interest were visualized using gradient-weighted class activation mapping (Grad-CAM). The AUC for the four models were 0.925, 0.921, 0.918, 0.909, respectively. Our results suggest the Jigsaw puzzle task is an effective pre-training method for breast cancer classification, with the potential to enhance diagnostic accuracy with limited data.

Automatic medical imaging segmentation via self-supervising large-scale convolutional neural networks

PURPOSE

This study aims to develop a robust, large-scale deep learning model for medical image segmentation, leveraging self-supervised learning to overcome the limitations of supervised learning and data variability in clinical settings.

METHODS AND MATERIALS

We curated a substantial multi-center CT dataset for self-supervised pre-training using masked image modeling with sparse submanifold convolution. We designed a series of Sparse Submanifold U-Nets (SS-UNets) of varying sizes and performed self-supervised pre-training. We fine-tuned the SS-UNets on the TotalSegmentator dataset. The evaluation encompassed robustness tests on four unseen datasets and transferability assessments on three additional datasets.

RESULTS

Our SS-UNets exhibited superior performance in comparison to state-of-the-art self-supervised methods, demonstrating higher Dice Similarity Coefficient (DSC) and Surface Dice Coefficient (SDC) metrics. SS-UNet-B achieved 84.3 % DSC and 88.0 % SDC in TotalSegmentator. We further demonstrated the scalability of our networks, with segmentation performance increasing with model size, demonstrated from 58 million to 1.4 billion parameters:4.6 % DSC and 3.2 % SDC improvement in TotalSegmentator from SS-UNet-B to SS-UNet-H.

CONCLUSIONS

We demonstrate the efficacy of self-supervised learning for medical image segmentation in the CT, MRI and PET domains. Our approach significantly reduces reliance on extensively labeled data, mitigates risks of overfitting, and enhances model generalizability. Future applications may allow accurate segmentation of organs and lesions across several imaging domains, potentially streamlining cancer detection and radiotherapy treatment planning.

Robust Myocardial Perfusion MRI Quantification With DeepFermi

Stress perfusion cardiac magnetic resonance is an important technique for examining and assessing the blood supply of the myocardium. Currently, the majority of clinical perfusion scans are evaluated based on visual assessment by experienced clinicians. This makes the process subjective, and to this end, quantitative methods have been proposed to offer a more user-independent assessment of perfusion. These methods, however, rely on time-consuming deconvolution analysis and are susceptible to data outliers caused by artifacts due to cardiac or respiratory motion. In our work, we introduce a novel deep-learning method that integrates the commonly used Fermi function with a neural network architecture for fast, accurate, and robust myocardial perfusion quantification. This approach employs the Fermi model to ensure that the perfusion maps are consistent with measured data, while also utilizing a prior based on a 3D convolutional neural network to generalize spatio-temporal information across different patient data. Our network is trained within a self-supervised learning framework, which circumvents the need for ground-truth perfusion labels that are challenging to obtain. Furthermore, we extended this training methodology by adopting a technique that ensures estimations are resistant to data outliers, thereby improving robustness against motion artifacts. Our simulation experiments demonstrated an overall improvement in the accuracy and robustness of perfusion parameter estimation, consistently outperforming traditional deconvolution analysis algorithms across varying Signal-to-Noise Ratio scenarios in the presence of data outliers. For the in vivo studies, our method generated robust perfusion estimates that aligned with clinical diagnoses, while being approximately five times faster than conventional algorithms.

Support Vector Machine for Stratification of Cognitive Impairment Using 3D T1WI in Patients with Type 2 Diabetes Mellitus

Purpose

To explore the potential of MRI-based radiomics in predicting cognitive dysfunction in patients with diagnosed type 2 diabetes mellitus (T2DM).

Patients and Methods

In this study, data on 158 patients with T2DM were retrospectively collected between September 2019 and December 2020. The participants were categorized into a normal cognitive function (N) group (n=30), a mild cognitive impairment (MCI) group (n=90), and a dementia (DM) group (n=38) according to the Chinese version of the Montréal Cognitive Assessment Scale-B (MoCA-B). Radiomics features were extracted from the brain tissue except ventricles and sulci in the 3D T1WI images, support vector machine (SVM) model was then established to identify the CI and N groups, and the MCI and DM groups, respectively. The models were evaluated based on their area under the receiver operating characteristic curve (AUC), Precision (P), Recall rate (Recall, R), F1-score, and Support. Finally, ROC curves were plotted for each model.

Results

The study consisted of 68 cases in the N and CI group, with 54 cases in the training set and 14 in the verification set, and 128 cases were included in the MCI and DM groups, with 90 training sets and 38 verification sets. The consistency for inter-group and intra-group of radiomics features in two physicians were 0.86 and 0.90, respectively. After features selection, there were 11 optimal features to distinguish N and CI and 12 optimal features to MCI and DM. In the test set, the AUC for the SVM classifier was 0.857 and the accuracy was 0.830 in distinguishing CI and N, while AUC was 0.821 and the accuracy was 0.830 in distinguishing MCI and DM.

Conclusion

The SVM model based on MRI radiomics exhibits high efficacy in the diagnosis of cognitive dysfunction and evaluation of its severity among patients with T2DM.

Automatic cervical lymph nodes detection and segmentation in heterogeneous computed tomography images using deep transfer learning

To develop a deep learning model using transfer learning for automatic detection and segmentation of neck lymph nodes (LNs) in computed tomography (CT) images, the study included 11,013 annotated LNs with a short-axis diameter ≥ 3 mm from 626 head and neck cancer patients across four hospitals. The nnUNet model was used as a baseline, pre-trained on a large-scale head and neck dataset, and then fine-tuned with 4,729 LNs from hospital A for detection and segmentation. Validation was conducted on an internal testing cohort (ITC A) and three external testing cohorts (ETCs B, C, and D), with 1684 and 4600 LNs, respectively. Detection was evaluated via sensitivity, positive predictive value (PPV), and false positive rate per case (FP/vol), while segmentation was assessed using the Dice similarity coefficient (DSC) and Hausdorff distance (HD95). For detection, the sensitivity, PPV, and FP/vol in ITC A were 54.6%, 69.0%, and 3.4, respectively. In ETCs, the sensitivity ranged from 45.7% at 3.9 FP/vol to 63.5% at 5.8 FP/vol. Segmentation achieved a mean DSC of 0.72 in ITC A and 0.72 to 0.74 in ETCs, as well as a mean HD95 of 3.78 mm in ITC A and 2.73 mm to 2.85 mm in ETCs. No significant sensitivity difference was found between contrast-enhanced and unenhanced CT images (p = 0.502) or repeated CT images (p = 0.815) during adaptive radiotherapy. The model's segmentation accuracy was comparable to that of experienced oncologists. The model shows promise in automatically detecting and segmenting neck LNs in CT images, potentially reducing oncologists' segmentation workload.

Associations of Postencephalitic Epilepsy Using Multi-Contrast Whole Brain MRI: A Large Self-Supervised Vision Foundation Model Strategy

BACKGROUND

Postencephalitic epilepsy (PEE) is a severe neurological complication following encephalitis. Early identification of individuals at high risk for PEE is important for timely intervention.

PURPOSE

To develop a large self-supervised vision foundation model using a big dataset of multi-contrast head MRI scans, followed by fine-tuning with MRI data and follow-up outcomes from patients with PEE to develop a PEE association model.

STUDY TYPE

Retrospective.

POPULATION

Fifty-seven thousand six hundred twenty-one contrast-enhanced head MRI scans from 34,871 patients for foundation model construction, and head MRI scans from 144 patients with encephalitis (64 PEE, 80 N-PEE) for the PEE association model.

FIELD STRENGTH/SEQUENCE

1.5-T, 3-T, T1-weighted imaging, T2-weighted imaging, fluid attenuated inversion recovery, T1-weighted contrast-enhanced imaging.

ASSESSMENT

The foundation model was developed using self-supervised learning and cross-contrast context recovery. Patients with encephalitis were monitored for a median of 3.7 years (range 0.7-7.5 years), with epilepsy diagnosed according to International League Against Epilepsy. Occlusion sensitivity mapping highlighted brain regions involved in PEE classifications. Model performance was compared with DenseNet without pre-trained weights.

STATISTICAL TESTS

Performance was assessed via confusion matrices, accuracy, sensitivity, specificity, precision, F1 score, and area under the receiver operating characteristic curve (AUC). The DeLong test evaluated AUC between the two models (P < 0.05 for statistical significance).

RESULTS

The PEE association model achieved accuracy, sensitivity, specificity, precision, F1 score, and AUC of 79.3% (95% CI: 0.71-0.92), 92.3% (95% CI: 0.80-1.00), 68.8% (95% CI: 0.55-0.87), 70.6% (95% CI: 0.61-0.90), 80.0% (95% CI: 0.71-0.93), and 81.0% (95% CI: 0.68-0.92), respectively. A significant AUC improvement was found compared to DenseNet (Delong test, P = 0.03). The association model focused on brain regions affected by encephalitis.

DATA CONCLUSION

Using extensive unlabeled data via self-supervised learning addressed the limitations of supervised tasks with limited data. The fine-tuned foundation model outperformed DenseNet, which was trained exclusively on task data.

PLAIN LANGUAGE SUMMARY

This research develops a model to assess the occurrence epilepsy after encephalitis, a severe brain inflammation condition. By using over 57,000 brain scans, the study trains a computer program to recognize patterns in brain images. The model analyzes whole-brain scans to identify areas commonly affected by the disease, such as the temporal and frontal lobes. It was tested on data from patients with encephalitis and showed better performance than older methods. The model can assess the risk of secondary epilepsy in patients with encephalitis, allowing doctors to intervene early and improve treatment outcomes for those affected by this condition.

EVIDENCE LEVEL

4 TECHNICAL EFFICACY: Stage 1.

Foundation Models in Radiology: What, How, Why, and Why Not

Recent advances in artificial intelligence have witnessed the emergence of large-scale deep learning models capable of interpreting and generating both textual and imaging data. Such models, typically referred to as foundation models (FMs), are trained on extensive corpora of unlabeled data and demonstrate high performance across various tasks. FMs have recently received extensive attention from academic, industry, and regulatory bodies. Given the potentially transformative impact that FMs can have on the field of radiology, radiologists must be aware of potential pathways to train these radiology-specific FMs, including understanding both the benefits and challenges. Thus, this review aims to explain the fundamental concepts and terms of FMs in radiology, with a specific focus on the requirements of training data, model training paradigms, model capabilities, and evaluation strategies. Overall, the goal of this review is to unify technical advances and clinical needs for safe and responsible training of FMs in radiology to ultimately benefit patients, providers, and radiologists.

A Comprehensive Survey of Deep Learning Approaches in Image Processing

The integration of deep learning (DL) into image processing has driven transformative advancements, enabling capabilities far beyond the reach of traditional methodologies. This survey offers an in-depth exploration of the DL approaches that have redefined image processing, tracing their evolution from early innovations to the latest state-of-the-art developments. It also analyzes the progression of architectural designs and learning paradigms that have significantly enhanced the ability to process and interpret complex visual data. Key advancements, such as techniques improving model efficiency, generalization, and robustness, are examined, showcasing DL's ability to address increasingly sophisticated image-processing tasks across diverse domains. Metrics used for rigorous model evaluation are also discussed, underscoring the importance of performance assessment in varied application contexts. The impact of DL in image processing is highlighted through its ability to tackle complex challenges and generate actionable insights. Finally, this survey identifies potential future directions, including the integration of emerging technologies like quantum computing and neuromorphic architectures for enhanced efficiency and federated learning for privacy-preserving training. Additionally, it highlights the potential of combining DL with emerging technologies such as edge computing and explainable artificial intelligence (AI) to address scalability and interpretability challenges. These advancements are positioned to further extend the capabilities and applications of DL, driving innovation in image processing.

CXR-LLaVA: a multimodal large language model for interpreting chest X-ray images

OBJECTIVE

This study aimed to develop an open-source multimodal large language model (CXR-LLaVA) for interpreting chest X-ray images (CXRs), leveraging recent advances in large language models (LLMs) to potentially replicate the image interpretation skills of human radiologists.

MATERIALS AND METHODS

For training, we collected 592,580 publicly available CXRs, of which 374,881 had labels for certain radiographic abnormalities (Dataset 1) and 217,699 provided free-text radiology reports (Dataset 2). After pre-training a vision transformer with Dataset 1, we integrated it with an LLM influenced by the LLaVA network. Then, the model was fine-tuned, primarily using Dataset 2. The model's diagnostic performance for major pathological findings was evaluated, along with the acceptability of radiologic reports by human radiologists, to gauge its potential for autonomous reporting.

RESULTS

The model demonstrated impressive performance in test sets, achieving an average F1 score of 0.81 for six major pathological findings in the MIMIC internal test set and 0.56 for six major pathological findings in the external test set. The model's F1 scores surpassed those of GPT-4-vision and Gemini-Pro-Vision in both test sets. In human radiologist evaluations of the external test set, the model achieved a 72.7% success rate in autonomous reporting, slightly below the 84.0% rate of ground truth reports.

CONCLUSION

This study highlights the significant potential of multimodal LLMs for CXR interpretation, while also acknowledging the performance limitations. Despite these challenges, we believe that making our model open-source will catalyze further research, expanding its effectiveness and applicability in various clinical contexts.

KEY POINTS

Question How can a multimodal large language model be adapted to interpret chest X-rays and generate radiologic reports? Findings The developed CXR-LLaVA model effectively detects major pathological findings in chest X-rays and generates radiologic reports with a higher accuracy compared to general-purpose models. Clinical relevance This study demonstrates the potential of multimodal large language models to support radiologists by autonomously generating chest X-ray reports, potentially reducing diagnostic workloads and improving radiologist efficiency.

A Foundation Language-Image Model of the Retina (FLAIR): encoding expert knowledge in text supervision

Foundation vision-language models are currently transforming computer vision, and are on the rise in medical imaging fueled by their very promising generalization capabilities. However, the initial attempts to transfer this new paradigm to medical imaging have shown less impressive performances than those observed in other domains, due to the significant domain shift and the complex, expert domain knowledge inherent to medical-imaging tasks. Motivated by the need for domain-expert foundation models, we present FLAIR, a pre-trained vision-language model for universal retinal fundus image understanding. To this end, we compiled 38 open-access, mostly categorical fundus imaging datasets from various sources, with up to 101 different target conditions and 288,307 images. We integrate the expert's domain knowledge in the form of descriptive textual prompts, during both pre-training and zero-shot inference, enhancing the less-informative categorical supervision of the data. Such a textual expert's knowledge, which we compiled from the relevant clinical literature and community standards, describes the fine-grained features of the pathologies as well as the hierarchies and dependencies between them. We report comprehensive evaluations, which illustrate the benefit of integrating expert knowledge and the strong generalization capabilities of FLAIR under difficult scenarios with domain shifts or unseen categories. When adapted with a lightweight linear probe, FLAIR outperforms fully-trained, dataset-focused models, more so in the few-shot regimes. Interestingly, FLAIR outperforms by a wide margin larger-scale generalist image-language models and retina domain-specific self-supervised networks, which emphasizes the potential of embedding experts' domain knowledge and the limitations of generalist models in medical imaging. The pre-trained model is available at: https://github.com/jusiro/FLAIR.

O-PRESS: Boosting OCT axial resolution with Prior guidance, Recurrence, and Equivariant Self-Supervision

Optical coherence tomography (OCT) is a noninvasive technology that enables real-time imaging of tissue microanatomies. The axial resolution of OCT is intrinsically constrained by the spectral bandwidth of the employed light source while maintaining a fixed center wavelength for a specific application. Physically extending this bandwidth faces strong limitations and requires a substantial cost. We present a novel computational approach, called as O-PRESS, for boosting the axial resolution of OCT with Prior guidance, a Recurrent mechanism, and Equivariant Self-Supervision. Diverging from conventional deconvolution methods that rely on physical models or data-driven techniques, our method seamlessly integrates OCT modeling and deep learning, enabling us to achieve real-time axial-resolution enhancement exclusively from measurements without a need for paired images. Our approach solves two primary tasks of resolution enhancement and noise reduction with one treatment. Both tasks are executed in a self-supervised manner, with equivariance imaging and free space priors guiding their respective processes. Experimental evaluations, encompassing both quantitative metrics and visual assessments, consistently verify the efficacy and superiority of our approach, which exhibits performance on par with fully supervised methods. Importantly, the robustness of our model is affirmed, showcasing its dual capability to enhance axial resolution while concurrently improving the signal-to-noise ratio.

Using the Regression Slope of Training Loss to Optimize Chest X-ray Generation in Deep Convolutional Generative Adversarial Networks

Diffusion models, variational autoencoders, and generative adversarial networks (GANs) are three common types of generative artificial intelligence models for image generation. Among these, GANs are the most frequently used for medical image generation and are often employed for data augmentation in various studies. However, due to the adversarial nature of GANs, where the generator and discriminator compete against each other, the training process can sometimes end with the model unable to generate meaningful images or even producing noise. This phenomenon is rarely discussed in the literature, and no studies have proposed solutions to address this issue. Such outcomes can introduce significant bias when GANs are used for data augmentation in medical image training. Moreover, GANs often require substantial computational power and storage, adding to the challenges. In this study, we used deep convolutional GANs for chest X-ray generation, and three typical training outcomes were found. Two scenarios generated meaningful medical images and one failed to produce usable images. By analyzing the loss history during training, we observed that the regression line of the overall losses tends to diverge slowly. After excluding outlier losses, we found that the slope of the regression line within the stable loss segment indicates the optimal point to terminate training, ensuring the generation of meaningful medical images.

Improving coronary artery segmentation with self-supervised learning and automated pericoronary adipose tissue segmentation: a multi-institutional study on coronary computed tomography angiography images

Purpose

Coronary artery disease (CAD) is a leading cause of morbidity and mortality worldwide, with coronary computed tomography angiography (CCTA) playing a crucial role in its diagnosis. The mean Hounsfield unit (HU) of pericoronary adipose tissue (PCAT) is linked to cardiovascular risk. We utilized a self-supervised learning framework (SSL) to improve the accuracy and generalizability of coronary artery segmentation on CCTA volumes while addressing the limitations of small-annotated datasets.

Approach

We utilized self-supervised pretraining followed by supervised fine-tuning to segment coronary arteries. To evaluate the data efficiency of SSL, we varied the number of CCTA volumes used during pretraining. In addition, we developed an automated PCAT segmentation algorithm utilizing centerline extraction, spatial-geometric coronary identification, and landmark detection. We evaluated our method on a multi-institutional dataset by assessing coronary artery and PCAT segmentation accuracy via Dice scores and comparing mean PCAT HU values with the ground truth.

Results

Our approach significantly improved coronary artery segmentation, achieving Dice scores up to 0.787 after self-supervised pretraining. The automated PCAT segmentation achieved near-perfect performance, with R -squared values of 0.9998 for both the left anterior descending artery and the right coronary artery indicating excellent agreement between predicted and actual mean PCAT HU values. Self-supervised pretraining notably enhanced model generalizability on external datasets, improving overall segmentation accuracy.

Conclusions

We demonstrate the potential of SSL to advance CCTA image analysis, enabling more accurate CAD diagnostics. Our findings highlight the robustness of SSL for automated coronary artery and PCAT segmentation, offering promising advancements in cardiovascular care.

Partial Attention in Global Context and Local Interaction for Addressing Noisy Labels and Weighted Redundancies on Medical Images

Recently, the application of deep neural networks to detect anomalies on medical images has been facing the appearance of noisy labels, including overlapping objects and similar classes. Therefore, this study aims to address this challenge by proposing a unique attention module that can assist deep neural networks in focusing on important object features in noisy medical image conditions. This module integrates global context modeling to create long-range dependencies and local interactions to enable channel attention ability by using 1D convolution that not only performs well with noisy labels but also consumes significantly less resources without any dimensionality reduction. The module is then named Global Context and Local Interaction (GCLI). We have further experimented and proposed a partial attention strategy for the proposed GCLI module, aiming to efficiently reduce weighted redundancies. This strategy utilizes a subset of channels for GCLI to produce attention weights instead of considering every single channel. As a result, this strategy can greatly reduce the risk of introducing weighted redundancies caused by modeling global context. For classification, our proposed method is able to assist ResNet34 in achieving up to 82.5% accuracy on the Chaoyang test set, which is the highest figure among the other SOTA attention modules without using any processing filter to reduce the effect of noisy labels. For object detection, the GCLI is able to boost the capability of YOLOv8 up to 52.1% mAP50 on the GRAZPEDWRI-DX test set, demonstrating the highest performance among other attention modules and ranking second in the mAP50 metric on the VinDR-CXR test set. In terms of model complexity, our proposed GCLI module can consume fewer extra parameters up to 225 times and has inference speed faster than 30% compared to the other attention modules.

Applications of Artificial Intelligence in Gastrointestinal Endoscopic Ultrasound: Current Developments, Limitations and Future Directions

Endoscopic ultrasound (EUS) effectively diagnoses malignant and pre-malignant gastrointestinal lesions. In the past few years, artificial intelligence (AI) has shown promising results in enhancing EUS sensitivity and accuracy, particularly for subepithelial lesions (SELs) like gastrointestinal stromal tumors (GISTs). Furthermore, AI models have shown high accuracy in predicting malignancy in gastric GISTs and distinguishing between benign and malignant intraductal papillary mucinous neoplasms (IPMNs). The utility of AI has also been applied to existing and emerging technologies involved in the performance and evaluation of EUS-guided biopsies. These advancements may improve training in EUS, allowing trainees to focus on technical skills and image interpretation. This review evaluates the current state of AI in EUS, covering imaging diagnosis, EUS-guided biopsies, and training advancements. It discusses early feasibility studies and recent developments, while also addressing the limitations and challenges. This article aims to review AI applications to EUS and its applications in clinical practice while addressing pitfalls and challenges.

Cervical OCT image classification using contrastive masked autoencoders with Swin Transformer

BACKGROUND AND OBJECTIVE

Cervical cancer poses a major health threat to women globally. Optical coherence tomography (OCT) imaging has recently shown promise for non-invasive cervical lesion diagnosis. However, obtaining high-quality labeled cervical OCT images is challenging and time-consuming as they must correspond precisely with pathological results. The scarcity of such high-quality labeled data hinders the application of supervised deep-learning models in practical clinical settings. This study addresses the above challenge by proposing CMSwin, a novel self-supervised learning (SSL) framework combining masked image modeling (MIM) with contrastive learning based on the Swin-Transformer architecture to utilize abundant unlabeled cervical OCT images.

METHODS

In this contrastive-MIM framework, mixed image encoding is combined with a latent contextual regressor to solve the inconsistency problem between pre-training and fine-tuning and separate the encoder's feature extraction task from the decoder's reconstruction task, allowing the encoder to extract better image representations. Besides, contrastive losses at the patch and image levels are elaborately designed to leverage massive unlabeled data.

RESULTS

We validated the superiority of CMSwin over the state-of-the-art SSL approaches with five-fold cross-validation on an OCT image dataset containing 1,452 patients from a multi-center clinical study in China, plus two external validation sets from top-ranked Chinese hospitals: the Huaxi dataset from the West China Hospital of Sichuan University and the Xiangya dataset from the Xiangya Second Hospital of Central South University. A human-machine comparison experiment on the Huaxi and Xiangya datasets for volume-level binary classification also indicates that CMSwin can match or exceed the average level of four skilled medical experts, especially in identifying high-risk cervical lesions.

CONCLUSION

Our work has great potential to assist gynecologists in intelligently interpreting cervical OCT images in clinical settings. Additionally, the integrated GradCAM module of CMSwin enables cervical lesion visualization and interpretation, providing good interpretability for gynecologists to diagnose cervical diseases efficiently.

Less is More: Selective reduction of CT data for self-supervised pre-training of deep learning models with contrastive learning improves downstream classification performance

BACKGROUND

Self-supervised pre-training of deep learning models with contrastive learning is a widely used technique in image analysis. Current findings indicate a strong potential for contrastive pre-training on medical images. However, further research is necessary to incorporate the particular characteristics of these images.

METHOD

We hypothesize that the similarity of medical images hinders the success of contrastive learning in the medical imaging domain. To this end, we investigate different strategies based on deep embedding, information theory, and hashing in order to identify and reduce redundancy in medical pre-training datasets. The effect of these different reduction strategies on contrastive learning is evaluated on two pre-training datasets and several downstream classification tasks.

RESULTS

In all of our experiments, dataset reduction leads to a considerable performance gain in downstream tasks, e.g., an AUC score improvement from 0.78 to 0.83 for the COVID CT Classification Grand Challenge, 0.97 to 0.98 for the OrganSMNIST Classification Challenge and 0.73 to 0.83 for a brain hemorrhage classification task. Furthermore, pre-training is up to nine times faster due to the dataset reduction.

CONCLUSIONS

In conclusion, the proposed approach highlights the importance of dataset quality and provides a transferable approach to improve contrastive pre-training for classification downstream tasks on medical images.

Enhancing dental caries classification in CBCT images by using image processing and self-supervised learning

Diagnosing dental caries poses a significant challenge in dentistry, necessitating precise and early detection for effective management. This study utilizes Self-Supervised Learning (SSL) tasks to improve the classification of dental caries in Cone Beam Computed Tomography (CBCT) images, employing the International Caries Detection and Assessment System (ICDAS). Faced with the challenge of scarce annotated medical images, our research employs SSL to utilize unlabeled data, thereby improving model performance. We have developed a pipeline incorporating unlabeled data extraction from CBCT exams and subsequent model training using SSL tasks. A distinctive aspect of our approach is the integration of image processing techniques with SSL tasks, along with exploring the necessity for unlabeled data. Our research aims to identify the most effective image processing techniques for data extraction, the most efficient deep learning architectures for caries classification, the impact of unlabeled dataset sizes on model performance, and the comparative effectiveness of different SSL approaches in this domain. Among the tested architectures, ResNet-18, combined with the SimCLR task, demonstrated an average F1-score macro of 88.42%, Precision macro of 90.44%, and Sensitivity macro of 86.67%, reaching a 5.5% increase in F1-score compared to models using only deep learning architecture. These results suggest that SSL can significantly enhance the accuracy and efficiency of caries classification in CBCT images.

Effective automatic classification methods via deep learning for myopic maculopathy

Background

Pathologic myopia (PM) associated with myopic maculopathy (MM) is a significant cause of visual impairment, especially in East Asia, where its prevalence has surged. Early detection and accurate classification of myopia-related fundus lesions are critical for managing PM. Traditional clinical analysis of fundus images is time-consuming and dependent on specialist expertise, driving the need for automated, accurate diagnostic tools.

Methods

This study developed a deep learning-based system for classifying five types of MM using color fundus photographs. Five architectures-ResNet50, EfficientNet-B0, Vision Transformer (ViT), Contrastive Language-Image Pre-Training (CLIP), and RETFound-were utilized. An ensemble learning approach with weighted voting was employed to enhance model performance. The models were trained on a dataset of 2,159 annotated images from Shenzhen Eye Hospital, with performance evaluated using accuracy, sensitivity, specificity, F1-Score, Cohen's Kappa, and area under the receiver operating characteristic curve (AUC).

Results

The ensemble model achieved superior performance across all metrics, with an accuracy of 95.4% (95% CI: 93.0-97.0%), sensitivity of 95.4% (95% CI: 86.8-97.5%), specificity of 98.9% (95% CI: 97.1-99.5%), F1-Score of 95.3% (95% CI: 93.2-97.2%), Kappa value of 0.976 (95% CI: 0.957-0.989), and AUC of 0.995 (95% CI: 0.992-0.998). The voting ensemble method demonstrated robustness and high generalization ability in classifying complex lesions, outperforming individual models.

Conclusion

The ensemble deep learning system significantly enhances the accuracy and reliability of MM classification. This system holds potential for assisting ophthalmologists in early detection and precise diagnosis, thereby improving patient outcomes. Future work could focus on expanding the dataset, incorporating image quality assessment, and optimizing the ensemble algorithm for better efficiency and broader applicability.

Artificial intelligence tools trained on human-labeled data reflect human biases: a case study in a large clinical consecutive knee osteoarthritis cohort

Humans have been shown to have biases when reading medical images, raising questions about whether humans are uniform in their disease gradings. Artificial intelligence (AI) tools trained on human-labeled data may have inherent human non-uniformity. In this study, we used a radiographic knee osteoarthritis external validation dataset of 50 patients and a six-year retrospective consecutive clinical cohort of 8,273 patients. An FDA-approved and CE-marked AI tool was tested for potential non-uniformity in Kellgren-Lawrence grades between the right and left sides of the images. We flipped the images horizontally so that a left knee looked like a right knee and vice versa. According to human review, the AI tool showed non-uniformity with 20-22% disagreements on the external validation dataset and 13.6% on the cohort. However, we found no evidence of a significant difference in the accuracy compared to senior radiologists on the external validation dataset, or age bias or sex bias on the cohort. AI non-uniformity can boost the evaluated performance against humans, but image areas with inferior performance should be investigated.

Multi-modal representation learning in retinal imaging using self-supervised learning for enhanced clinical predictions

Self-supervised learning has become the cornerstone of building generalizable and transferable artificial intelligence systems in medical imaging. In particular, contrastive representation learning techniques trained on large multi-modal datasets have demonstrated impressive capabilities of producing highly transferable representations for different downstream tasks. In ophthalmology, large multi-modal datasets are abundantly available and conveniently accessible as modern retinal imaging scanners acquire both 2D fundus images and 3D optical coherence tomography (OCT) scans to assess the eye. In this context, we introduce a novel multi-modal contrastive learning-based pipeline to facilitate learning joint representations for the two retinal imaging modalities. After self-supervised pre-training on 153,306 scan pairs, we show that such a pre-training framework can provide both a retrieval system and encoders that produce comprehensive OCT and fundus image representations that generalize well for various downstream tasks on three independent external datasets, explicitly focusing on clinically pertinent prediction tasks. In addition, we show that interchanging OCT with lower-cost fundus imaging can preserve the predictive power of the trained models.

Identification of veterinary and medically important blood parasites using contrastive loss-based self-supervised learning

Background and Aim

Zoonotic diseases caused by various blood parasites are important public health concerns that impact animals and humans worldwide. The traditional method of microscopic examination for parasite diagnosis is labor-intensive, time-consuming, and prone to variability among observers, necessitating highly skilled and experienced personnel. Therefore, an innovative approach is required to enhance the conventional method. This study aimed to develop a self-supervised learning (SSL) approach to identify zoonotic blood parasites from microscopic images, with an initial focus on parasite species classification.

Materials and Methods

We acquired a public dataset featuring microscopic images of Giemsa-stained thin blood films of trypanosomes and other blood parasites, including Babesia, Leishmania, Plasmodium, Toxoplasma, and Trichomonad, as well as images of both white and red blood cells. The input data were subjected to SSL model training using the Bootstrap Your Own Latent (BYOL) algorithm with Residual Network 50 (ResNet50), ResNet101, and ResNet152 as the backbones. The performance of the proposed SSL model was then compared to that of baseline models.

Results

The proposed BYOL SSL model outperformed supervised learning models across all classes. Among the SSL models, ResNet50 consistently achieved high accuracy, reaching 0.992 in most classes, which aligns well with the patterns observed in the pre-trained uniform manifold approximation and projection representations. Fine-tuned SSL models exhibit high performance, achieving 95% accuracy and a 0.960 area under the curve of the receiver operating characteristics (ROC) curve even when fine-tuned with 1% of the data in the downstream process. Furthermore, 20% of the data for training with SSL models yielded ≥95% in all other statistical metrics, including accuracy, recall, precision, specification, F1 score, and ROC curve. As a result, multi-class classification prediction demonstrated that model performance exceeded 91% for the F1 score, except for the early stage of Trypanosoma evansi, which showed an F1 score of 87%. This may be due to the model being exposed to high levels of variation during the developmental stage.

Conclusion

This approach can significantly enhance active surveillance efforts to improve disease control and prevent outbreaks, particularly in resource-limited settings. In addition, SSL addresses significant challenges, such as data variability and the requirement for extensive class labeling, which are common in biology and medical fields.

Self-supervised learning framework application for medical image analysis: a review and summary

Manual annotation of medical image datasets is labor-intensive and prone to biases. Moreover, the rate at which image data accumulates significantly outpaces the speed of manual annotation, posing a challenge to the advancement of machine learning, particularly in the realm of supervised learning. Self-supervised learning is an emerging field that capitalizes on unlabeled data for training, thereby circumventing the need for extensive manual labeling. This learning paradigm generates synthetic pseudo-labels through pretext tasks, compelling the network to acquire image representations in a pseudo-supervised manner and subsequently fine-tuning with a limited set of annotated data to achieve enhanced performance. This review begins with an overview of prevalent types and advancements in self-supervised learning, followed by an exhaustive and systematic examination of methodologies within the medical imaging domain from 2018 to September 2024. The review encompasses a range of medical image modalities, including CT, MRI, X-ray, Histology, and Ultrasound. It addresses specific tasks, such as Classification, Localization, Segmentation, Reduction of False Positives, Improvement of Model Performance, and Enhancement of Image Quality. The analysis reveals a descending order in the volume of related studies, with CT and MRI leading the list, followed by X-ray, Histology, and Ultrasound. Except for CT and MRI, there is a greater prevalence of studies focusing on contrastive learning methods over generative learning approaches. The performance of MRI/Ultrasound classification and all image types segmentation still has room for further exploration. Generally, this review can provide conceptual guidance for medical professionals to combine self-supervised learning with their research.

Self-supervised learning for accurately modelling hierarchical evolutionary patterns of cerebrovasculature

Cerebrovascular abnormalities are critical indicators of stroke and neurodegenerative diseases like Alzheimer's disease (AD). Understanding the normal evolution of brain vessels is essential for detecting early deviations and enabling timely interventions. Here, for the first time, we proposed a pipeline exploring the joint evolution of cortical volumes (CVs) and arterial volumes (AVs) in a large cohort of 2841 individuals. Using advanced deep learning for vessel segmentation, we built normative models of CVs and AVs across spatially hierarchical brain regions. We found that while AVs generally decline with age, distinct trends appear in regions like the circle of Willis. Comparing healthy individuals with those affected by AD or stroke, we identified significant reductions in both CVs and AVs, wherein patients with AD showing the most severe impact. Our findings reveal gender-specific effects and provide critical insights into how these conditions alter brain structure, potentially guiding future clinical assessments and interventions.

A Semi-Supervised Learning Framework for Classifying Colorectal Neoplasia Based on the NICE Classification

Labelling medical images is an arduous and costly task that necessitates clinical expertise and large numbers of qualified images. Insufficient samples can lead to underfitting during training and poor performance of supervised learning models. In this study, we aim to develop a SimCLR-based semi-supervised learning framework to classify colorectal neoplasia based on the NICE classification. First, the proposed framework was trained under self-supervised learning using a large unlabelled dataset; subsequently, it was fine-tuned on a limited labelled dataset based on the NICE classification. The model was evaluated on an independent dataset and compared with models based on supervised transfer learning and endoscopists using accuracy, Matthew's correlation coefficient (MCC), and Cohen's kappa. Finally, Grad-CAM and t-SNE were applied to visualize the models' interpretations. A ResNet-backboned SimCLR model (accuracy of 0.908, MCC of 0.862, and Cohen's kappa of 0.896) outperformed supervised transfer learning-based models (means: 0.803, 0.698, and 0.742) and junior endoscopists (0.816, 0.724, and 0.863), while performing only slightly worse than senior endoscopists (0.916, 0.875, and 0.944). Moreover, t-SNE showed a better clustering of ternary samples through self-supervised learning in SimCLR than through supervised transfer learning. Compared with traditional supervised learning, semi-supervised learning enables deep learning models to achieve improved performance with limited labelled endoscopic images.

Medical image foundation models in assisting diagnosis of brain tumors: a pilot study

OBJECTIVES

To build self-supervised foundation models for multicontrast MRI of the whole brain and evaluate their efficacy in assisting diagnosis of brain tumors.

METHODS

In this retrospective study, foundation models were developed using 57,621 enhanced head MRI scans through self-supervised learning with a pretext task of cross-contrast context restoration with two different content dropout schemes. Downstream classifiers were constructed based on the pretrained foundation models and fine-tuned for brain tumor detection, discrimination, and molecular status prediction. Metrics including accuracy, sensitivity, specificity, and area under the ROC curve (AUC) were used to evaluate the performance. Convolutional neural networks trained exclusively on downstream task data were employed for comparative analysis.

RESULTS

The pretrained foundation models demonstrated their ability to extract effective representations from multicontrast whole-brain volumes. The best classifiers, endowed with pretrained weights, showed remarkable performance with accuracies of 94.9, 92.3, and 80.4%, and corresponding AUC values of 0.981, 0.972, and 0.852 on independent test datasets in brain tumor detection, discrimination, and molecular status prediction, respectively. The classifiers with pretrained weights outperformed the convolutional classifiers trained from scratch by approximately 10% in terms of accuracy and AUC across all tasks. The saliency regions in the correctly predicted cases are mainly clustered around the tumors. Classifiers derived from the two dropout schemes differed significantly only in the detection of brain tumors.

CONCLUSIONS

Foundation models obtained from self-supervised learning have demonstrated encouraging potential for scalability and interpretability in downstream brain tumor-related tasks and hold promise for extension to neurological diseases with diffusely distributed lesions.

CLINICAL RELEVANCE STATEMENT

The application of our proposed method to the prediction of key molecular status in gliomas is expected to improve treatment planning and patient outcomes. Additionally, the foundation model we developed could serve as a cornerstone for advancing AI applications in the diagnosis of brain-related diseases.

Applications of machine learning to MR imaging of pediatric low-grade gliomas

INTRODUCTION

Machine learning (ML) shows promise for the automation of routine tasks related to the treatment of pediatric low-grade gliomas (pLGG) such as tumor grading, typing, and segmentation. Moreover, it has been shown that ML can identify crucial information from medical images that is otherwise currently unattainable. For example, ML appears to be capable of preoperatively identifying the underlying genetic status of pLGG.

METHODS

In this chapter, we reviewed, to the best of our knowledge, all published works that have used ML techniques for the imaging-based evaluation of pLGGs. Additionally, we aimed to provide some context on what it will take to go from the exploratory studies we reviewed to clinically deployed models.

RESULTS

Multiple studies have demonstrated that ML can accurately grade, type, and segment and detect the genetic status of pLGGs. We compared the approaches used between the different studies and observed a high degree of variability throughout the methodologies. Standardization and cooperation between the numerous groups working on these approaches will be key to accelerating the clinical deployment of these models.

CONCLUSION

The studies reviewed in this chapter detail the potential for ML techniques to transform the treatment of pLGG. However, there are still challenges that need to be overcome prior to clinical deployment.

Ocular biomarkers: useful incidental findings by deep learning algorithms in fundus photographs

BACKGROUND/OBJECTIVES

Artificial intelligence can assist with ocular image analysis for screening and diagnosis, but it is not yet capable of autonomous full-spectrum screening. Hypothetically, false-positive results may have unrealized screening potential arising from signals persisting despite training and/or ambiguous signals such as from biomarker overlap or high comorbidity. The study aimed to explore the potential to detect clinically useful incidental ocular biomarkers by screening fundus photographs of hypertensive adults using diabetic deep learning algorithms.

SUBJECTS/METHODS

Patients referred for treatment-resistant hypertension were imaged at a hospital unit in Perth, Australia, between 2016 and 2022. The same 45° colour fundus photograph selected for each of the 433 participants imaged was processed by three deep learning algorithms. Two expert retinal specialists graded all false-positive results for diabetic retinopathy in non-diabetic participants.

RESULTS

Of the 29 non-diabetic participants misclassified as positive for diabetic retinopathy, 28 (97%) had clinically useful retinal biomarkers. The models designed to screen for fewer diseases captured more incidental disease. All three algorithms showed a positive correlation between severity of hypertensive retinopathy and misclassified diabetic retinopathy.

CONCLUSIONS

The results suggest that diabetic deep learning models may be responsive to hypertensive and other clinically useful retinal biomarkers within an at-risk, hypertensive cohort. Observing that models trained for fewer diseases captured more incidental pathology increases confidence in signalling hypotheses aligned with using self-supervised learning to develop autonomous comprehensive screening. Meanwhile, non-referable and false-positive outputs of other deep learning screening models could be explored for immediate clinical use in other populations.

Self-supervised learning for classifying paranasal anomalies in the maxillary sinus

PURPOSE

Paranasal anomalies, frequently identified in routine radiological screenings, exhibit diverse morphological characteristics. Due to the diversity of anomalies, supervised learning methods require large labelled dataset exhibiting diverse anomaly morphology. Self-supervised learning (SSL) can be used to learn representations from unlabelled data. However, there are no SSL methods designed for the downstream task of classifying paranasal anomalies in the maxillary sinus (MS).

METHODS

Our approach uses a 3D convolutional autoencoder (CAE) trained in an unsupervised anomaly detection (UAD) framework. Initially, we train the 3D CAE to reduce reconstruction errors when reconstructing normal maxillary sinus (MS) image. Then, this CAE is applied to an unlabelled dataset to generate coarse anomaly locations by creating residual MS images. Following this, a 3D convolutional neural network (CNN) reconstructs these residual images, which forms our SSL task. Lastly, we fine-tune the encoder part of the 3D CNN on a labelled dataset of normal and anomalous MS images.

RESULTS

The proposed SSL technique exhibits superior performance compared to existing generic self-supervised methods, especially in scenarios with limited annotated data. When trained on just 10% of the annotated dataset, our method achieves an area under the precision-recall curve (AUPRC) of 0.79 for the downstream classification task. This performance surpasses other methods, with BYOL attaining an AUPRC of 0.75, SimSiam at 0.74, SimCLR at 0.73 and masked autoencoding using SparK at 0.75.

CONCLUSION

A self-supervised learning approach that inherently focuses on localizing paranasal anomalies proves to be advantageous, particularly when the subsequent task involves differentiating normal from anomalous maxillary sinuses. Access our code at https://github.com/mtec-tuhh/self-supervised-paranasal-anomaly .

Self-supervised learning-enhanced deep learning method for identifying myopic maculopathy in high myopia patients

Accurate detection and timely care for patients with high myopia present significant challenges. We developed a deep learning (DL) system enhanced by a self-supervised learning (SSL) approach to improve the automatic diagnosis of myopic maculopathy (MM). Using a dataset of 7,906 images from the Shanghai High Myopia Screening Project and a public validation set of 1,391 images from MMAC2023, our method significantly outperformed conventional techniques. Internally, it achieved 96.8% accuracy, 83.1% sensitivity, and 95.6% specificity, with AUC values of 0.982 and 0.999. Externally, it maintained 89.0% accuracy, 71.7% sensitivity, and 87.8% specificity, with AUC values of 0.978 and 0.973. The model's Cohen's kappa values exceeded 0.8, indicating substantial agreement with retinal experts. Our SSL-enhanced DL approach offers high accuracy and potential to enhance large-scale myopia screenings, demonstrating broader significance in improving early detection and treatment of MM.

Evaluation of Effectiveness of Self-Supervised Learning in Chest X-Ray Imaging to Reduce Annotated Images

A significant challenge in machine learning-based medical image analysis is the scarcity of medical images. Obtaining a large number of labeled medical images is difficult because annotating medical images is a time-consuming process that requires specialized knowledge. In addition, inappropriate annotation processes can increase model bias. Self-supervised learning (SSL) is a type of unsupervised learning method that extracts image representations. Thus, SSL can be an effective method to reduce the number of labeled images. In this study, we investigated the feasibility of reducing the number of labeled images in a limited set of unlabeled medical images. The unlabeled chest X-ray (CXR) images were pretrained using the SimCLR framework, and then the representations were fine-tuned as supervised learning for the target task. A total of 2000 task-specific CXR images were used to perform binary classification of coronavirus disease 2019 (COVID-19) and normal cases. The results demonstrate that the performance of pretraining on task-specific unlabeled CXR images can be maintained when the number of labeled CXR images is reduced by approximately 40%. In addition, the performance was significantly better than that obtained without pretraining. In contrast, a large number of pretrained unlabeled images are required to maintain performance regardless of task specificity among a small number of labeled CXR images. In summary, to reduce the number of labeled images using SimCLR, we must consider both the number of images and the task-specific characteristics of the target images.

Radiology and multi-scale data integration for precision oncology

In this Perspective paper we explore the potential of integrating radiological imaging with other data types, a critical yet underdeveloped area in comparison to the fusion of other multi-omic data. Radiological images provide a comprehensive, three-dimensional view of cancer, capturing features that would be missed by biopsies or other data modalities. This paper explores the complexities and challenges of incorporating medical imaging into data integration models, in the context of precision oncology. We present the different categories of imaging-omics integration and discuss recent progress, highlighting the opportunities that arise from bringing together spatial data on different scales.

Wearable Data From Subjects Playing Super Mario, Taking University Exams, or Performing Physical Exercise Help Detect Acute Mood Disorder Episodes via Self-Supervised Learning: Prospective, Exploratory, Observational Study

BACKGROUND

Personal sensing, leveraging data passively and near-continuously collected with wearables from patients in their ecological environment, is a promising paradigm to monitor mood disorders (MDs), a major determinant of the worldwide disease burden. However, collecting and annotating wearable data is resource intensive. Studies of this kind can thus typically afford to recruit only a few dozen patients. This constitutes one of the major obstacles to applying modern supervised machine learning techniques to MD detection.

OBJECTIVE

In this paper, we overcame this data bottleneck and advanced the detection of acute MD episodes from wearables' data on the back of recent advances in self-supervised learning (SSL). This approach leverages unlabeled data to learn representations during pretraining, subsequently exploited for a supervised task.

METHODS

We collected open access data sets recording with the Empatica E4 wristband spanning different, unrelated to MD monitoring, personal sensing tasks-from emotion recognition in Super Mario players to stress detection in undergraduates-and devised a preprocessing pipeline performing on-/off-body detection, sleep/wake detection, segmentation, and (optionally) feature extraction. With 161 E4-recorded subjects, we introduced E4SelfLearning, the largest-to-date open access collection, and its preprocessing pipeline. We developed a novel E4-tailored transformer (E4mer) architecture, serving as the blueprint for both SSL and fully supervised learning; we assessed whether and under which conditions self-supervised pretraining led to an improvement over fully supervised baselines (ie, the fully supervised E4mer and pre-deep learning algorithms) in detecting acute MD episodes from recording segments taken in 64 (n=32, 50%, acute, n=32, 50%, stable) patients.

RESULTS

SSL significantly outperformed fully supervised pipelines using either our novel E4mer or extreme gradient boosting (XGBoost): n=3353 (81.23%) against n=3110 (75.35%; E4mer) and n=2973 (72.02%; XGBoost) correctly classified recording segments from a total of 4128 segments. SSL performance was strongly associated with the specific surrogate task used for pretraining, as well as with unlabeled data availability.

CONCLUSIONS

We showed that SSL, a paradigm where a model is pretrained on unlabeled data with no need for human annotations before deployment on the supervised target task of interest, helps overcome the annotation bottleneck; the choice of the pretraining surrogate task and the size of unlabeled data for pretraining are key determinants of SSL success. We introduced E4mer, which can be used for SSL, and shared the E4SelfLearning collection, along with its preprocessing pipeline, which can foster and expedite future research into SSL for personal sensing.

Contrastive Self-supervised Learning for Neurodegenerative Disorder Classification

Neurodegenerative diseases such as Alzheimer's disease (AD) or frontotemporal lobar degeneration (FTLD) involve specific loss of brain volume, detectable in vivo using T1-weighted MRI scans. Supervised machine learning approaches classifying neurodegenerative diseases require diagnostic-labels for each sample. However, it can be difficult to obtain expert labels for a large amount of data. Self-supervised learning (SSL) offers an alternative for training machine learning models without data-labels. We investigated if the SSL models can applied to distinguish between different neurodegenerative disorders in an interpretable manner. Our method comprises a feature extractor and a downstream classification head. A deep convolutional neural network trained in a contrastive self-supervised way serves as the feature extractor, learning latent representation, while the classifier head is a single-layer perceptron. We used N=2694 T1-weighted MRI scans from four data cohorts: two ADNI datasets, AIBL and FTLDNI, including cognitively normal controls (CN), cases with prodromal and clinical AD, as well as FTLD cases differentiated into its sub-types. Our results showed that the feature extractor trained in a self-supervised way provides generalizable and robust representations for the downstream classification. For AD vs. CN, our model achieves 82% balanced accuracy on the test subset and 80% on an independent holdout dataset. Similarly, the Behavioral variant of frontotemporal dementia (BV) vs. CN model attains an 88% balanced accuracy on the test subset. The average feature attribution heatmaps obtained by the Integrated Gradient method highlighted hallmark regions, i.e., temporal gray matter atrophy for AD, and insular atrophy for BV. In conclusion, our models perform comparably to state-of-the-art supervised deep learning approaches. This suggests that the SSL methodology can successfully make use of unannotated neuroimaging datasets as training data while remaining robust and interpretable.