Not-so-supervised: A survey of semi-supervised, multi-instance, and transfer learning in medical image analysis

A unified approach to medical image segmentation by leveraging mixed supervision and self and transfer learning (MIST)

Medical image segmentation is important for quantitative disease diagnosis and treatment but relies on accurate pixel-wise labels, which are costly, time-consuming, and require domain expertise. This work introduces MIST (MIxed supervision, Self, and Transfer learning) to reduce manual labeling in medical image segmentation. A small set of cases was manually annotated ("strong labels"), while the rest used automated, less accurate labels ("weak labels"). Both label types trained a dual-branch network with a shared encoder and two decoders. Self-training iteratively refined weak labels, and transfer learning reduced computational costs by freezing the encoder and fine-tuning the decoders. Applied to segmenting muscle, subcutaneous, and visceral adipose tissue, MIST used only 100 manually labeled slices from 20 CT scans to generate accurate labels for all slices of 102 internal scans, which were then used to train a 3D nnU-Net model. Using MIST to update weak labels significantly improved nnU-Net segmentation accuracy compared to training directly on strong and weak labels. Dice similarity coefficient (DSC) increased for muscle (89.2 ± 4.3% to 93.2 ± 2.1%), subcutaneous (75.1 ± 14.4% to 94.2 ± 2.8%), and visceral adipose tissue (66.6 ± 16.4% to 77.1 ± 19.0% ) on an internal dataset (p<.05). DSC improved for muscle (80.5 ± 6.9% to 86.6 ± 3.9%) and subcutaneous adipose tissue (61.8 ± 12.5% to 82.7 ± 11.1%) on an external dataset (p<.05). MIST reduced the annotation burden by 99%, enabling efficient, accurate pixel-wise labeling for medical image segmentation. Code is available at https://github.com/rsummers11/NIH_CADLab_Body_Composition.

Exploring the values underlying machine learning research in medical image analysis

Machine learning has emerged as a crucial tool for medical image analysis, largely due to recent developments in deep artificial neural networks addressing numerous, diverse clinical problems. As with any conceptual tool, the effective use of machine learning should be predicated on an understanding of its underlying motivations just as much as algorithms or theory - and to do so, we need to explore its philosophical foundations. One of these foundations is the understanding of how values, despite being non-empirical, nevertheless affect scientific research. This article has three goals: to introduce the reader to values in a way that is specific to medical image analysis; to characterise a particular set of technical decisions (what we call the end-to-end vs. separable learning spectrum) that are fundamental to machine learning for medical image analysis; and to create a simple and structured method to show how these values can be rigorously connected to these technical decisions. This better understanding of how the philosophy of science can clarify fundamental elements of how medical image analysis research is performed and can be improved.

Semantic segmentation dataset authoring with simplified labels

PURPOSE

Semantic segmentation of laparoscopic images is a key problem in surgical scene understanding. Creating ground truth labels for semantic segmentation tasks is time consuming, and in the medical field a need for medical training of annotators adds further complications, leading to reliance on a small pool of experts. Previous research has focused on reducing the time to author datasets, by using spatially weak labels, pseudolabels, and synthetic data. In this paper, we address the difficulties caused by the need for medically trained annotators, hoping to enable non-medical annotators to participate in medical annotation tasks, to ease the creation of large datasets.

METHODS

We propose simplified labels, labels that are semantically weak. Our labels allow non-medical annotators to participate in medical dataset authoring, by lowering the need for medical expertise. We simulate authoring processes with mixtures of medical and non-medical annotators and measure the impact adding non-medical annotators has on accuracy. We also show that simplified labels offer a simple formulation for multi-dataset training.

RESULTS

We show that simplified labels are a viable approach to dataset authoring. Including non-medical annotators in the authoring process is beneficial, but medically trained annotators are worth multiple non-medical annotators, with maximal Dice score increases of 9.3% for 1 medically trained annotator and 6.9% for 3 non-medical annotators. We also show that the labels offer a simple formulation for multi-dataset training, even with no overlapping classes. We find that converting the labels of a secondary incompatible dataset into simplified labels and jointly training on both datasets improves performance.

CONCLUSION

Simplified labels offer a framework that can be applied both to dataset authoring and to multi-dataset training. Using the proposed method, non-medical annotators can participate in semantic segmentation dataset authoring. Labels of incompatible datasets can be converted into simplified datasets, enabling multi-dataset training.

Retinal OCT image segmentation with deep learning: A review of advances, datasets, and evaluation metrics

Optical coherence tomography (OCT) is a widely used imaging technology in ophthalmic clinical practice, providing non-invasive access to high-resolution retinal images. Segmentation of anatomical structures and pathological lesions in retinal OCT images, directly impacts clinical decisions. While commercial OCT devices segment multiple retinal layers in healthy eyes, their performance degrades severely under pathological conditions. In recent years, the rapid advancements in deep learning have significantly driven research in OCT image segmentation. This review provides a comprehensive overview of the latest developments in deep learning-based segmentation methods for retinal OCT images. Additionally, it summarizes the medical significance, publicly available datasets, and commonly used evaluation metrics in this field. The review also discusses the current challenges faced by the research community and highlights potential future directions.

Benefit from public unlabeled data: A Frangi filter-based pretraining network for 3D cerebrovascular segmentation

Precise cerebrovascular segmentation in Time-of-Flight Magnetic Resonance Angiography (TOF-MRA) data is crucial for computer-aided clinical diagnosis. The sparse distribution of cerebrovascular structures within TOF-MRA images often results in high costs for manual data labeling. Leveraging unlabeled TOF-MRA data can significantly enhance model performance. In this study, we have constructed the largest preprocessed unlabeled TOF-MRA dataset to date, comprising 1510 subjects. Additionally, we provide manually annotated segmentation masks for 113 subjects based on existing external image datasets to facilitate evaluation. We propose a simple yet effective pretraining strategy utilizing the Frangi filter, known for its capability to enhance vessel-like structures, to optimize the use of the unlabeled data for 3D cerebrovascular segmentation. This involves a Frangi filter-based preprocessing workflow tailored for large-scale unlabeled datasets and a multi-task pretraining strategy to efficiently utilize the preprocessed data. This approach ensures maximal extraction of useful knowledge from the unlabeled data. The efficacy of the pretrained model is assessed across four cerebrovascular segmentation datasets, where it demonstrates superior performance, improving the clDice metric by approximately 2%-3% compared to the latest semi- and self-supervised methods. Additionally, ablation studies validate the generalizability and effectiveness of our pretraining method across various backbone structures. The code and data have been open source at: https://github.com/shigen-StoneRoot/FFPN.

Limitations of nomogram models in predicting survival outcomes for glioma patients

Purpose

Glioma represents a prevalent and malignant tumor of the central nervous system (CNS), and it is essential to accurately predict the survival of glioma patients to optimize their subsequent treatment plans. This review outlines the most recent advancements and viewpoints regarding the application of nomograms in glioma prognosis research.

Design

With an emphasis on the precision and external applicability of predictive models, we carried out a comprehensive review of the literature on the application of nomograms in glioma and provided a step-by-step guide for developing and evaluating nomograms.

Results

A summary of thirty-nine articles was produced. The majority of nomogram-building research has used limited patient samples, disregarded the proportional hazards (PH) assumption in Cox regression models, and some of them have failed to incorporate external validation. Furthermore, the predictive capability of nomograms is influenced by the selection of incorporated risk factors. Overall, the current predictive accuracy of nomograms is moderately credible.

Conclusion

The development and validation of nomogram models ought to adhere to a standardized set of criteria, thereby augmenting their worth in clinical decision-making and clinician-patient communication. Prior to the clinical application of a nomogram, it is imperative to thoroughly scrutinize its statistical foundation, rigorously evaluate its accuracy, and, whenever feasible, assess its external applicability utilizing multicenter databases.

A semi-supervised domain adaptive medical image segmentation method based on dual-level multi-scale alignment

In the actual image segmentation tasks in the medical field, the phenomenon of limited labeled data accompanied by domain shifts often occurs and such domain shifts may exist in homologous or even heterologous data. In the study, a novel method was proposed to deal with this challenging phenomenon. Firstly, a model was trained with labeled data in source and target domains so as to adapt to unlabeled data. Then, the alignment at two main levels was realized. At the style level, based on multi-scale stylistic features, the alignment of unlabeled target images was maximized and unlabeled target image features were enhanced. At the inter-domain level, the similarity of the category centroids between target domain data and mixed image data was also maximized. Additionally, a fused supervised loss and alignment loss computation method was proposed. In validation experiments, two cross-domain medical image datasets were constructed: homologous and heterologous datasets. Experimental results showed that the proposed method had the more advantageous comprehensive performance than common semi-supervised and domain adaptation methods.

Deep learning-based image analysis in muscle histopathology using photo-realistic synthetic data

BACKGROUND

Artificial intelligence (AI), specifically Deep learning (DL), has revolutionized biomedical image analysis, but its efficacy is limited by the need for representative, high-quality large datasets with manual annotations. While latest research on synthetic data using AI-based generative models has shown promising results to tackle this problem, several challenges such as lack of interpretability and need for vast amounts of real data remain. This study aims to introduce a new approach-SYNTA-for the generation of photo-realistic synthetic biomedical image data to address the challenges associated with state-of-the art generative models and DL-based image analysis.

METHODS

The SYNTA method employs a fully parametric approach to create photo-realistic synthetic training datasets tailored to specific biomedical tasks. Its applicability is tested in the context of muscle histopathology and skeletal muscle analysis. This new approach is evaluated for two real-world datasets to validate its applicability to solve complex image analysis tasks on real data.

RESULTS

Here we show that SYNTA enables expert-level segmentation of unseen real-world biomedical data using only synthetic training data. By addressing the lack of representative and high-quality real-world training data, SYNTA achieves robust performance in muscle histopathology image analysis, offering a scalable, controllable and interpretable alternative to generative models such as Generative Adversarial Networks (GANs) or Diffusion Models.

CONCLUSIONS

SYNTA demonstrates great potential to accelerate and improve biomedical image analysis. Its ability to generate high-quality photo-realistic synthetic data reduces reliance on extensive collection of data and manual annotations, paving the way for advancements in histopathology and medical research.

Artificial intelligence in musculoskeletal applications: a primer for radiologists

As an umbrella term, artificial intelligence (AI) covers machine learning and deep learning. This review aimed to elaborate on these terms to act as a primer for radiologists to learn more about the algorithms commonly used in musculoskeletal radiology. It also aimed to familiarize them with the common practices and issues in the use of AI in this domain.

Development of a deep learning radiomics model combining lumbar CT, multi-sequence MRI, and clinical data to predict high-risk cage subsidence after lumbar fusion: a retrospective multicenter study

BACKGROUND

To develop and validate a model that integrates clinical data, deep learning radiomics, and radiomic features to predict high-risk patients for cage subsidence (CS) after lumbar fusion.

METHODS

This study analyzed preoperative CT and MRI data from 305 patients undergoing lumbar fusion surgery from three centers. Using a deep learning model based on 3D vision transformations, the data were divided the dataset into training (n = 214), validation (n = 61), and test (n = 30) groups. Feature selection was performed using LASSO regression, followed by the development of a logistic regression model. The predictive ability of the model was assessed using various machine learning algorithms, and a combined clinical model was also established.

RESULTS

Ultimately, 11 traditional radiomic features, 5 deep learning radiomic features, and 1 clinical feature were selected. The combined model demonstrated strong predictive performance, with area under the curve (AUC) values of 0.941, 0.832, and 0.935 for the training, validation, and test groups, respectively. Notably, our model outperformed predictions made by two experienced surgeons.

CONCLUSIONS

This study developed a robust predictive model that integrates clinical features and imaging data to identify high-risk patients for CS following lumbar fusion. This model has the potential to improve clinical decision-making and reduce the need for revision surgeries, easing the burden on healthcare systems.

Patch-based feature mapping with generative adversarial networks for auxiliary hip fracture detection

BACKGROUND

Hip fractures are a significant public health issue, particularly among the elderly population. Pelvic radiographs (PXRs) play a crucial role in diagnosing hip fractures and are commonly used for their evaluation. Previous research has demonstrated promising performance in classification models for hip fracture detection. However, these models sometimes focus on the images' non-fracture regions, reducing their explainability. This study applies weakly supervised learning techniques to address this issue and improve the model's focus on the fracture region. Additionally, we introduce a method to quantitatively evaluate the model's focus on the region of interest (ROI).

METHODS

We propose a new auxiliary module called the patch-auxiliary generative adversarial network (PAGAN) for weakly supervised learning tasks. PAGAN can be integrated with any state-of-the-art (SOTA) classification model, such as EfficientNetB0, ResNet50, and DenseNet121, to enhance hip fracture detection. This training strategy incorporates global information (the entire PXR image) and local information (the hip region patch) for more effective learning. Furthermore, we employ GradCAM to generate attention heatmaps, highlighting the focus areas within the classification model. The intersection over union (IOU) and dice coefficient (Dise) are then computed between the attention heatmap and the fracture area, enabling a quantitative assessment of the model's explainability.

RESULTS AND CONCLUSIONS

Incorporating PAGAN improved the performance of the classification models. The accuracy of EfficientNetB0 increased from 93.61 % to 95.97 %, ResNet50 improved from 90.66 % to 94.89 %, and DenseNet121 saw an increase from 93.51 % to 94.49 %. Regarding model explainability, the integration of PAGAN into classification models led to a more pronounced attention to ROI. The average IOU improved from 0.32 to 0.54 for EfficientNetB0, from 0.28 to 0.40 for ResNet50, and from 0.37 to 0.51 for DenseNet121. These results indicate that PAGAN improves hip fracture classification performance and substantially enhances the model's focus on the fracture region, thereby increasing its explainability.

A comparative analysis of image harmonization techniques in mitigating differences in CT acquisition and reconstruction

Objective. The study aims to systematically characterize the effect of CT parameter variations on images and lung radiomic and deep features, and to evaluate the ability of different image harmonization methods to mitigate the observed variations.Approach. A retrospective in-house sinogram dataset of 100 low-dose chest CT scans was reconstructed by varying radiation dose (100%, 25%, 10%) and reconstruction kernels (smooth, medium, sharp). A set of image processing, convolutional neural network (CNNs), and generative adversarial network-based (GANs) methods were trained to harmonize all image conditions to a reference condition (100% dose, medium kernel). Harmonized scans were evaluated for image similarity using peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and learned perceptual image patch similarity (LPIPS), and for the reproducibility of radiomic and deep features using concordance correlation coefficient (CCC).Main Results. CNNs consistently yielded higher image similarity metrics amongst others; for Sharp/10%, which exhibited the poorest visual similarity, PSNR increased from a mean ± CI of 17.763 ± 0.492 to 31.925 ± 0.571, SSIM from 0.219 ± 0.009 to 0.754 ± 0.017, and LPIPS decreased from 0.490 ± 0.005 to 0.275 ± 0.016. Texture-based radiomic features exhibited a greater degree of variability across conditions, i.e. a CCC of 0.500 ± 0.332, compared to intensity-based features (0.972 ± 0.045). GANs achieved the highest CCC (0.969 ± 0.009 for radiomic and 0.841 ± 0.070 for deep features) amongst others. CNNs are suitable if downstream applications necessitate visual interpretation of images, whereas GANs are better alternatives for generating reproducible quantitative image features needed for machine learning applications.Significance. Understanding the efficacy of harmonization in addressing multi-parameter variability is crucial for optimizing diagnostic accuracy and a critical step toward building generalizable models suitable for clinical use.

Automatic maxillary sinus segmentation and age estimation model for the northwestern Chinese Han population

BACKGROUND

Age estimation is vital in forensic science, with maxillary sinus development serving as a reliable indicator. This study developed an automatic segmentation model for maxillary sinus identification and parameter measurement, combined with regression and machine learning models for age estimation.

METHODS

Cone Beam Computed Tomography (CBCT) images from 292 Han individuals (ranging from 5 to 53 years) were used to train and validate the segmentation model. Measurements included sinus dimensions (length, width, height), inter-sinus distance, and volume. Age estimation models using multiple linear regression and random forest algorithms were built based on these variables.

RESULTS

The automatic segmentation model achieved high accuracy, which yielded a Dice similarity coefficient (DSC) of 0.873, an Intersection over Union (IoU) of 0.7753, a Hausdorff Distance 95% (HD95) of 9.8337, and an Average Surface Distance (ASD) of 2.4507. The regression model performed best, with mean absolute errors (MAE) of 1.45 years (under 18) and 3.51 years (aged 18 and above), providing relatively precise age predictions.

CONCLUSION

The maxillary sinus-based model is a promising tool for age estimation, particularly in adults, and could be enhanced by incorporating additional variables like dental dimensions.

Retrieval Augmented Medical Diagnosis System

Subjective variability in human interpretation of diagnostic imaging presents significant clinical limitations, potentially resulting in diagnostic errors and increased healthcare costs. While artificial intelligence (AI) algorithms offer promising solutions to reduce interpreter subjectivity, they frequently demonstrate poor generalizability across different healthcare settings. To address these issues, we introduce Retrieval Augmented Medical Diagnosis System (RAMDS), which integrates an AI classification model with a similar image model. This approach retrieves historical cases and their diagnoses to provide context for the AI predictions. By weighing similar image diagnoses alongside AI predictions, RAMDS produces a final weighted prediction, aiding physicians in understanding the diagnosis process. Moreover, RAMDS does not require complete retraining when applied to new datasets; rather, it simply necessitates re-calibration of the weighing system. When RAMDS fine-tuned for negative predictive value was evaluated on breast ultrasounds for cancer classification, RAMDS improved sensitivity by 21% and negative predictive value by 9% compared to ResNet-34. Offering enhanced metrics, explainability, and adaptability, RAMDS represents a notable advancement in medical AI. RAMDS is a new approach in medical AI that has the potential for pan-pathological uses, though further research is needed to optimize its performance and integrate multimodal data.

Reinforcement-based leveraging transfer learning for multiclass optical coherence tomography images classification

The accurate diagnosis of retinal diseases, such as Diabetic Macular Edema (DME) and Age-related Macular Degeneration (AMD), is essential for preventing vision loss. Optical Coherence Tomography (OCT) imaging plays a crucial role in identifying these conditions, especially given the increasing prevalence of AMD. This study introduces a novel Reinforcement-Based Leveraging Transfer Learning (RBLTL) framework, which integrates reinforcement Q-learning with transfer learning using pre-trained models, including InceptionV3, DenseNet201, and InceptionResNetV2. The RBLTL framework dynamically optimizes hyperparameters, improving classification accuracy and generalization while mitigating overfitting. Experimental evaluations demonstrate remarkable performance, achieving testing accuracies of 98.75%, 98.90%, and 99.20% across three scenarios for multiclass OCT image classification. These results highlight the effectiveness of the RBLTL framework in categorizing OCT images for conditions like DME and AMD, establishing it as a reliable and versatile approach for automated medical image classification with significant implications for clinical diagnostics.

Identification of benign and malignant breast nodules on ultrasound: comparison of multiple deep learning models and model interpretation

Background and Purpose

Deep learning (DL) algorithms generally require full supervision of annotating the region of interest (ROI), a process that is both labor-intensive and susceptible to bias. We aimed to develop a weakly supervised algorithm to differentiate between benign and malignant breast tumors in ultrasound images without image annotation.

Methods

We developed and validated the models using two publicly available datasets: breast ultrasound image (BUSI) and GDPH&SYSUCC breast ultrasound datasets. After removing the poor quality images, a total of 3049 images were included, divided into two classes: benign (N = 1320 images) and malignant (N = 1729 images). Weakly-supervised DL algorithms were implemented with four networks (DenseNet121, ResNet50, EffientNetb0, and Vision Transformer) and trained using 2136 unannotated breast ultrasound images. 609 and 304 images were used for validation and test sets, respectively. Diagnostic performances were calculated as the area under the receiver operating characteristic curve (AUC). Using the class activation map to interpret the prediction results of weakly supervised DL algorithms.

Results

The DenseNet121 model, utilizing complete image inputs without ROI annotations, demonstrated superior diagnostic performance in distinguishing between benign and malignant breast nodules when compared to ResNet50, EfficientNetb0, and Vision Transformer models. DenseNet121 achieved the highest AUC, with values of 0.94 on the validation set and 0.93 on the test set, significantly surpassing the performance of the other models across both datasets (all P < 0.05).

Conclusion

The weakly supervised DenseNet121 model developed in this study demonstrated feasibility for ultrasound diagnosis of breast tumor and showed good capabilities in differential diagnosis. This model may help radiologists, especially novice doctors, to improve the accuracy of breast tumor diagnosis using ultrasound.

Machine Learning in Solid-State Hydrogen Storage Materials: Challenges and Perspectives

Machine learning (ML) has emerged as a pioneering tool in advancing the research application of high-performance solid-state hydrogen storage materials (HSMs). This review summarizes the state-of-the-art research of ML in resolving crucial issues such as low hydrogen storage capacity and unfavorable de-/hydrogenation cycling conditions. First, the datasets, feature descriptors, and prevalent ML models tailored for HSMs are described. Specific examples include the successful application of ML in titanium-based, rare-earth-based, solid solution, magnesium-based, and complex HSMs, showcasing its role in exploiting composition-structure-property relationships and designing novel HSMs for specific applications. One of the representative ML works is the single-phase Ti-based HSM with superior cost-effective and comprehensive properties, tailored to fuel cell hydrogen feeding system at ambient temperature and pressure through high-throughput composition-performance scanning. More importantly, this review also identifies and critically analyzes the key challenges faced by ML in this domain, including poor data quality and availability, and the balance between model interpretability and accuracy, together with feasible countermeasures suggested to ameliorate these problems. In summary, this work outlines a roadmap for enhancing ML's utilization in solid-state hydrogen storage research, promoting more efficient and sustainable energy storage solutions.

Let it shine: Autofluorescence of Papanicolaou-stain improves AI-based cytological oral cancer detection

BACKGROUND AND OBJECTIVES

Oral cancer is a global health challenge. The disease can be successfully treated if detected early, but the survival rate drops significantly for late stage cases. There is a growing interest in a shift from the current standard of invasive and time-consuming tissue sampling and histological examination, towards non-invasive brush biopsies and cytological examination, facilitating continued risk group monitoring. For cost effective and accurate cytological analysis there is a great need for reliable computer-assisted data-driven approaches. However, infeasibility of accurate cell-level annotation hinders model performance, and limits evaluation and interpretation of the results. This study aims to improve AI-based oral cancer detection by introducing additional information through multimodal imaging and deep multimodal information fusion.

METHODS

We combine brightfield and fluorescence whole slide microscopy imaging to analyze Papanicolaou-stained liquid-based cytology slides of brush biopsies collected from both healthy and cancer patients. Given the challenge of detailed cytological annotations, we utilize a weakly supervised deep learning approach only relying on patient-level labels. We evaluate various multimodal information fusion strategies, including early, late, and three recent intermediate fusion methods.

RESULTS

Our experiments demonstrate that: (i) there is substantial diagnostic information to gain from fluorescence imaging of Papanicolaou-stained cytological samples, (ii) multimodal information fusion improves classification performance and cancer detection accuracy, compared to single-modality approaches. Intermediate fusion emerges as the leading method among the studied approaches. Specifically, the Co-Attention Fusion Network (CAFNet) model achieves impressive results, with an F1 score of 83.34% and an accuracy of 91.79% at cell level, surpassing human performance on the task. Additional tests highlight the importance of accurate image registration to maximize the benefits of the multimodal analysis.

CONCLUSION

This study advances the field of cytopathology by integrating deep learning methods, multimodal imaging and information fusion to enhance non-invasive early detection of oral cancer. Our approach not only improves diagnostic accuracy, but also allows an efficient, yet uncomplicated, clinical workflow. The developed pipeline has potential applications in other cytological analysis settings. We provide a validated open-source analysis framework and share a unique multimodal oral cancer dataset to support further research and innovation.

AttriPrompter: Auto-Prompting With Attribute Semantics for Zero-Shot Nuclei Detection via Visual-Language Pre-Trained Models

Large-scale visual-language pre-trained models (VLPMs) have demonstrated exceptional performance in downstream object detection through text prompts for natural scenes. However, their application to zero-shot nuclei detection on histopathology images remains relatively unexplored, mainly due to the significant gap between the characteristics of medical images and the web-originated text-image pairs used for pre-training. This paper aims to investigate the potential of the object-level VLPM, Grounded Language-Image Pre-training (GLIP), for zero-shot nuclei detection. Specifically, we propose an innovative auto-prompting pipeline, named AttriPrompter, comprising attribute generation, attribute augmentation, and relevance sorting, to avoid subjective manual prompt design. AttriPrompter utilizes VLPMs' text-to-image alignment to create semantically rich text prompts, which are then fed into GLIP for initial zero-shot nuclei detection. Additionally, we propose a self-trained knowledge distillation framework, where GLIP serves as the teacher with its initial predictions used as pseudo labels, to address the challenges posed by high nuclei density, including missed detections, false positives, and overlapping instances. Our method exhibits remarkable performance in label-free nuclei detection, outperforming all existing unsupervised methods and demonstrating excellent generality. Notably, this work highlights the astonishing potential of VLPMs pre-trained on natural image-text pairs for downstream tasks in the medical field as well. Code will be released at github.com/AttriPrompter.

Evaluating the generalisability of region-naïve machine learning algorithms for the identification of epilepsy in low-resource settings

OBJECTIVES

Approximately 80% of people with epilepsy live in low- and middle-income countries (LMICs), where limited resources and stigma hinder accurate diagnosis and treatment. Clinical machine learning models have demonstrated substantial promise in supporting the diagnostic process in LMICs by aiding in preliminary screening and detection of possible epilepsy cases without relying on specialised or trained personnel. How well these models generalise to naïve regions is, however, underexplored. Here, we use a novel approach to assess the suitability and applicability of such clinical tools to aid screening and diagnosis of active convulsive epilepsy in settings beyond their original training contexts.

METHODS

We sourced data from the Study of Epidemiology of Epilepsy in Demographic Sites dataset, which includes demographic information and clinical variables related to diagnosing epilepsy across five sub-Saharan African sites. For each site, we developed a region-specific (single-site) predictive model for epilepsy and assessed its performance at other sites. We then iteratively added sites to a multi-site model and evaluated model performance on the omitted regions. Model performances and parameters were then compared across every permutation of sites. We used a leave-one-site-out cross-validation analysis to assess the impact of incorporating individual site data in the model.

RESULTS

Single-site clinical models performed well within their own regions, but generally worse when evaluated in other regions (p<0.05). Model weights and optimal thresholds varied markedly across sites. When the models were trained using data from an increasing number of sites, mean internal performance decreased while external performance improved.

CONCLUSIONS

Clinical models for epilepsy diagnosis in LMICs demonstrate characteristic traits of ML models, such as limited generalisability and a trade-off between internal and external performance. The relationship between predictors and model outcomes also varies across sites, suggesting the need to update specific model aspects with local data before broader implementation. Variations are likely to be particular to the cultural context of diagnosis. We recommend developing models adapted to the cultures and contexts of their intended deployment and caution against deploying region- and culture-naïve models without thorough prior evaluation.

Moment-Consistent Contrastive CycleGAN for Cross-Domain Pancreatic Image Segmentation

CT and MR are currently the most common imaging techniques for pancreatic cancer diagnosis. Accurate segmentation of the pancreas in CT and MR images can provide significant help in the diagnosis and treatment of pancreatic cancer. Traditional supervised segmentation methods require a large number of labeled CT and MR training data, which is usually time-consuming and laborious. Meanwhile, due to domain shift, traditional segmentation networks are difficult to be deployed on different imaging modality datasets. Cross-domain segmentation can utilize labeled source domain data to assist unlabeled target domains in solving the above problems. In this paper, a cross-domain pancreas segmentation algorithm is proposed based on Moment-Consistent Contrastive Cycle Generative Adversarial Networks (MC-CCycleGAN). MC-CCycleGAN is a style transfer network, in which the encoder of its generator is used to extract features from real images and style transfer images, constrain feature extraction through a contrastive loss, and fully extract structural features of input images during style transfer while eliminate redundant style features. The multi-order central moments of the pancreas are proposed to describe its anatomy in high dimensions and a contrastive loss is also proposed to constrain the moment consistency, so as to maintain consistency of the pancreatic structure and shape before and after style transfer. Multi-teacher knowledge distillation framework is proposed to transfer the knowledge from multiple teachers to a single student, so as to improve the robustness and performance of the student network. The experimental results have demonstrated the superiority of our framework over state-of-the-art domain adaptation methods.

Semi-Supervised Detection Model Based on Adaptive Ensemble Learning for Medical Images

Introducing deep learning technologies into the medical image processing field requires accuracy guarantee, especially for high-resolution images relayed through endoscopes. Moreover, works relying on supervised learning are powerless in the case of inadequate labeled samples. Therefore, for end-to-end medical image detection with overcritical efficiency and accuracy in endoscope detection, an ensemble-learning-based model with a semi-supervised mechanism is developed in this work. To gain a more accurate result through multiple detection models, we propose a new ensemble mechanism, termed alternative adaptive boosting method (Al-Adaboost), combining the decision-making of two hierarchical models. Specifically, the proposal consists of two modules. One is a local region proposal model with attentive temporal-spatial pathways for bounding box regression and classification, and the other one is a recurrent attention model (RAM) to provide more precise inferences for further classification according to the regression result. The proposal Al-Adaboost will adjust the weights of labeled samples and the two classifiers adaptively, and the nonlabel samples are assigned pseudolabels by our model. We investigate the performance of Al-Adaboost on both the colonoscopy and laryngoscopy data coming from CVC-ClinicDB and the affiliated hospital of Kaohsiung Medical University. The experimental results prove the feasibility and superiority of our model.

Resource-Constrained Multisource Instance-Based Transfer Learning

In today's machine learning (ML), the need for vast amounts of training data has become a significant challenge. Transfer learning (TL) offers a promising solution by leveraging knowledge across different domains/tasks, effectively addressing data scarcity. However, TL encounters computational and communication challenges in resource-constrained scenarios, and negative transfer (NT) can arise from specific data distributions. This article presents a novel focus on maximizing the accuracy of instance-based TL in multisource resource-constrained environments while mitigating NT, a key concern in TL. Previous studies have overlooked the impact of resource consumption in addressing the NT problem. To address these challenges, we introduce an optimization model named multisource resource-constrained optimized TL (MSOPTL), which employs a convex combination of empirical sources and target errors while considering feasibility and resource constraints. Moreover, we enhance one of the generalization error upper bounds in domain adaptation setting by demonstrating the potential to substitute the divergence with the Kullback-Leibler (KL) divergence. We utilize this enhanced error upper bound as one of the feasibility constraints of MSOPTL. Our suggested model can be applied as a versatile framework for various ML methods. Our approach is extensively validated in a neural network (NN)-based classification problem, demonstrating the efficiency of MSOPTL in achieving the desired trade-offs between TL's benefits and associated costs. This advancement holds tremendous potential for enhancing edge artificial intelligence (AI) applications in resource-constrained environments.

Consistency-Guided Differential Decoding for Enhancing Semi-Supervised Medical Image Segmentation

Semi-supervised learning (SSL) has been proven beneficial for mitigating the issue of limited labeled data, especially on volumetric medical image segmentation. Unlike previous SSL methods which focus on exploring highly confident pseudo-labels or developing consistency regularization schemes, our empirical findings suggest that differential decoder features emerge naturally when two decoders strive to generate consistent predictions. Based on the observation, we first analyze the treasure of discrepancy in learning towards consistency, under both pseudo-labeling and consistency regularization settings, and subsequently propose a novel SSL method called LeFeD, which learns the feature-level discrepancies obtained from two decoders, by feeding such information as feedback signals to the encoder. The core design of LeFeD is to enlarge the discrepancies by training differential decoders, and then learn from the differential features iteratively. We evaluate LeFeD against eight state-of-the-art (SOTA) methods on three public datasets. Experiments show LeFeD surpasses competitors without any bells and whistles, such as uncertainty estimation and strong constraints, as well as setting a new state of the art for semi-supervised medical image segmentation. Code has been released at https://github.com/maxwell0027/LeFeD.

WGAN-GP for Synthetic Retinal Image Generation: Enhancing Sensor-Based Medical Imaging for Classification Models

Accurate synthetic image generation is crucial for addressing data scarcity challenges in medical image classification tasks, particularly in sensor-derived medical imaging. In this work, we propose a novel method using a Wasserstein Generative Adversarial Network with Gradient Penalty (WGAN-GP) and nearest-neighbor interpolation to generate high-quality synthetic images for diabetic retinopathy classification. Our approach enhances training datasets by generating realistic retinal images that retain critical pathological features. We evaluated the method across multiple retinal image datasets, including Retinal-Lesions, Fine-Grained Annotated Diabetic Retinopathy (FGADR), Indian Diabetic Retinopathy Image Dataset (IDRiD), and the Kaggle Diabetic Retinopathy dataset. The proposed method outperformed traditional generative models, such as conditional GANs and PathoGAN, achieving the best performance on key metrics: a Fréchet Inception Distance (FID) of 15.21, a Mean Squared Error (MSE) of 0.002025, and a Structural Similarity Index (SSIM) of 0.89 in the Kaggle dataset. Additionally, expert evaluations revealed that only 56.66% of synthetic images could be distinguished from real ones, demonstrating the high fidelity and clinical relevance of the generated data. These results highlight the effectiveness of our approach in improving medical image classification by generating realistic and diverse synthetic datasets.

Cracking the code of adaptive immunity: The role of computational tools

In recent years, the advances in high-throughput and deep sequencing have generated a diverse amount of adaptive immune repertoire data. This surge in data has seen a proportional increase in computational methods aimed to characterize T cell receptor (TCR) repertoires. In this perspective, we will provide a brief commentary on the various domains of TCR repertoire analysis, their respective computational methods, and the ongoing challenges. Given the breadth of methods and applications of TCR analysis, we will focus our perspective on sequence-based computational methods.

WISE: Efficient WSI selection for active learning in histopathology

Deep neural network (DNN) models have been applied to a wide variety of medical image analysis tasks, often with the successful performance outcomes that match those of medical doctors. However, given that even minor errors in a model can impact patients' life, it is critical that these models are continuously improved. Hence, active learning (AL) has garnered attention as an effective and sustainable strategy for enhancing DNN models for the medical domain. Extant AL research in histopathology has primarily focused on patch datasets derived from whole-slide images (WSIs), a standard form of cancer diagnostic images obtained from a high-resolution scanner. However, this approach has failed to address the selection of WSIs, which can impede the performance improvement of deep learning models and increase the number of WSIs needed to achieve the target performance. This study introduces a WSI-level AL method, termed WSI-informative selection (WISE). WISE is designed to select informative WSIs using a newly formulated WSI-level class distance metric. This method aims to identify diverse and uncertain cases of WSIs, thereby contributing to model performance enhancement. WISE demonstrates state-of-the-art performance across the Colon and Stomach datasets, collected in the real world, as well as the public DigestPath dataset, significantly reducing the required number of WSIs by more than threefold compared to the one-pool dataset setting, which has been dominantly used in the field.

Deep-DM: Deep-Driven Deformable Model for 3D Image Segmentation Using Limited Data

Objective - Medical image segmentation is essential for several clinical tasks, including diagnosis, surgical and treatment planning, and image-guided interventions. Deep Learning (DL) methods have become the state-of-the-art for several image segmentation scenarios. However, a large and well-annotated dataset is required to effectively train a DL model, which is usually difficult to obtain in clinical practice, especially for 3D images. Methods - In this paper, we proposed Deep-DM, a learning-guided deformable model framework for 3D medical imaging segmentation using limited training data. In the proposed method, an energy function is learned by a Convolutional Neural Network (CNN) and integrated into an explicit deformable model to drive the evolution of an initial surface towards the object to segment. Specifically, the learning-based energy function is iteratively retrieved from localized anatomical representations of the image containing the image information around the evolving surface at each iteration. By focusing on localized regions of interest, this representation excludes irrelevant image information, facilitating the learning process. Results and conclusion - The performance of the proposed method is demonstrated for the tasks of left ventricle and fetal head segmentation in ultrasound, left atrium segmentation in Magnetic Resonance, and bladder segmentation in Computed Tomography, using different numbers of training volumes in each study. The results obtained showed the feasibility of the proposed method to segment different anatomical structures in different imaging modalities. Moreover, the results also showed that the proposed approach is less dependent on the size of the training dataset in comparison with state-of-the-art DL-based segmentation methods, outperforming them for all tasks when a low number of samples is available. Significance - Overall, by offering a more robust and less data-intensive approach to accurately segmenting anatomical structures, the proposed method has the potential to enhance clinical tasks that require image segmentation strategies.

PET/CT radiomics and deep learning in the diagnosis of benign and malignant pulmonary nodules: progress and challenges

Lung cancer is currently the leading cause of cancer-related deaths, and early diagnosis and screening can significantly reduce its mortality rate. Since some early-stage lung cancers lack obvious clinical symptoms and only present as pulmonary nodules (PNs) in imaging examinations, accurately determining the benign or malignant nature of PNs is crucial for improving patient survival rates. 18F-FDG PET/CT is important in diagnosing PNs, but its specificity needs improvement. Radiomics can provide information beyond traditional visual assessment, overcoming its limitations by extracting high-throughput quantitative features from medical images. Radiomics features based on 18F-FDG PET/CT and deep learning methods have shown great potential in the noninvasive diagnosis of PNs. This paper reviews the latest advancements in these methods and discusses their contributions to improving diagnostic accuracy and the challenges they face.

Global review of publicly available image datasets for the anterior segment of the eye

This study comprehensively reviewed publicly available image datasets for the anterior segment, with a focus on cataract, refractive, and corneal surgeries. The goal was to assess characteristics of existing datasets and identify areas for improvement. PubMED and Google searches were performed using the search terms "refractive surgery," "anterior segment," "cornea," "corneal," "cataract" AND "database," with the related word of "imaging." Results of each of these searches were collated, identifying 26 publicly available anterior segment image datasets. Imaging modalities included optical coherence tomography, photography, and confocal microscopy. Most datasets were small, 80% originated in the U.S., China, or Europe. Over 50% of images were from normal eyes. Disease states represented included keratoconus, corneal ulcers, and Fuchs dystrophy. Most of the datasets were incompletely described. To promote accessibility going forward to 2030, the ESCRS Digital Health Special Interest Group will annually update a list of available image datasets for anterior segment at www.escrs.org .

An Immunofluorescence-Guided Segmentation Model in Hematoxylin and Eosin Images Is Enabled by Tissue Artifact Correction Using a Cycle-Consistent Generative Adversarial Network

Despite recent advances, the adoption of computer vision methods into clinical and commercial applications has been hampered by the limited availability of accurate ground truth tissue annotations required to train robust supervised models. Generating such ground truth can be accelerated by annotating tissue molecularly using immunofluorescence (IF) staining and mapping these annotations to a post-IF hematoxylin and eosin (H&E) (terminal H&E) stain. Mapping the annotations between IF and terminal H&E increases both the scale and accuracy by which ground truth could be generated. However, discrepancies between terminal H&E and conventional H&E caused by IF tissue processing have limited this implementation. We sought to overcome this challenge and achieve compatibility between these parallel modalities using synthetic image generation, in which a cycle-consistent generative adversarial network was applied to transfer the appearance of conventional H&E such that it emulates terminal H&E. These synthetic emulations allowed us to train a deep learning model for the segmentation of epithelium in terminal H&E that could be validated against the IF staining of epithelial-based cytokeratins. The combination of this segmentation model with the cycle-consistent generative adversarial network stain transfer model enabled performative epithelium segmentation in conventional H&E images. The approach demonstrates that the training of accurate segmentation models for the breadth of conventional H&E data can be executed free of human expert annotations by leveraging molecular annotation strategies such as IF, so long as the tissue impacts of the molecular annotation protocol are captured by generative models that can be deployed prior to the segmentation process.

Data-driven prediction of spinal cord injury recovery: An exploration of current status and future perspectives

Spinal Cord Injury (SCI) presents a significant challenge in rehabilitation medicine, with recovery outcomes varying widely among individuals. Machine learning (ML) is a promising approach to enhance the prediction of recovery trajectories, but its integration into clinical practice requires a thorough understanding of its efficacy and applicability. We systematically reviewed the current literature on data-driven models of SCI recovery prediction. The included studies were evaluated based on a range of criteria assessing the approach, implementation, input data preferences, and the clinical outcomes aimed to forecast. We observe a tendency to utilize routinely acquired data, such as International Standards for Neurological Classification of SCI (ISNCSCI), imaging, and demographics, for the prediction of functional outcomes derived from the Spinal Cord Independence Measure (SCIM) III and Functional Independence Measure (FIM) scores with a focus on motor ability. Although there has been an increasing interest in data-driven studies over time, traditional machine learning architectures, such as linear regression and tree-based approaches, remained the overwhelmingly popular choices for implementation. This implies ample opportunities for exploring architectures addressing the challenges of predicting SCI recovery, including techniques for learning from limited longitudinal data, improving generalizability, and enhancing reproducibility. We conclude with a perspective, highlighting possible future directions for data-driven SCI recovery prediction and drawing parallels to other application fields in terms of diverse data types (imaging, tabular, sequential, multimodal), data challenges (limited, missing, longitudinal data), and algorithmic needs (causal inference, robustness).

The Integration of Radiomics and Artificial Intelligence in Modern Medicine

With profound effects on patient care, the role of artificial intelligence (AI) in radiomics has become a disruptive force in contemporary medicine. Radiomics, the quantitative feature extraction and analysis from medical images, offers useful imaging biomarkers that can reveal important information about the nature of diseases, how well patients respond to treatment and patient outcomes. The use of AI techniques in radiomics, such as machine learning and deep learning, has made it possible to create sophisticated computer-aided diagnostic systems, predictive models, and decision support tools. The many uses of AI in radiomics are examined in this review, encompassing its involvement of quantitative feature extraction from medical images, the machine learning, deep learning and computer-aided diagnostic (CAD) systems approaches in radiomics, and the effect of radiomics and AI on improving workflow automation and efficiency, optimize clinical trials and patient stratification. This review also covers the predictive modeling improvement by machine learning in radiomics, the multimodal integration and enhanced deep learning architectures, and the regulatory and clinical adoption considerations for radiomics-based CAD. Particular emphasis is given to the enormous potential for enhancing diagnosis precision, treatment personalization, and overall patient outcomes.

Reducing annotating load: Active learning with synthetic images in surgical instrument segmentation

Accurate instrument segmentation in the endoscopic vision of minimally invasive surgery is challenging due to complex instruments and environments. Deep learning techniques have shown competitive performance in recent years. However, deep learning usually requires a large amount of labeled data to achieve accurate prediction, which poses a significant workload. To alleviate this workload, we propose an active learning-based framework to generate synthetic images for efficient neural network training. In each active learning iteration, a small number of informative unlabeled images are first queried by active learning and manually labeled. Next, synthetic images are generated based on these selected images. The instruments and backgrounds are cropped out and randomly combined with blending and fusion near the boundary. The proposed method leverages the advantage of both active learning and synthetic images. The effectiveness of the proposed method is validated on two sinus surgery datasets and one intraabdominal surgery dataset. The results indicate a considerable performance improvement, especially when the size of the annotated dataset is small. All the code is open-sourced at: https://github.com/HaonanPeng/active_syn_generator.

Uncertainty-aware multiple-instance learning for reliable classification: Application to optical coherence tomography

Deep learning classification models for medical image analysis often perform well on data from scanners that were used to acquire the training data. However, when these models are applied to data from different vendors, their performance tends to drop substantially. Artifacts that only occur within scans from specific scanners are major causes of this poor generalizability. We aimed to enhance the reliability of deep learning classification models using a novel method called Uncertainty-Based Instance eXclusion (UBIX). UBIX is an inference-time module that can be employed in multiple-instance learning (MIL) settings. MIL is a paradigm in which instances (generally crops or slices) of a bag (generally an image) contribute towards a bag-level output. Instead of assuming equal contribution of all instances to the bag-level output, UBIX detects instances corrupted due to local artifacts on-the-fly using uncertainty estimation, reducing or fully ignoring their contributions before MIL pooling. In our experiments, instances are 2D slices and bags are volumetric images, but alternative definitions are also possible. Although UBIX is generally applicable to diverse classification tasks, we focused on the staging of age-related macular degeneration in optical coherence tomography. Our models were trained on data from a single scanner and tested on external datasets from different vendors, which included vendor-specific artifacts. UBIX showed reliable behavior, with a slight decrease in performance (a decrease of the quadratic weighted kappa (κw) from 0.861 to 0.708), when applied to images from different vendors containing artifacts; while a state-of-the-art 3D neural network without UBIX suffered from a significant detriment of performance (κw from 0.852 to 0.084) on the same test set. We showed that instances with unseen artifacts can be identified with OOD detection. UBIX can reduce their contribution to the bag-level predictions, improving reliability without retraining on new data. This potentially increases the applicability of artificial intelligence models to data from other scanners than the ones for which they were developed. The source code for UBIX, including trained model weights, is publicly available through https://github.com/qurAI-amsterdam/ubix-for-reliable-classification.

Multicenter radio-multiomic analysis for predicting breast cancer outcome and unravelling imaging-biological connection

Radiomics offers a noninvasive avenue for predicting clinicopathological factors. However, thorough investigations into a robust breast cancer outcome-predicting model and its biological significance remain limited. This study develops a robust radiomic model for prognosis prediction, and further excavates its biological foundation and transferring prediction performance. We retrospectively collected preoperative dynamic contrast-enhanced MRI data from three distinct breast cancer patient cohorts. In FUSCC cohort (n = 466), Lasso was used to select features correlated with patient prognosis and multivariate Cox regression was utilized to integrate these features and build the radiomic risk model, while multiomic analysis was conducted to investigate the model's biological implications. DUKE cohort (n = 619) and I-SPY1 cohort (n = 128) were used to test the performance of the radiomic signature in outcome prediction. A thirteen-feature radiomic signature was identified in the FUSCC cohort training set and validated in the FUSCC cohort testing set, DUKE cohort and I-SPY1 cohort for predicting relapse-free survival (RFS) and overall survival (OS) (RFS: p = 0.013, p = 0.024 and p = 0.035; OS: p = 0.036, p = 0.005 and p = 0.027 in the three cohorts). Multiomic analysis uncovered metabolic dysregulation underlying the radiomic signature (ATP metabolic process: NES = 1.84, p-adjust = 0.02; cholesterol biosynthesis: NES = 1.79, p-adjust = 0.01). Regarding the therapeutic implications, the radiomic signature exhibited value when combining clinical factors for predicting the pathological complete response to neoadjuvant chemotherapy (DUKE cohort, AUC = 0.72; I-SPY1 cohort, AUC = 0.73). In conclusion, our study identified a breast cancer outcome-predicting radiomic signature in a multicenter radio-multiomic study, along with its correlations with multiomic features in prognostic risk assessment, laying the groundwork for future prospective clinical trials in personalized risk stratification and precision therapy.

Semi-supervised learning towards automated segmentation of PET images with limited annotations: application to lymphoma patients

Manual segmentation poses a time-consuming challenge for disease quantification, therapy evaluation, treatment planning, and outcome prediction. Convolutional neural networks (CNNs) hold promise in accurately identifying tumor locations and boundaries in PET scans. However, a major hurdle is the extensive amount of supervised and annotated data necessary for training. To overcome this limitation, this study explores semi-supervised approaches utilizing unlabeled data, specifically focusing on PET images of diffuse large B-cell lymphoma (DLBCL) and primary mediastinal large B-cell lymphoma (PMBCL) obtained from two centers. We considered 2-[18F]FDG PET images of 292 patients PMBCL (n = 104) and DLBCL (n = 188) (n = 232 for training and validation, and n = 60 for external testing). We harnessed classical wisdom embedded in traditional segmentation methods, such as the fuzzy clustering loss function (FCM), to tailor the training strategy for a 3D U-Net model, incorporating both supervised and unsupervised learning approaches. Various supervision levels were explored, including fully supervised methods with labeled FCM and unified focal/Dice loss, unsupervised methods with robust FCM (RFCM) and Mumford-Shah (MS) loss, and semi-supervised methods combining FCM with supervised Dice loss (MS + Dice) or labeled FCM (RFCM + FCM). The unified loss function yielded higher Dice scores (0.73 ± 0.11; 95% CI 0.67-0.8) than Dice loss (p value < 0.01). Among the semi-supervised approaches, RFCM + αFCM (α = 0.3) showed the best performance, with Dice score of 0.68 ± 0.10 (95% CI 0.45-0.77), outperforming MS + αDice for any supervision level (any α) (p < 0.01). Another semi-supervised approach with MS + αDice (α = 0.2) achieved Dice score of 0.59 ± 0.09 (95% CI 0.44-0.76) surpassing other supervision levels (p < 0.01). Given the time-consuming nature of manual delineations and the inconsistencies they may introduce, semi-supervised approaches hold promise for automating medical imaging segmentation workflows.

A universal multiple instance learning framework for whole slide image analysis

BACKGROUND

The emergence of digital whole slide image (WSI) has driven the development of computational pathology. However, obtaining patch-level annotations is challenging and time-consuming due to the high resolution of WSI, which limits the applicability of fully supervised methods. We aim to address the challenges related to patch-level annotations.

METHODS

We propose a universal framework for weakly supervised WSI analysis based on Multiple Instance Learning (MIL). To achieve effective aggregation of instance features, we design a feature aggregation module from multiple dimensions by considering feature distribution, instances correlation and instance-level evaluation. First, we implement instance-level standardization layer and deep projection unit to improve the separation of instances in the feature space. Then, a self-attention mechanism is employed to explore dependencies between instances. Additionally, an instance-level pseudo-label evaluation method is introduced to enhance the available information during the weak supervision process. Finally, a bag-level classifier is used to obtain preliminary WSI classification results. To achieve even more accurate WSI label predictions, we have designed a key instance selection module that strengthens the learning of local features for instances. Combining the results from both modules leads to an improvement in WSI prediction accuracy.

RESULTS

Experiments conducted on Camelyon16, TCGA-NSCLC, SICAPv2, PANDA and classical MIL benchmark datasets demonstrate that our proposed method achieves a competitive performance compared to some recent methods, with maximum improvement of 14.6 % in terms of classification accuracy.

CONCLUSION

Our method can improve the classification accuracy of whole slide images in a weakly supervised way, and more accurately detect lesion areas.

A review of self-supervised, generative, and few-shot deep learning methods for data-limited magnetic resonance imaging segmentation

Magnetic resonance imaging (MRI) is a ubiquitous medical imaging technology with applications in disease diagnostics, intervention, and treatment planning. Accurate MRI segmentation is critical for diagnosing abnormalities, monitoring diseases, and deciding on a course of treatment. With the advent of advanced deep learning frameworks, fully automated and accurate MRI segmentation is advancing. Traditional supervised deep learning techniques have advanced tremendously, reaching clinical-level accuracy in the field of segmentation. However, these algorithms still require a large amount of annotated data, which is oftentimes unavailable or impractical. One way to circumvent this issue is to utilize algorithms that exploit a limited amount of labeled data. This paper aims to review such state-of-the-art algorithms that use a limited number of annotated samples. We explain the fundamental principles of self-supervised learning, generative models, few-shot learning, and semi-supervised learning and summarize their applications in cardiac, abdomen, and brain MRI segmentation. Throughout this review, we highlight algorithms that can be employed based on the quantity of annotated data available. We also present a comprehensive list of notable publicly available MRI segmentation datasets. To conclude, we discuss possible future directions of the field-including emerging algorithms, such as contrastive language-image pretraining, and potential combinations across the methods discussed-that can further increase the efficacy of image segmentation with limited labels.

WONOEP 2022: Neurotechnology for the diagnosis of epilepsy

Epilepsy's myriad causes and clinical presentations ensure that accurate diagnoses and targeted treatments remain a challenge. Advanced neurotechnologies are needed to better characterize individual patients across multiple modalities and analytical techniques. At the XVIth Workshop on Neurobiology of Epilepsy: Early Onset Epilepsies: Neurobiology and Novel Therapeutic Strategies (WONOEP 2022), the session on "advanced tools" highlighted a range of approaches, from molecular phenotyping of genetic epilepsy models and resected tissue samples to imaging-guided localization of epileptogenic tissue for surgical resection of focal malformations. These tools integrate cutting edge research, clinical data acquisition, and advanced computational methods to leverage the rich information contained within increasingly large datasets. A number of common challenges and opportunities emerged, including the need for multidisciplinary collaboration, multimodal integration, potential ethical challenges, and the multistage path to clinical translation. Despite these challenges, advanced epilepsy neurotechnologies offer the potential to improve our understanding of the underlying causes of epilepsy and our capacity to provide patient-specific treatment.

Evaluation of transfer ensemble learning-based convolutional neural network models for the identification of chronic gingivitis from oral photographs

BACKGROUND

To evaluate the performances of several advanced deep convolutional neural network models (AlexNet, VGG, GoogLeNet, ResNet) based on ensemble learning for recognizing chronic gingivitis from screening oral images.

METHODS

A total of 683 intraoral clinical images acquired from 134 volunteers were used to construct the database and evaluate the models. Four deep ConvNet models were developed using ensemble learning and outperformed a single model. The performances of the different models were evaluated by comparing the accuracy and sensitivity for recognizing the existence of gingivitis from intraoral images.

RESULTS

The ResNet model achieved an area under the curve (AUC) value of 97%, while the AUC values for the GoogLeNet, AlexNet, and VGG models were 94%, 92%, and 89%, respectively. Although the ResNet and GoogLeNet models performed best in classifying gingivitis from images, the sensitivity outcomes were not significantly different among the ResNet, GoogLeNet, and Alexnet models (p>0.05). However, the sensitivity of the VGGNet model differed significantly from those of the other models (p < 0.001).

CONCLUSION

The ResNet and GoogLeNet models show promise for identifying chronic gingivitis from images. These models can help doctors diagnose periodontal diseases efficiently or based on self-examination of the oral cavity by patients.

Federated learning for medical image analysis: A survey

Machine learning in medical imaging often faces a fundamental dilemma, namely, the small sample size problem. Many recent studies suggest using multi-domain data pooled from different acquisition sites/centers to improve statistical power. However, medical images from different sites cannot be easily shared to build large datasets for model training due to privacy protection reasons. As a promising solution, federated learning, which enables collaborative training of machine learning models based on data from different sites without cross-site data sharing, has attracted considerable attention recently. In this paper, we conduct a comprehensive survey of the recent development of federated learning methods in medical image analysis. We have systematically gathered research papers on federated learning and its applications in medical image analysis published between 2017 and 2023. Our search and compilation were conducted using databases from IEEE Xplore, ACM Digital Library, Science Direct, Springer Link, Web of Science, Google Scholar, and PubMed. In this survey, we first introduce the background of federated learning for dealing with privacy protection and collaborative learning issues. We then present a comprehensive review of recent advances in federated learning methods for medical image analysis. Specifically, existing methods are categorized based on three critical aspects of a federated learning system, including client end, server end, and communication techniques. In each category, we summarize the existing federated learning methods according to specific research problems in medical image analysis and also provide insights into the motivations of different approaches. In addition, we provide a review of existing benchmark medical imaging datasets and software platforms for current federated learning research. We also conduct an experimental study to empirically evaluate typical federated learning methods for medical image analysis. This survey can help to better understand the current research status, challenges, and potential research opportunities in this promising research field.

A Multimodal Transfer Learning Approach for Histopathology and SR-microCT Low-Data Regimes Image Segmentation

Osteocyte-lacunar bone structures are a discerning marker for bone pathophysiology, given their geometric alterations observed during aging and diseases. Deep Learning (DL) image analysis has showcased the potential to comprehend bone health associated with their mechanisms. However, DL examination requires labeled and multimodal datasets, which is arduous with high-dimensional images. Within this context, we propose a method for segmenting osteocytes and lacunae in human bone histopathology and Synchrotron Radiation micro-Computed Tomography (SR-microCT) images, employing a deep U-Net in an intra-domain and multimodal transfer learning setting with a limited number of training images. Our strategy allows achieving 63.92±4.69 and 63.94±4.05 Dice Similarity Coefficient (DSC) osteocytes and lacunae segmentation, while up to 20.38 and 5.86 average DSC improvements over selected baselines even if 44× smaller datasets are employed for training.Clinical relevance-The proposed method analyzes bone histopathologies and SR-microCT images in a multimodal and low-data setting, easing the bone microscale investigations while supporting the study of osteocyte-lacunar pathophysiology.

Weakly Supervised Lesion Detection and Diagnosis for Breast Cancers With Partially Annotated Ultrasound Images

Deep learning (DL) has proven highly effective for ultrasound-based computer-aided diagnosis (CAD) of breast cancers. In an automatic CAD system, lesion detection is critical for the following diagnosis. However, existing DL-based methods generally require voluminous manually-annotated region of interest (ROI) labels and class labels to train both the lesion detection and diagnosis models. In clinical practice, the ROI labels, i.e. ground truths, may not always be optimal for the classification task due to individual experience of sonologists, resulting in the issue of coarse annotation to limit the diagnosis performance of a CAD model. To address this issue, a novel Two-Stage Detection and Diagnosis Network (TSDDNet) is proposed based on weakly supervised learning to improve diagnostic accuracy of the ultrasound-based CAD for breast cancers. In particular, all the initial ROI-level labels are considered as coarse annotations before model training. In the first training stage, a candidate selection mechanism is then designed to refine manual ROIs in the fully annotated images and generate accurate pseudo-ROIs for the partially annotated images under the guidance of class labels. The training set is updated with more accurate ROI labels for the second training stage. A fusion network is developed to integrate detection network and classification network into a unified end-to-end framework as the final CAD model in the second training stage. A self-distillation strategy is designed on this model for joint optimization to further improves its diagnosis performance. The proposed TSDDNet is evaluated on three B-mode ultrasound datasets, and the experimental results indicate that it achieves the best performance on both lesion detection and diagnosis tasks, suggesting promising application potential.

A 3D ray traced biological neural network learning model

Training large neural networks on big datasets requires significant computational resources and time. Transfer learning reduces training time by pre-training a base model on one dataset and transferring the knowledge to a new model for another dataset. However, current choices of transfer learning algorithms are limited because the transferred models always have to adhere to the dimensions of the base model and can not easily modify the neural architecture to solve other datasets. On the other hand, biological neural networks (BNNs) are adept at rearranging themselves to tackle completely different problems using transfer learning. Taking advantage of BNNs, we design a dynamic neural network that is transferable to any other network architecture and can accommodate many datasets. Our approach uses raytracing to connect neurons in a three-dimensional space, allowing the network to grow into any shape or size. In the Alcala dataset, our transfer learning algorithm trains the fastest across changing environments and input sizes. In addition, we show that our algorithm also outperformance the state of the art in EEG dataset. In the future, this network may be considered for implementation on real biological neural networks to decrease power consumption.

GMIM: Self-supervised pre-training for 3D medical image segmentation with adaptive and hierarchical masked image modeling

Self-supervised pre-training and fully supervised fine-tuning paradigms have received much attention to solve the data annotation problem in deep learning fields. Compared with traditional pre-training on large natural image datasets, medical self-supervised learning methods learn rich representations derived from unlabeled data itself thus avoiding the distribution shift between different image domains. However, nowadays state-of-the-art medical pre-training methods were specifically designed for downstream tasks making them less flexible and difficult to apply to new tasks. In this paper, we propose grid mask image modeling, a flexible and general self-supervised method to pre-train medical vision transformers for 3D medical image segmentation. Our goal is to guide networks to learn the correlations between organs and tissues by reconstructing original images based on partial observations. The relationships are consistent within the human body and invariant to disease type or imaging modality. To achieve this, we design a Siamese framework consisting of an online branch and a target branch. An adaptive and hierarchical masking strategy is employed in the online branch to (1) learn the boundaries or small contextual mutation regions within images; (2) to learn high-level semantic representations from deeper layers of the multiscale encoder. In addition, the target branch provides representations for contrastive learning to further reduce representation redundancy. We evaluate our method through segmentation performance on two public datasets. The experimental results demonstrate our method outperforms other self-supervised methods. Codes are available at https://github.com/mobiletomb/Gmim.

Multi-organ segmentation: a progressive exploration of learning paradigms under scarce annotation

Precise delineation of multiple organs or abnormal regions in the human body from medical images plays an essential role in computer-aided diagnosis, surgical simulation, image-guided interventions, and especially in radiotherapy treatment planning. Thus, it is of great significance to explore automatic segmentation approaches, among which deep learning-based approaches have evolved rapidly and witnessed remarkable progress in multi-organ segmentation. However, obtaining an appropriately sized and fine-grained annotated dataset of multiple organs is extremely hard and expensive. Such scarce annotation limits the development of high-performance multi-organ segmentation models but promotes many annotation-efficient learning paradigms. Among these, studies on transfer learning leveraging external datasets, semi-supervised learning including unannotated datasets and partially-supervised learning integrating partially-labeled datasets have led the dominant way to break such dilemmas in multi-organ segmentation. We first review the fully supervised method, then present a comprehensive and systematic elaboration of the 3 abovementioned learning paradigms in the context of multi-organ segmentation from both technical and methodological perspectives, and finally summarize their challenges and future trends.

Constructing a Deep Learning Radiomics Model Based on X-ray Images and Clinical Data for Predicting and Distinguishing Acute and Chronic Osteoporotic Vertebral Fractures: A Multicenter Study

RATIONALE AND OBJECTIVES

To construct and validate a deep learning radiomics (DLR) model based on X-ray images for predicting and distinguishing acute and chronic osteoporotic vertebral fractures (OVFs).

METHODS

A total of 942 cases (1076 vertebral bodies) with both vertebral X-ray examination and MRI scans were included in this study from three hospitals. They were divided into a training cohort (n = 712), an internal validation cohort (n = 178), an external validation cohort (n = 111), and a prospective validation cohort (n = 75). The ResNet-50 model architecture was used for deep transfer learning (DTL), with pre-training performed on RadImageNet and ImageNet datasets. DTL features and radiomics features were extracted from lateral X-ray images of OVFs patients and fused together. A logistic regression model with the least absolute shrinkage and selection operator was established, with MRI showing bone marrow edema as the gold standard for acute OVFs. The performance of the model was evaluated using receiver operating characteristic curves. Eight machine learning classification models were evaluated for their ability to distinguish between acute and chronic OVFs. The Nomogram was constructed by combining clinical baseline data to achieve visualized classification assessment. The predictive performance of the best RadImageNet model and ImageNet model was compared using the Delong test. The clinical value of the Nomogram was evaluated using decision curve analysis (DCA).

RESULTS

Pre-training resulted in 34 and 39 fused features after feature selection and fusion. The most effective machine learning algorithm in both DLR models was Light Gradient Boosting Machine. Using the Delong test, the area under the curve (AUC) for distinguishing between acute and chronic OVFs in the training cohort was 0.979 and 0.972 for the RadImageNet and ImageNet models, respectively, with no statistically significant difference between them (P = 0.235). In the internal validation cohort, external validation cohort, and prospective validation cohort, the AUCs for the two models were 0.967 vs 0.629, 0.886 vs 0.817, and 0.933 vs 0.661, respectively, with statistically significant differences in all comparisons (P < 0.05). The deep learning radiomics nomogram (DLRN) was constructed by combining the predictive model of RadImageNet with clinical baseline features, resulting in AUCs of 0.981, 0.974, 0.895, and 0.902 in the training cohort, internal validation cohort, external validation cohort, and prospective validation cohort, respectively. Using the Delong test, the AUCs for the fused feature model and the DLRN in the training cohort were 0.979 and 0.981, respectively, with no statistically significant difference between them (P = 0.169). In the internal validation cohort, external validation cohort, and prospective validation cohort, the AUCs for the two models were 0.967 vs 0.974, 0.886 vs 0.895, and 0.933 vs 0.902, respectively, with statistically significant differences in all comparisons (P < 0.05). The Nomogram showed a slight improvement in predictive performance in the internal and external validation cohort, but a slight decrease in the prospective validation cohort (0.933 vs 0.902). DCA showed that the Nomogram provided more benefits to patients compared to the DLR models.

CONCLUSION

Compared to the ImageNet model, the RadImageNet model has higher diagnostic value in distinguishing between acute and chronic OVFs. Furthermore, the diagnostic performance of the model is further improved when combined with clinical baseline features to construct the Nomogram.