Colonoscopy polyp detection and classification: Dataset creation and comparative evaluations

Optimizing colorectal polyp detection and localization: Impact of RGB color adjustment on CNN performance

Colorectal cancer, arising from adenomatous polyps, is a leading cause of cancer-related mortality, making early detection and removal crucial for preventing cancer progression. Machine learning is increasingly used to enhance polyp detection during colonoscopy, the gold standard for colorectal cancer screening, despite its operator-dependent miss rates. This study explores the impact of RGB color adjustment on Convolutional Neural Network (CNN) models for improving polyp detection and localization in colonoscopic images. Using datasets from Harvard Dataverse for training and internal validation, and LDPolypVideo-Benchmark for external validation, RGB color adjustments were applied, and YOLOv8s was used to develop models. Bayesian optimization identified the best RGB adjustments, with performance assessed using mean average precision (mAP) and F1-scores. Results showed that RGB adjustment with 1.0 R-1.0 G-0.8 B improved polyp detection, achieving an mAP of 0.777 and an F1-score of 0.720 on internal test sets, and localization performance with an F1-score of 0.883 on adjusted images. External validation showed improvement but with a lower F1-score of 0.556. While RGB adjustments improved performance in our study, their generalizability to diverse datasets and clinical settings has yet to be validated. Thus, although RGB color adjustment enhances CNN model performance for detecting and localizing colorectal polyps, further research is needed to verify these improvements across diverse datasets and clinical settings.•RGB Color Adjustment: Applied RGB color adjustments to colonoscopic images to enhance the performance of Convolutional Neural Network (CNN) models.•Model Development: Used YOLOv8s for polyp detection and localization, with Bayesian optimization to identify the best RGB adjustments.•Performance Evaluation: Assessed model performance using mAP and F1-scores on both internal and external validation datasets.

Polyp-Size: A Precise Endoscopic Dataset for AI-Driven Polyp Sizing

Colorectal cancer often arises from precancerous polyps, where accurate size assessment is vital for clinical decisions but challenged by subjective methods. While artificial intelligence (AI) has shown promise in improving the accuracy of polyp size estimation, its development depends on large, meticulously annotated datasets. We present Polyp-Size, a dataset of 42 high-resolution white-light colonoscopy videos with polyp sizes precisely measured post-resection using vernier calipers to submillimeter precision. Unlike existing datasets primarily focused on polyp detection or segmentation, Polyp-Size offers validated size annotations, diverse polyp features (Paris classification, anatomical location and histological type), and standardized video formats, enabling robust AI models for size estimation. By making this resource publicly available, we aim to foster research collaboration and innovation in automated polyp measurement to ultimately improve clinical practice.

Multiclassification of Colorectal Polyps from Colonoscopy Images Using AI for Early Diagnosis

Background/Objectives: Colorectal cancer (CRC) remains one of the leading causes of cancer-related mortality worldwide, emphasizing the critical need for the accurate classification of precancerous polyps. This research presents an extensive analysis of the multiclassification framework leveraging various deep learning (DL) architectures for the automated classification of colorectal polyps from colonoscopy images. Methods: The proposed methodology integrates real-time data for training and utilizes a publicly available dataset for testing, ensuring generalizability. The real-time images were cautiously annotated and verified by a panel of experts, including post-graduate medical doctors and gastroenterology specialists. The DL models were designed to categorize the preprocessed colonoscopy images into four clinically significant classes: hyperplastic, serrated, adenoma, and normal. A suite of state-of-the-art models, including VGG16, VGG19, ResNet50, DenseNet121, EfficientNetV2, InceptionNetV3, Vision Transformer (ViT), and the custom-developed CRP-ViT, were trained and rigorously evaluated for this task. Results: Notably, the CRP-ViT model exhibited superior capability in capturing intricate features, achieving an impressive accuracy of 97.28% during training and 96.02% during validation with real-time images. Furthermore, the model demonstrated remarkable performance during testing on the public dataset, attaining an accuracy of 95.69%. To facilitate real-time interaction and clinical applicability, a user-friendly interface was developed using Gradio, allowing healthcare professionals to upload colonoscopy images and receive instant classification results. Conclusions: The CRP-ViT model effectively predicts and categorizes colonoscopy images into clinically relevant classes, aiding gastroenterologists in decision-making. This study highlights the potential of integrating AI-driven models into routine clinical practice to improve colorectal cancer screening outcomes and reduce diagnostic variability.

Improving Foundation Model for Endoscopy Video Analysis via Representation Learning on Long Sequences

Recent advancements in endoscopy video analysis have relied on the utilization of relatively short video clips extracted from longer videos or millions of individual frames. However, these approaches tend to neglect the domain-specific characteristics of endoscopy data, which is typically presented as a long stream containing valuable semantic spatial and temporal information. To address this limitation, we propose EndoFM-LV, a foundation model developed under a minute-level pre-training framework upon long endoscopy video sequences. To be specific, we propose a novel masked token modeling scheme within a teacher-student framework for self-supervised video pre-training, which is tailored for learning representations from long video sequences. For pre-training, we construct a large-scale long endoscopy video dataset comprising 6,469 long endoscopic video samples, each longer than 1 minute and totaling over 13 million frames. Our EndoFM-LV is evaluated on four types of endoscopy tasks, namely classification, segmentation, detection, and workflow recognition, serving as the backbone or temporal module. Extensive experimental results demonstrate that our framework outperforms previous state-of-the-art video-based and frame-based approaches by a significant margin, surpassing Endo-FM (5.6% F1, 9.3% Dice, 8.4% F1, and 3.3% accuracy for classification, segmentation, detection, and workflow recognition) and EndoSSL (5.0% F1, 8.1% Dice, 9.3% F1 and 3.1% accuracy for classification, segmentation, detection, and workflow recognition).

Real-World Colonoscopy Video Integration to Improve Artificial Intelligence Polyp Detection Performance and Reduce Manual Annotation Labor

Background/Objectives: Artificial intelligence (AI) integration in colon polyp detection often exhibits high sensitivity but notably low specificity in real-world settings, primarily due to reliance on publicly available datasets alone. To address this limitation, we proposed a semi-automatic annotation method using real colonoscopy videos to enhance AI model performance and reduce manual labeling labor. Methods: An integrated AI model was trained and validated on 86,258 training images and 17,616 validation images. Model 1 utilized only publicly available datasets, while Model 2 additionally incorporated images obtained from real colonoscopy videos of patients through a semi-automatic annotation process, significantly reducing the labeling burden on expert endoscopists. Results: The integrated AI model (Model 2) significantly outperformed the public-dataset-only model (Model 1). At epoch 35, Model 2 achieved a sensitivity of 90.6%, a specificity of 96.0%, an overall accuracy of 94.5%, and an F1 score of 89.9%. All polyps in the test videos were successfully detected, demonstrating considerable enhancement in detection performance compared to the public-dataset-only model. Conclusions: Integrating real-world colonoscopy video data using semi-automatic annotation markedly improved diagnostic accuracy while potentially reducing the need for extensive manual annotation typically performed by expert endoscopists. However, the findings need validation through multicenter external datasets to ensure generalizability.

From data to artificial intelligence: evaluating the readiness of gastrointestinal endoscopy datasets

The incorporation of artificial intelligence (AI) into gastrointestinal (GI) endoscopy represents a promising advancement in gastroenterology. With over 40 published randomized controlled trials and numerous ongoing clinical trials, gastroenterology leads other medical disciplines in AI research. Computer-aided detection algorithms for identifying colorectal polyps have achieved regulatory approval and are in routine clinical use, while other AI applications for GI endoscopy are in advanced development stages. Near-term opportunities include the potential for computer-aided diagnosis to replace conventional histopathology for diagnosing small colon polyps and increased AI automation in capsule endoscopy. Despite significant development in research settings, the generalizability and robustness of AI models in real clinical practice remain inconsistent. The GI field lags behind other medical disciplines in the breadth of novel AI algorithms, with only 13 out of 882 Food and Drug Administration (FDA)-approved AI models focussed on GI endoscopy as of June 2024. Additionally, existing GI endoscopy image databases are disproportionately focussed on colon polyps, lacking representation of the diversity of other endoscopic findings. High-quality datasets, encompassing a wide range of patient demographics, endoscopic equipment types, and disease states, are crucial for developing effective AI models for GI endoscopy. This article reviews the current state of GI endoscopy datasets, barriers to progress, including dataset size, data diversity, annotation quality, and ethical issues in data collection and usage, and future needs for advancing AI in GI endoscopy.

Synthesized colonoscopy dataset from high-fidelity virtual colon with abnormal simulation

With the advent of the deep learning-based colonoscopy system, the need for a vast amount of high-quality colonoscopy image datasets for training is crucial. However, the generalization ability of deep learning models is challenged by the limited availability of colonoscopy images due to regulatory restrictions and privacy concerns. In this paper, we propose a method for rendering high-fidelity 3D colon models and synthesizing diversified colonoscopy images with abnormalities such as polyps, bleeding, and ulcers, which can be used to train deep learning models. The geometric model of the colon is derived from CT images. We employed dedicated surface mesh deformation to mimic the shapes of polyps and ulcers and applied texture mapping techniques to generate realistic, lifelike appearances. The generated polyp models were then attached to the inner surface of the colon model, while the ulcers were created directly on the inner surface of the colon model. To realistically model blood behavior, we developed a simulation of the blood diffusion process on the colon's inner surface and colored vertices in the traversed region to reflect blood flow. Ultimately, we generated a comprehensive dataset comprising high-fidelity rendered colonoscopy images with the abnormalities. To validate the effectiveness of the synthesized colonoscopy dataset, we trained state-of-the-art deep learning models on it and other publicly available datasets and assessed the performance of these models in abnormal classification, detection, and segmentation. Notably, the models trained on the synthesized dataset exhibit an enhanced performance in the aforementioned tasks, as evident from the results.

IPNet: An Interpretable Network With Progressive Loss for Whole-Stage Colorectal Disease Diagnosis

Colorectal cancer plays a dominant role in cancer-related deaths, primarily due to the absence of obvious early-stage symptoms. Whole-stage colorectal disease diagnosis is crucial for assessing lesion evolution and determining treatment plans. However, locality difference and disease progression lead to intra-class disparities and inter-class similarities for colorectal lesion representation. In addition, interpretable algorithms explaining the lesion progression are still lacking, making the prediction process a "black box". In this paper, we propose IPNet, a dual-branch interpretable network with progressive loss for whole-stage colorectal disease diagnosis. The dual-branch architecture captures unbiased features representing diverse localities to suppress intra-class variation. The progressive loss function considers inter-class relationship, using prior knowledge of disease evolution to guide classification. Furthermore, a novel Grain-CAM is designed to interpret IPNet by visualizing pixel-wise attention maps from shallow to deep layers, providing regions semantically related to IPNet's progressive classification. We conducted whole-stage diagnosis on two image modalities, i.e., colorectal lesion classification on 129,893 endoscopic optical images and rectal tumor T-staging on 11,072 endoscopic ultrasound images. IPNet is shown to surpass other state-of-the-art algorithms, accordingly achieving an accuracy of 93.15% and 89.62%. Especially, it establishes effective decision boundaries for challenges like polyp vs. adenoma and T2 vs. T3. The results demonstrate an explainable attempt for colorectal lesion classification at a whole-stage level, and rectal tumor T-staging by endoscopic ultrasound is also unprecedentedly explored. IPNet is expected to be further applied, assisting physicians in whole-stage disease diagnosis and enhancing diagnostic interpretability.

Colorectal cancer detection with enhanced precision using a hybrid supervised and unsupervised learning approach

The current work introduces the hybrid ensemble framework for the detection and segmentation of colorectal cancer. This framework will incorporate both supervised classification and unsupervised clustering methods to present more understandable and accurate diagnostic results. The method entails several steps with CNN models: ADa-22 and AD-22, transformer networks, and an SVM classifier, all inbuilt. The CVC ClinicDB dataset supports this process, containing 1650 colonoscopy images classified as polyps or non-polyps. The best performance in the ensembles was done by the AD-22 + Transformer + SVM model, with an AUC of 0.99, a training accuracy of 99.50%, and a testing accuracy of 99.00%. This group also saw a high accuracy of 97.50% for Polyps and 99.30% for Non-Polyps, together with a recall of 97.80% for Polyps and 98.90% for Non-Polyps, hence performing very well in identifying both cancerous and healthy regions. The framework proposed here uses K-means clustering in combination with the visualisation of bounding boxes, thereby improving segmentation and yielding a silhouette score of 0.73 with the best cluster configuration. It discusses how to combine feature interpretation challenges into medical imaging for accurate localization and precise segmentation of malignant regions. A good balance between performance and generalization shall be done by hyperparameter optimization-heavy learning rates; dropout rates and overfitting shall be suppressed effectively. The hybrid schema of this work treats the deficiencies of the previous approaches, such as incorporating CNN-based effective feature extraction, Transformer networks for developing attention mechanisms, and finally the fine decision boundary of the support vector machine. Further, we refine this process via unsupervised clustering for the purpose of enhancing the visualisation of such a procedure. Such a holistic framework, hence, further boosts classification and segmentation results by generating understandable outcomes for more rigorous benchmarking of detecting colorectal cancer and with higher reality towards clinical application feasibility.

Deep learning-based object detection algorithms in medical imaging: Systematic review

Over the past decade, Deep Learning (DL) techniques have demonstrated remarkable advancements across various domains, driving their widespread adoption. Particularly in medical image analysis, DL received greater attention for tasks like image segmentation, object detection, and classification. This paper provides an overview of DL-based object recognition in medical images, exploring recent methods and emphasizing different imaging techniques and anatomical applications. Utilizing a meticulous quantitative and qualitative analysis following PRISMA guidelines, we examined publications based on citation rates to explore into the utilization of DL-based object detectors across imaging modalities and anatomical domains. Our findings reveal a consistent rise in the utilization of DL-based object detection models, indicating unexploited potential in medical image analysis. Predominantly within Medicine and Computer Science domains, research in this area is most active in the US, China, and Japan. Notably, DL-based object detection methods have gotten significant interest across diverse medical imaging modalities and anatomical domains. These methods have been applied to a range of techniques including CR scans, pathology images, and endoscopic imaging, showcasing their adaptability. Moreover, diverse anatomical applications, particularly in digital pathology and microscopy, have been explored. The analysis underscores the presence of varied datasets, often with significant discrepancies in size, with a notable percentage being labeled as private or internal, and with prospective studies in this field remaining scarce. Our review of existing trends in DL-based object detection in medical images offers insights for future research directions. The continuous evolution of DL algorithms highlighted in the literature underscores the dynamic nature of this field, emphasizing the need for ongoing research and fitted optimization for specific applications.

Advancements in the use of AI in the diagnosis and management of inflammatory bowel disease

Inflammatory bowel disease (IBD) causes chronic inflammation of the colon and digestive tract, and it can be classified as Crohn's disease (CD) and Ulcerative colitis (UC). IBD is more prevalent in Europe and North America, however, since the beginning of the 21st century it has been increasing in South America, Asia, and Africa, leading to its consideration as a worldwide problem. Optical colonoscopy is one of the crucial tests in diagnosing and assessing the progression and prognosis of IBD, as it allows a real-time optical visualization of the colonic wall and ileum and allows for the collection of tissue samples. The accuracy of colonoscopy procedures depends on the expertise and ability of the endoscopists. Therefore, algorithms based on Deep Learning (DL) and Convolutional Neural Networks (CNN) for colonoscopy images and videos are growing in popularity, especially for the detection and classification of colorectal polyps. The performance of this system is dependent on the quality and quantity of the data used for training. There are several datasets publicly available for endoscopy images and videos, but most of them are solely specialized in polyps. The use of DL algorithms to detect IBD is still in its inception, most studies are based on assessing the severity of UC. As artificial intelligence (AI) grows in popularity there is a growing interest in the use of these algorithms for diagnosing and classifying the IBDs and managing their progression. To tackle this, more annotated colonoscopy images and videos will be required for the training of new and more reliable AI algorithms. This article discusses the current challenges in the early detection of IBD, focusing on the available AI algorithms, and databases, and the challenges ahead to improve the detection rate.

Colonoscopy polyp classification via enhanced scattering wavelet Convolutional Neural Network

Among the most common cancers, colorectal cancer (CRC) has a high death rate. The best way to screen for colorectal cancer (CRC) is with a colonoscopy, which has been shown to lower the risk of the disease. As a result, Computer-aided polyp classification technique is applied to identify colorectal cancer. But visually categorizing polyps is difficult since different polyps have different lighting conditions. Different from previous works, this article presents Enhanced Scattering Wavelet Convolutional Neural Network (ESWCNN), a polyp classification technique that combines Convolutional Neural Network (CNN) and Scattering Wavelet Transform (SWT) to improve polyp classification performance. This method concatenates simultaneously learnable image filters and wavelet filters on each input channel. The scattering wavelet filters can extract common spectral features with various scales and orientations, while the learnable filters can capture image spatial features that wavelet filters may miss. A network architecture for ESWCNN is designed based on these principles and trained and tested using colonoscopy datasets (two public datasets and one private dataset). An n-fold cross-validation experiment was conducted for three classes (adenoma, hyperplastic, serrated) achieving a classification accuracy of 96.4%, and 94.8% accuracy in two-class polyp classification (positive and negative). In the three-class classification, correct classification rates of 96.2% for adenomas, 98.71% for hyperplastic polyps, and 97.9% for serrated polyps were achieved. The proposed method in the two-class experiment reached an average sensitivity of 96.7% with 93.1% specificity. Furthermore, we compare the performance of our model with the state-of-the-art general classification models and commonly used CNNs. Six end-to-end models based on CNNs were trained using 2 dataset of video sequences. The experimental results demonstrate that the proposed ESWCNN method can effectively classify polyps with higher accuracy and efficacy compared to the state-of-the-art CNN models. These findings can provide guidance for future research in polyp classification.

A Semi-Supervised Learning Framework for Classifying Colorectal Neoplasia Based on the NICE Classification

Labelling medical images is an arduous and costly task that necessitates clinical expertise and large numbers of qualified images. Insufficient samples can lead to underfitting during training and poor performance of supervised learning models. In this study, we aim to develop a SimCLR-based semi-supervised learning framework to classify colorectal neoplasia based on the NICE classification. First, the proposed framework was trained under self-supervised learning using a large unlabelled dataset; subsequently, it was fine-tuned on a limited labelled dataset based on the NICE classification. The model was evaluated on an independent dataset and compared with models based on supervised transfer learning and endoscopists using accuracy, Matthew's correlation coefficient (MCC), and Cohen's kappa. Finally, Grad-CAM and t-SNE were applied to visualize the models' interpretations. A ResNet-backboned SimCLR model (accuracy of 0.908, MCC of 0.862, and Cohen's kappa of 0.896) outperformed supervised transfer learning-based models (means: 0.803, 0.698, and 0.742) and junior endoscopists (0.816, 0.724, and 0.863), while performing only slightly worse than senior endoscopists (0.916, 0.875, and 0.944). Moreover, t-SNE showed a better clustering of ternary samples through self-supervised learning in SimCLR than through supervised transfer learning. Compared with traditional supervised learning, semi-supervised learning enables deep learning models to achieve improved performance with limited labelled endoscopic images.

A complete benchmark for polyp detection, segmentation and classification in colonoscopy images

Introduction

Colorectal cancer (CRC) is one of the main causes of deaths worldwide. Early detection and diagnosis of its precursor lesion, the polyp, is key to reduce its mortality and to improve procedure efficiency. During the last two decades, several computational methods have been proposed to assist clinicians in detection, segmentation and classification tasks but the lack of a common public validation framework makes it difficult to determine which of them is ready to be deployed in the exploration room.

Methods

This study presents a complete validation framework and we compare several methodologies for each of the polyp characterization tasks.

Results

Results show that the majority of the approaches are able to provide good performance for the detection and segmentation task, but that there is room for improvement regarding polyp classification.

Discussion

While studied show promising results in the assistance of polyp detection and segmentation tasks, further research should be done in classification task to obtain reliable results to assist the clinicians during the procedure. The presented framework provides a standarized method for evaluating and comparing different approaches, which could facilitate the identification of clinically prepared assisting methods.

Lesion-Decoupling-Based Segmentation With Large-Scale Colon and Esophageal Datasets for Early Cancer Diagnosis

Lesions of early cancers often show flat, small, and isochromatic characteristics in medical endoscopy images, which are difficult to be captured. By analyzing the differences between the internal and external features of the lesion area, we propose a lesion-decoupling-based segmentation (LDS) network for assisting early cancer diagnosis. We introduce a plug-and-play module called self-sampling similar feature disentangling module (FDM) to obtain accurate lesion boundaries. Then, we propose a feature separation loss (FSL) function to separate pathological features from normal ones. Moreover, since physicians make diagnoses with multimodal data, we propose a multimodal cooperative segmentation network with two different modal images as input: white-light images (WLIs) and narrowband images (NBIs). Our FDM and FSL show a good performance for both single-modal and multimodal segmentations. Extensive experiments on five backbones prove that our FDM and FSL can be easily applied to different backbones for a significant lesion segmentation accuracy improvement, and the maximum increase of mean Intersection over Union (mIoU) is 4.58. For colonoscopy, we can achieve up to mIoU of 91.49 on our Dataset A and 84.41 on the three public datasets. For esophagoscopy, mIoU of 64.32 is best achieved on the WLI dataset and 66.31 on the NBI dataset.

Assessing clinical efficacy of polyp detection models using open-access datasets

Background

Ensuring accurate polyp detection during colonoscopy is essential for preventing colorectal cancer (CRC). Recent advances in deep learning-based computer-aided detection (CADe) systems have shown promise in enhancing endoscopists' performances. Effective CADe systems must achieve high polyp detection rates from the initial seconds of polyp appearance while maintaining low false positive (FP) detection rates throughout the procedure.

Method

We integrated four open-access datasets into a unified platform containing over 340,000 images from various centers, including 380 annotated polyps, with distinct data splits for comprehensive model development and benchmarking. The REAL-Colon dataset, comprising 60 full-procedure colonoscopy videos from six centers, is used as the fifth dataset of the platform to simulate clinical conditions for model evaluation on unseen center data. Performance assessment includes traditional object detection metrics and new metrics that better meet clinical needs. Specifically, by defining detection events as sequences of consecutive detections, we compute per-polyp recall at early detection stages and average per-patient FPs, enabling the generation of Free-Response Receiver Operating Characteristic (FROC) curves.

Results

Using YOLOv7, we trained and tested several models across the proposed data splits, showcasing the robustness of our open-access platform for CADe system development and benchmarking. The introduction of new metrics allows for the optimization of CADe operational parameters based on clinically relevant criteria, such as per-patient FPs and early polyp detection. Our findings also reveal that omitting full-procedure videos leads to non-realistic assessments and that detecting small polyp bounding boxes poses the greatest challenge.

Conclusion

This study demonstrates how newly available open-access data supports ongoing research progress in environments that closely mimic clinical settings. The introduced metrics and FROC curves illustrate CADe clinical efficacy and can aid in tuning CADe hyperparameters.

Federated learning for medical image analysis: A survey

Machine learning in medical imaging often faces a fundamental dilemma, namely, the small sample size problem. Many recent studies suggest using multi-domain data pooled from different acquisition sites/centers to improve statistical power. However, medical images from different sites cannot be easily shared to build large datasets for model training due to privacy protection reasons. As a promising solution, federated learning, which enables collaborative training of machine learning models based on data from different sites without cross-site data sharing, has attracted considerable attention recently. In this paper, we conduct a comprehensive survey of the recent development of federated learning methods in medical image analysis. We have systematically gathered research papers on federated learning and its applications in medical image analysis published between 2017 and 2023. Our search and compilation were conducted using databases from IEEE Xplore, ACM Digital Library, Science Direct, Springer Link, Web of Science, Google Scholar, and PubMed. In this survey, we first introduce the background of federated learning for dealing with privacy protection and collaborative learning issues. We then present a comprehensive review of recent advances in federated learning methods for medical image analysis. Specifically, existing methods are categorized based on three critical aspects of a federated learning system, including client end, server end, and communication techniques. In each category, we summarize the existing federated learning methods according to specific research problems in medical image analysis and also provide insights into the motivations of different approaches. In addition, we provide a review of existing benchmark medical imaging datasets and software platforms for current federated learning research. We also conduct an experimental study to empirically evaluate typical federated learning methods for medical image analysis. This survey can help to better understand the current research status, challenges, and potential research opportunities in this promising research field.

Artificial intelligence in colonoscopy: from detection to diagnosis

This study reviews the recent progress of artificial intelligence for colonoscopy from detection to diagnosis. The source of data was 27 original studies in PubMed. The search terms were "colonoscopy" (title) and "deep learning" (abstract). The eligibility criteria were: (1) the dependent variable of gastrointestinal disease; (2) the interventions of deep learning for classification, detection and/or segmentation for colonoscopy; (3) the outcomes of accuracy, sensitivity, specificity, area under the curve (AUC), precision, F1, intersection of union (IOU), Dice and/or inference frames per second (FPS); (3) the publication year of 2021 or later; (4) the publication language of English. Based on the results of this study, different deep learning methods would be appropriate for different tasks for colonoscopy, e.g., Efficientnet with neural architecture search (AUC 99.8%) in the case of classification, You Only Look Once with the instance tracking head (F1 96.3%) in the case of detection, and Unet with dense-dilation-residual blocks (Dice 97.3%) in the case of segmentation. Their performance measures reported varied within 74.0-95.0% for accuracy, 60.0-93.0% for sensitivity, 60.0-100.0% for specificity, 71.0-99.8% for the AUC, 70.1-93.3% for precision, 81.0-96.3% for F1, 57.2-89.5% for the IOU, 75.1-97.3% for Dice and 66-182 for FPS. In conclusion, artificial intelligence provides an effective, non-invasive decision support system for colonoscopy from detection to diagnosis.

REAL-Colon: A dataset for developing real-world AI applications in colonoscopy

Detection and diagnosis of colon polyps are key to preventing colorectal cancer. Recent evidence suggests that AI-based computer-aided detection (CADe) and computer-aided diagnosis (CADx) systems can enhance endoscopists' performance and boost colonoscopy effectiveness. However, most available public datasets primarily consist of still images or video clips, often at a down-sampled resolution, and do not accurately represent real-world colonoscopy procedures. We introduce the REAL-Colon (Real-world multi-center Endoscopy Annotated video Library) dataset: a compilation of 2.7 M native video frames from sixty full-resolution, real-world colonoscopy recordings across multiple centers. The dataset contains 350k bounding-box annotations, each created under the supervision of expert gastroenterologists. Comprehensive patient clinical data, colonoscopy acquisition information, and polyp histopathological information are also included in each video. With its unprecedented size, quality, and heterogeneity, the REAL-Colon dataset is a unique resource for researchers and developers aiming to advance AI research in colonoscopy. Its openness and transparency facilitate rigorous and reproducible research, fostering the development and benchmarking of more accurate and reliable colonoscopy-related algorithms and models.

A New Dataset and Comparative Study for Aphid Cluster Detection and Segmentation in Sorghum Fields

Aphid infestations are one of the primary causes of extensive damage to wheat and sorghum fields and are one of the most common vectors for plant viruses, resulting in significant agricultural yield losses. To address this problem, farmers often employ the inefficient use of harmful chemical pesticides that have negative health and environmental impacts. As a result, a large amount of pesticide is wasted on areas without significant pest infestation. This brings to attention the urgent need for an intelligent autonomous system that can locate and spray sufficiently large infestations selectively within the complex crop canopies. We have developed a large multi-scale dataset for aphid cluster detection and segmentation, collected from actual sorghum fields and meticulously annotated to include clusters of aphids. Our dataset comprises a total of 54,742 image patches, showcasing a variety of viewpoints, diverse lighting conditions, and multiple scales, highlighting its effectiveness for real-world applications. In this study, we trained and evaluated four real-time semantic segmentation models and three object detection models specifically for aphid cluster segmentation and detection. Considering the balance between accuracy and efficiency, Fast-SCNN delivered the most effective segmentation results, achieving 80.46% mean precision, 81.21% mean recall, and 91.66 frames per second (FPS). For object detection, RT-DETR exhibited the best overall performance with a 61.63% mean average precision (mAP), 92.6% mean recall, and 72.55 on an NVIDIA V100 GPU. Our experiments further indicate that aphid cluster segmentation is more suitable for assessing aphid infestations than using detection models.

Two-stage deep-learning-based colonoscopy polyp detection incorporating fisheye and reflection correction

BACKGROUND AND AIM

Colonoscopy is a useful method for the diagnosis and management of colorectal diseases. Many computer-aided systems have been developed to assist clinicians in detecting colorectal lesions by analyzing colonoscopy images. However, fisheye-lens distortion and light reflection in colonoscopy images can substantially affect the clarity of these images and their utility in detecting polyps. This study proposed a two-stage deep-learning model to correct distortion and reflections in colonoscopy images and thus facilitate polyp detection.

METHODS

Images were collected from the PolypSet dataset, the Kvasir-SEG dataset, and one medical center's patient archiving and communication system. The training, validation, and testing datasets comprised 808, 202, and 1100 images, respectively. The first stage involved the correction of fisheye-related distortion in colonoscopy images and polyp detection, which was performed using a convolutional neural network. The second stage involved the use of generative and adversarial networks for correcting reflective colonoscopy images before the convolutional neural network was used for polyp detection.

RESULTS

The model had higher accuracy when it was validated using corrected images than when it was validated using uncorrected images (96.8% vs 90.8%, P < 0.001). The model's accuracy in detecting polyps in the Kvasir-SEG dataset reached 96%, and the area under the receiver operating characteristic curve was 0.94.

CONCLUSION

The proposed model can facilitate the clinical diagnosis of colorectal polyps and improve the quality of colonoscopy.

TFCNet: A texture-aware and fine-grained feature compensated polyp detection network

PURPOSE

Abnormal tissue detection is a prerequisite for medical image analysis and computer-aided diagnosis and treatment. The use of neural networks (CNN) to achieve accurate detection of intestinal polyps is beneficial to the early diagnosis and treatment of colorectal cancer. Currently, image detection models using multi-scale feature processing perform well in polyp detection. However, these methods do not fully consider the misalignment of information in the process of feature scale change, resulting in the loss of fine-grained features, and eventually cause the missed and false detection of targets.

METHOD

To solve this problem, a texture-aware and fine-grained feature compensated polyp detection network (TFCNet) is proposed in this paper. Firstly, design Texture Awareness Module (TAM) to excavate the rich texture information from the low-level layers and utilize high-level semantic information for background suppression, thereby capturing purer fine-grained features. Secondly, the Texture Feature Enhancement Module (TFEM) is designed to enhance the low-level texture information in TAM, and the enhanced texture features were fused with the high-level features. By making full use of the low-level texture features and multi-scale context information, the semantic consistency and integrity of the features were ensured. Finally, the Residual Pyramid Splittable Attention Module (RPSA) is designed to balance the loss of channel information caused by skip connections, and further improve the detection performance of the network.

RESULTS

Experimental results on 4 datasets demonstrate that the TFCNet network outperforms existing methods. Particularly, on the large dataset PolypSets, the mAP@0.5-0.95 has been improved to 88.9%. On the small datasets CVC-ClinicDB and Kvasir, the mAP@0.5-0.95 is increased by 2% and 1.6%, respectively, compared to the baseline, showcasing a significant superiority over competing methods.

CRPU-NET: a deep learning model based semantic segmentation for the detection of colorectal polyp in lower gastrointestinal tract

Purpose. The objectives of the proposed work are twofold. Firstly, to develop a specialized light weight CRPU-Net for the segmentation of polyps in colonoscopy images. Secondly, to conduct a comparative analysis of the performance of CRPU-Net with implemented state-of-the-art models.Methods. We have utilized two distinct colonoscopy image datasets such as CVC-ColonDB and CVC-ClinicDB. This paper introduces the CRPU-Net, a novel approach for the automated segmentation of polyps in colorectal regions. A comprehensive series of experiments was conducted using the CRPU-Net, and its performance was compared with that of state-of-the-art models such as VGG16, VGG19, U-Net and ResUnet++. Additional analysis such as ablation study, generalizability test and 5-fold cross validation were performed.Results. The CRPU-Net achieved the segmentation accuracy of 96.42% compared to state-of-the-art model like ResUnet++ (90.91%). The Jaccard coefficient of 93.96% and Dice coefficient of 95.77% was obtained by comparing the segmentation performance of the CRPU-Net with ground truth.Conclusion. The CRPU-Net exhibits outstanding performance in Segmentation of polyp and holds promise for integration into colonoscopy devices enabling efficient operation.

Aphid cluster recognition and detection in the wild using deep learning models

Aphid infestation poses a significant threat to crop production, rural communities, and global food security. While chemical pest control is crucial for maximizing yields, applying chemicals across entire fields is both environmentally unsustainable and costly. Hence, precise localization and management of aphids are essential for targeted pesticide application. The paper primarily focuses on using deep learning models for detecting aphid clusters. We propose a novel approach for estimating infection levels by detecting aphid clusters. To facilitate this research, we have captured a large-scale dataset from sorghum fields, manually selected 5447 images containing aphids, and annotated each individual aphid cluster within these images. To facilitate the use of machine learning models, we further process the images by cropping them into patches, resulting in a labeled dataset comprising 151,380 image patches. Then, we implemented and compared the performance of four state-of-the-art object detection models (VFNet, GFLV2, PAA, and ATSS) on the aphid dataset. Extensive experimental results show that all models yield stable similar performance in terms of average precision and recall. We then propose to merge close neighboring clusters and remove tiny clusters caused by cropping, and the performance is further boosted by around 17%. The study demonstrates the feasibility of automatically detecting and managing insects using machine learning models. The labeled dataset will be made openly available to the research community.

Gender, Smoking History, and Age Prediction from Laryngeal Images

Flexible laryngoscopy is commonly performed by otolaryngologists to detect laryngeal diseases and to recognize potentially malignant lesions. Recently, researchers have introduced machine learning techniques to facilitate automated diagnosis using laryngeal images and achieved promising results. The diagnostic performance can be improved when patients' demographic information is incorporated into models. However, the manual entry of patient data is time-consuming for clinicians. In this study, we made the first endeavor to employ deep learning models to predict patient demographic information to improve the detector model's performance. The overall accuracy for gender, smoking history, and age was 85.5%, 65.2%, and 75.9%, respectively. We also created a new laryngoscopic image set for the machine learning study and benchmarked the performance of eight classical deep learning models based on CNNs and Transformers. The results can be integrated into current learning models to improve their performance by incorporating the patient's demographic information.

Modeling the survival of colorectal cancer patients based on colonoscopic features in a feature ensemble vision transformer

The prognosis of patients with colorectal cancer (CRC) mostly relies on the classic tumor node metastasis (TNM) staging classification. A more accurate and convenient prediction model would provide a better prognosis and assist in treatment. From May 2014 to December 2017, patients who underwent an operation for CRC were enrolled. The proposed feature ensemble vision transformer (FEViT) used ensemble classifiers to benefit the combinations of relevant colonoscopy features from the pretrained vision transformer and clinical features, including sex, age, family history of CRC, and tumor location, to establish the prognostic model. A total of 1729 colonoscopy images were enrolled in the current retrospective study. For the prediction of patient survival, FEViT achieved an accuracy of 94 % with an area under the receiver operating characteristic curve of 0.93, which was better than the TNM staging classification (90 %, 0.83) in the experiment. FEViT reduced the limited receptive field and gradient disappearance in the conventional convolutional neural network and was a relatively effective and efficient procedure. The promising accuracy of FEViT in modeling survival makes the prognosis of CRC patients more predictable and practical.

Negative Samples for Improving Object Detection-A Case Study in AI-Assisted Colonoscopy for Polyp Detection

Deep learning object-detection models are being successfully applied to develop computer-aided diagnosis systems for aiding polyp detection during colonoscopies. Here, we evidence the need to include negative samples for both (i) reducing false positives during the polyp-finding phase, by including images with artifacts that may confuse the detection models (e.g., medical instruments, water jets, feces, blood, excessive proximity of the camera to the colon wall, blurred images, etc.) that are usually not included in model development datasets, and (ii) correctly estimating a more realistic performance of the models. By retraining our previously developed YOLOv3-based detection model with a dataset that includes 15% of additional not-polyp images with a variety of artifacts, we were able to generally improve its F1 performance in our internal test datasets (from an average F1 of 0.869 to 0.893), which now include such type of images, as well as in four public datasets that include not-polyp images (from an average F1 of 0.695 to 0.722).

Role of artificial intelligence in risk prediction, prognostication, and therapy response assessment in colorectal cancer: current state and future directions

Artificial Intelligence (AI) is a branch of computer science that utilizes optimization, probabilistic and statistical approaches to analyze and make predictions based on a vast amount of data. In recent years, AI has revolutionized the field of oncology and spearheaded novel approaches in the management of various cancers, including colorectal cancer (CRC). Notably, the applications of AI to diagnose, prognosticate, and predict response to therapy in CRC, is gaining traction and proving to be promising. There have also been several advancements in AI technologies to help predict metastases in CRC and in Computer-Aided Detection (CAD) Systems to improve miss rates for colorectal neoplasia. This article provides a comprehensive review of the role of AI in predicting risk, prognosis, and response to therapies among patients with CRC.

Intraprocedure Artificial Intelligence Alert System for Colonoscopy Examination

Colonoscopy is a valuable tool for preventing and reducing the incidence and mortality of colorectal cancer. Although several computer-aided colorectal polyp detection and diagnosis systems have been proposed for clinical application, many remain susceptible to interference problems, including low image clarity, unevenness, and low accuracy for the analysis of dynamic images; these drawbacks affect the robustness and practicality of these systems. This study proposed an intraprocedure alert system for colonoscopy examination developed on the basis of deep learning. The proposed system features blurred image detection, foreign body detection, and polyp detection modules facilitated by convolutional neural networks. The training and validation datasets included high-quality images and low-quality images, including blurred images and those containing folds, fecal matter, and opaque water. For the detection of blurred images and images containing folds, fecal matter, and opaque water, the accuracy rate was 96.2%. Furthermore, the study results indicated a per-polyp detection accuracy of 100% when the system was applied to video images. The recall rates for high-quality image frames and polyp image frames were 95.7% and 92%, respectively. The overall alert accuracy rate and the false-positive rate of low quality for video images obtained through per-frame analysis were 95.3% and 0.18%, respectively. The proposed system can be used to alert colonoscopists to the need to slow their procedural speed or to perform flush or lumen inflation in cases where the colonoscope is being moved too rapidly, where fecal residue is present in the intestinal tract, or where the colon has been inadequately distended.

N-Net: Lesion region segmentations using the generalized hybrid dilated convolutions for polyps in colonoscopy images

Colorectal cancer is the cancer with the second highest and the third highest incidence rates for the female and the male, respectively. Colorectal polyps are potential prognostic indicators of colorectal cancer, and colonoscopy is the gold standard for the biopsy and the removal of colorectal polyps. In this scenario, one of the main concerns is to ensure the accuracy of lesion region identifications. However, the missing rate of polyps through manual observations in colonoscopy can reach 14%–30%. In this paper, we focus on the identifications of polyps in clinical colonoscopy images and propose a new N-shaped deep neural network (N-Net) structure to conduct the lesion region segmentations. The encoder-decoder framework is adopted in the N-Net structure and the DenseNet modules are implemented in the encoding path of the network. Moreover, we innovatively propose the strategy to design the generalized hybrid dilated convolution (GHDC), which enables flexible dilated rates and convolutional kernel sizes, to facilitate the transmission of the multi-scale information with the respective fields expanded. Based on the strategy of GHDC designing, we design four GHDC blocks to connect the encoding and the decoding paths. Through the experiments on two publicly available datasets on polyp segmentations of colonoscopy images: the Kvasir-SEG dataset and the CVC-ClinicDB dataset, the rationality and superiority of the proposed GHDC blocks and the proposed N-Net are verified. Through the comparative studies with the state-of-the-art methods, such as TransU-Net, DeepLabV3+ and CA-Net, we show that even with a small amount of network parameters, the N-Net outperforms with the Dice of 94.45%, the average symmetric surface distance (ASSD) of 0.38 pix and the mean intersection-over-union (mIoU) of 89.80% on the Kvasir-SEG dataset, and with the Dice of 97.03%, the ASSD of 0.16 pix and the mIoU of 94.35% on the CVC-ClinicDB dataset.

Dynamic Label Assignment for Object Detection by Combining Predicted IoUs and Anchor IoUs

Label assignment plays a significant role in modern object detection models. Detection models may yield totally different performances with different label assignment strategies. For anchor-based detection models, the IoU (Intersection over Union) threshold between the anchors and their corresponding ground truth bounding boxes is the key element since the positive samples and negative samples are divided by the IoU threshold. Early object detectors simply utilize the fixed threshold for all training samples, while recent detection algorithms focus on adaptive thresholds based on the distribution of the IoUs to the ground truth boxes. In this paper, we introduce a simple while effective approach to perform label assignment dynamically based on the training status with predictions. By introducing the predictions in label assignment, more high-quality samples with higher IoUs to the ground truth objects are selected as the positive samples, which could reduce the discrepancy between the classification scores and the IoU scores, and generate more high-quality boundary boxes. Our approach shows improvements in the performance of the detection models with the adaptive label assignment algorithm and lower bounding box losses for those positive samples, indicating more samples with higher-quality predicted boxes are selected as positives.

Examining the Factors That Affect the Diagnosis of Patients with Positive Fecal Occult Blood Test Results

Due to the threat of colorectal cancer (CRC) to health, Taiwan included the fecal occult blood test (FOBT) under preventive health services in 2010. We examined the factors that affect the diagnosis of people with positive FOBT results. Data were retrospectively collected from the CRC screening database. In the model predicting factors that affect the diagnosis of 89,046 people with positive FOBT results, the risks of disease in the CRC group were lower in medical institutions that conducted follow-up examinations in regions such as Northern Taiwan compared to that in Eastern Taiwan (p = 0.013); they were lower in the age group of 50 to 65 years than those in the age group of 71 to 75 years (p < 0.001, p = 0.016), and lower in the outpatient medical units that conducted follow-up examinations than those in the inpatient medical units by 0.565 times (p < 0.001, 95% CI: 0.493−0.647). Factors affecting the diagnosis of patients with positive FOBT results were gender, the region of the medical institution, medical unit for follow-up examinations, age, screening site, family history, type of follow-up examinations, and follow-up time. Therefore, the identification of characteristics of patients with positive FOBT results and the promotion of follow-up examination are important prevention strategies for CRC.

Performance of Convolutional Neural Networks for Polyp Localization on Public Colonoscopy Image Datasets

Colorectal cancer is one of the most frequent malignancies. Colonoscopy is the de facto standard for precancerous lesion detection in the colon, i.e., polyps, during screening studies or after facultative recommendation. In recent years, artificial intelligence, and especially deep learning techniques such as convolutional neural networks, have been applied to polyp detection and localization in order to develop real-time CADe systems. However, the performance of machine learning models is very sensitive to changes in the nature of the testing instances, especially when trying to reproduce results for totally different datasets to those used for model development, i.e., inter-dataset testing. Here, we report the results of testing of our previously published polyp detection model using ten public colonoscopy image datasets and analyze them in the context of the results of other 20 state-of-the-art publications using the same datasets. The F1-score of our recently published model was 0.88 when evaluated on a private test partition, i.e., intra-dataset testing, but it decayed, on average, by 13.65% when tested on ten public datasets. In the published research, the average intra-dataset F1-score is 0.91, and we observed that it also decays in the inter-dataset setting to an average F1-score of 0.83.

A Domain Gap Aware Generative Adversarial Network for Multi-Domain Image Translation

Recent image-to-image translation models have shown great success in mapping local textures between two domains. Existing approaches rely on a cycle-consistency constraint that supervises the generators to learn an inverse mapping. However, learning the inverse mapping introduces extra trainable parameters and it is unable to learn the inverse mapping for some domains. As a result, they are ineffective in the scenarios where (i) multiple visual image domains are involved; (ii) both structure and texture transformations are required; and (iii) semantic consistency is preserved. To solve these challenges, the paper proposes a unified model to translate images across multiple domains with significant domain gaps. Unlike previous models that constrain the generators with the ubiquitous cycle-consistency constraint to achieve the content similarity, the proposed model employs a perceptual self-regularization constraint. With a single unified generator, the model can maintain consistency over the global shapes as well as the local texture information across multiple domains. Extensive qualitative and quantitative evaluations demonstrate the effectiveness and superior performance over state-of-the-art models. It is more effective in representing shape deformation in challenging mappings with significant dataset variation across multiple domains.

Deep Learning Approaches to Colorectal Cancer Diagnosis: A Review

Unprecedented breakthroughs in the development of graphical processing systems have led to great potential for deep learning (DL) algorithms in analyzing visual anatomy from high-resolution medical images. Recently, in digital pathology, the use of DL technologies has drawn a substantial amount of attention for use in the effective diagnosis of various cancer types, especially colorectal cancer (CRC), which is regarded as one of the dominant causes of cancer-related deaths worldwide. This review provides an in-depth perspective on recently published research articles on DL-based CRC diagnosis and prognosis. Overall, we provide a retrospective synopsis of simple image-processing-based and machine learning (ML)-based computer-aided diagnosis (CAD) systems, followed by a comprehensive appraisal of use cases with different types of state-of-the-art DL algorithms for detecting malignancies. We first list multiple standardized and publicly available CRC datasets from two imaging types: colonoscopy and histopathology. Secondly, we categorize the studies based on the different types of CRC detected (tumor tissue, microsatellite instability, and polyps), and we assess the data preprocessing steps and the adopted DL architectures before presenting the optimum diagnostic results. CRC diagnosis with DL algorithms is still in the preclinical phase, and therefore, we point out some open issues and provide some insights into the practicability and development of robust diagnostic systems in future health care and oncology.

Enhanced U-Net: A Feature Enhancement Network for Polyp Segmentation

Colonoscopy is a procedure to detect colorectal polyps which are the primary cause for developing colorectal cancer. However, polyp segmentation is a challenging task due to the diverse shape, size, color, and texture of polyps, shuttle difference between polyp and its background, as well as low contrast of the colonoscopic images. To address these challenges, we propose a feature enhancement network for accurate polyp segmentation in colonoscopy images. Specifically, the proposed network enhances the semantic information using the novel Semantic Feature Enhance Module (SFEM). Furthermore, instead of directly adding encoder features to the respective decoder layer, we introduce an Adaptive Global Context Module (AGCM), which focuses only on the encoder's significant and hard fine-grained features. The integration of these two modules improves the quality of features layer by layer, which in turn enhances the final feature representation. The proposed approach is evaluated on five colonoscopy datasets and demonstrates superior performance compared to other state-of-the-art models.