A comprehensive analysis of classification methods in gastrointestinal endoscopy imaging

Artificial intelligence alert system based on intraluminal view for colonoscopy intubation

Mucosal contact of the tip of colonoscopy causes red-out views, and more pressure may result in perforation. There is still a lack of quantitative analysis methods for red-out views. We aimed to develop an artificial intelligence (AI)-based system to assess red-out views during intubation in colonoscopy. Altogether, 479 colonoscopies performed by 34 colonoscopists were analysed using the proposed semi-supervised AI-based system. We compared the AI-based red-out avoiding scores among novice, intermediate, and experienced colonoscopists. The mean AI-based red-out avoiding scores were compared among groups stratified by expert-rated direct observation of procedure or skill (DOPS)-based tip control assessment results. Both the percentage of actual red-out views (p < 0.001) and AI-based red-out avoiding scores (p < 0.001) were significantly different among the novice, intermediate, and experienced groups. Colonoscopists who scored better on the DOPS-based tip control assessment also performed better on the AI-based red-out avoiding skill assessment. AI-based red-out avoiding score was negatively correlated with actual caecal intubation time and actual red-out percentage. Feedback of red-out avoiding score may help remind endoscopists to perform colonoscopy in an effective and safe manner. This system can be used as an auxiliary tool for colonoscopy training.

GastroFuse-Net: an ensemble deep learning framework designed for gastrointestinal abnormality detection in endoscopic images

Convolutional Neural Networks (CNNs) have received substantial attention as a highly effective tool for analyzing medical images, notably in interpreting endoscopic images, due to their capacity to provide results equivalent to or exceeding those of medical specialists. This capability is particularly crucial in the realm of gastrointestinal disorders, where even experienced gastroenterologists find the automatic diagnosis of such conditions using endoscopic pictures to be a challenging endeavor. Currently, gastrointestinal findings in medical diagnosis are primarily determined by manual inspection by competent gastrointestinal endoscopists. This evaluation procedure is labor-intensive, time-consuming, and frequently results in high variability between laboratories. To address these challenges, we introduced a specialized CNN-based architecture called GastroFuse-Net, designed to recognize human gastrointestinal diseases from endoscopic images. GastroFuse-Net was developed by combining features extracted from two different CNN models with different numbers of layers, integrating shallow and deep representations to capture diverse aspects of the abnormalities. The Kvasir dataset was used to thoroughly test the proposed deep learning model. This dataset contained images that were classified according to structures (cecum, z-line, pylorus), diseases (ulcerative colitis, esophagitis, polyps), or surgical operations (dyed resection margins, dyed lifted polyps). The proposed model was evaluated using various measures, including specificity, recall, precision, F1-score, Mathew's Correlation Coefficient (MCC), and accuracy. The proposed model GastroFuse-Net exhibited exceptional performance, achieving a precision of 0.985, recall of 0.985, specificity of 0.984, F1-score of 0.997, MCC of 0.982, and an accuracy of 98.5%.

Pathological Insights: Enhanced Vision Transformers for the Early Detection of Colorectal Cancer

Endoscopic pathological findings of the gastrointestinal tract are crucial for the early diagnosis of colorectal cancer (CRC). Previous deep learning works, aimed at improving CRC detection performance and reducing subjective analysis errors, are limited to polyp segmentation. Pathological findings were not considered and only convolutional neural networks (CNNs), which are not able to handle global image feature information, were utilized. This work introduces a novel vision transformer (ViT)-based approach for early CRC detection. The core components of the proposed approach are ViTCol, a boosted vision transformer for classifying endoscopic pathological findings, and PUTS, a vision transformer-based model for polyp segmentation. Results demonstrate the superiority of this vision transformer-based CRC detection method over existing CNN and vision transformer models. ViTCol exhibited an outstanding performance in classifying pathological findings, with an area under the receiver operating curve (AUC) value of 0.9999 ± 0.001 on the Kvasir dataset. PUTS provided outstanding results in segmenting polyp images, with mean intersection over union (mIoU) of 0.8673 and 0.9092 on the Kvasir-SEG and CVC-Clinic datasets, respectively. This work underscores the value of spatial transformers in localizing input images, which can seamlessly integrate into the main vision transformer network, enhancing the automated identification of critical image features for early CRC detection.

Real time anatomical landmarks and abnormalities detection in gastrointestinal tract

Gastrointestinal (GI) endoscopy is an active research field due to the lethal cancer diseases in the GI tract. Cancer treatments result better if diagnosed early and it increases the survival chances. There is a high miss rate in the detection of the abnormalities in the GI tract during endoscopy or colonoscopy due to the lack of attentiveness, tiring procedures, or the lack of required training. The procedure of the detection can be automated to the reduction of the risks by identifying and flagging the suspicious frames. A suspicious frame may have some of the abnormality or the information about anatomical landmark in the frame. The frame then can be analysed for the anatomical landmarks and the abnormalities for the detection of disease. In this research, a real-time endoscopic abnormalities detection system is presented that detects the abnormalities and the landmarks. The proposed system is based on a combination of handcrafted and deep features. Deep features are extracted from lightweight MobileNet convolutional neural network (CNN) architecture. There are some of the classes with a small inter-class difference and a higher intra-class differences, for such classes the same detection threshold is unable to distinguish. The threshold of such classes is learned from the training data using genetic algorithm. The system is evaluated on various benchmark datasets and resulted in an accuracy of 0.99 with the F1-score of 0.91 and Matthews correlation coefficient (MCC) of 0.91 on Kvasir datasets and F1-score of 0.93 on the dataset of DowPK. The system detects abnormalities in real-time with the detection speed of 41 frames per second.

Artificial intelligence and automation in endoscopy and surgery

Modern endoscopy relies on digital technology, from high-resolution imaging sensors and displays to electronics connecting configurable illumination and actuation systems for robotic articulation. In addition to enabling more effective diagnostic and therapeutic interventions, the digitization of the procedural toolset enables video data capture of the internal human anatomy at unprecedented levels. Interventional video data encapsulate functional and structural information about a patient's anatomy as well as events, activity and action logs about the surgical process. This detailed but difficult-to-interpret record from endoscopic procedures can be linked to preoperative and postoperative records or patient imaging information. Rapid advances in artificial intelligence, especially in supervised deep learning, can utilize data from endoscopic procedures to develop systems for assisting procedures leading to computer-assisted interventions that can enable better navigation during procedures, automation of image interpretation and robotically assisted tool manipulation. In this Perspective, we summarize state-of-the-art artificial intelligence for computer-assisted interventions in gastroenterology and surgery.

A deep ensemble learning method for colorectal polyp classification with optimized network parameters

Colorectal Cancer (CRC), a leading cause of cancer-related deaths, can be abated by timely polypectomy. Computer-aided classification of polyps helps endoscopists to resect timely without submitting the sample for histology. Deep learning-based algorithms are promoted for computer-aided colorectal polyp classification. However, the existing methods do not accommodate any information on hyperparametric settings essential for model optimisation. Furthermore, unlike the polyp types, i.e., hyperplastic and adenomatous, the third type, serrated adenoma, is difficult to classify due to its hybrid nature. Moreover, automated assessment of polyps is a challenging task due to the similarities in their patterns; therefore, the strength of individual weak learners is combined to form a weighted ensemble model for an accurate classification model by establishing the optimised hyperparameters. In contrast to existing studies on binary classification, multiclass classification require evaluation through advanced measures. This study compared six existing Convolutional Neural Networks in addition to transfer learning and opted for optimum performing architecture only for ensemble models. The performance evaluation on UCI and PICCOLO dataset of the proposed method in terms of accuracy (96.3%, 81.2%), precision (95.5%, 82.4%), recall (97.2%, 81.1%), F1-score (96.3%, 81.3%) and model reliability using Cohen’s Kappa Coefficient (0.94, 0.62) shows the superiority over existing models. The outcomes of experiments by other studies on the same dataset yielded 82.5% accuracy with 72.7% recall by SVM and 85.9% accuracy with 87.6% recall by other deep learning methods. The proposed method demonstrates that a weighted ensemble of optimised networks along with data augmentation significantly boosts the performance of deep learning-based CAD.

Forrest Classification for Bleeding Peptic Ulcer: A New Look at the Old Endoscopic Classification

The management of peptic ulcer bleeding is clinically challenging. For decades, the Forrest classification has been used for risk stratification for nonvariceal ulcer bleeding. The perception and interpretation of the Forrest classification vary among different endoscopists. The relationship between the bleeder and ulcer images and the different stages of the Forrest classification has not been studied yet. Endoscopic still images of 276 patients with peptic ulcer bleeding for the past 3 years were retrieved and reviewed. The intra-rater agreement and inter-rater agreement were compared. The obtained endoscopic images were manually drawn to delineate the extent of the ulcer and bleeding area. The areas of the region of interest were compared between the different stages of the Forrest classification. A total of 276 images were first classified by two experienced tutor endoscopists. The images were reviewed by six other endoscopists. A good intra-rater correlation was observed (0.92–0.98). A good inter-rater correlation was observed among the different levels of experience (0.639–0.859). The correlation was higher among tutor and junior endoscopists than among experienced endoscopists. Low-risk Forrest IIC and III lesions show distinct patterns compared to high-risk Forrest I, IIA, or IIB lesions. We found good agreement of the Forrest classification among different endoscopists in a single institution. This is the first study to quantitively analyze the obtained and explain the distinct patterns of bleeding ulcers from endoscopy images.