Attention by Selection: A Deep Selective Attention Approach to Breast Cancer Classification

Principles of artificial intelligence in radiooncology

PURPOSE

In the rapidly expanding field of artificial intelligence (AI) there is a wealth of literature detailing the myriad applications of AI, particularly in the realm of deep learning. However, a review that elucidates the technical principles of deep learning as relevant to radiation oncology in an easily understandable manner is still notably lacking. This paper aims to fill this gap by providing a comprehensive guide to the principles of deep learning that is specifically tailored toward radiation oncology.

METHODS

In light of the extensive variety of AI methodologies, this review selectively concentrates on the specific domain of deep learning. It emphasizes the principal categories of deep learning models and delineates the methodologies for training these models effectively.

RESULTS

This review initially delineates the distinctions between AI and deep learning as well as between supervised and unsupervised learning. Subsequently, it elucidates the fundamental principles of major deep learning models, encompassing multilayer perceptrons (MLPs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformers, generative adversarial networks (GANs), diffusion-based generative models, and reinforcement learning. For each category, it presents representative networks alongside their specific applications in radiation oncology. Moreover, the review outlines critical factors essential for training deep learning models, such as data preprocessing, loss functions, optimizers, and other pivotal training parameters including learning rate and batch size.

CONCLUSION

This review provides a comprehensive overview of deep learning principles tailored toward radiation oncology. It aims to enhance the understanding of AI-based research and software applications, thereby bridging the gap between complex technological concepts and clinical practice in radiation oncology.

ViT-MAENB7: An innovative breast cancer diagnosis model from 3D mammograms using advanced segmentation and classification process

Tumors are an important health concern in modern times. Breast cancer is one of the most prevalent causes of death for women. Breast cancer is rapidly becoming the leading cause of mortality among women globally. Early detection of breast cancer allows patients to obtain appropriate therapy, increasing their probability of survival. The adoption of 3-Dimensional (3D) mammography for the medical identification of abnormalities in the breast reduced the number of deaths dramatically. Classification and accurate detection of lumps in the breast in 3D mammography is especially difficult due to factors such as inadequate contrast and normal fluctuations in tissue density. Several Computer-Aided Diagnosis (CAD) solutions are under development to help radiologists accurately classify abnormalities in the breast. In this paper, a breast cancer diagnosis model is implemented to detect breast cancer in cancer patients to prevent death rates. The 3D mammogram images are gathered from the internet. Then, the gathered images are given to the preprocessing phase. The preprocessing is done using a median filter and image scaling method. The purpose of the preprocessing phase is to enhance the quality of the images and remove any noise or artifacts that may interfere with the detection of abnormalities. The median filter helps to smooth out any irregularities in the images, while the image scaling method adjusts the size and resolution of the images for better analysis. Once the preprocessing is complete, the preprocessed image is given to the segmentation phase. The segmentation phase is crucial in medical image analysis as it helps to identify and separate different structures within the image, such as organs or tumors. This process involves dividing the preprocessed image into meaningful regions or segments based on intensity, color, texture, or other features. The segmentation process is done using Adaptive Thresholding with Region Growing Fusion Model (AT-RGFM)". This model combines the advantages of both thresholding and region-growing techniques to accurately identify and delineate specific structures within the image. By utilizing AT-RGFM, the segmentation phase can effectively differentiate between different parts of the image, allowing for more precise analysis and diagnosis. It plays a vital role in the medical image analysis process, providing crucial insights for healthcare professionals. Here, the Modified Garter Snake Optimization Algorithm (MGSOA) is used to optimize the parameters. It helps to optimize parameters for accurately identifying and delineating specific structures within medical images and also helps healthcare professionals in providing more precise analysis and diagnosis, ultimately playing a vital role in the medical image analysis process. MGSOA enhances the segmentation phase by effectively differentiating between different parts of the image, leading to more accurate results. Then, the segmented image is fed into the detection phase. The tumor detection is performed by the Vision Transformer-based Multiscale Adaptive EfficientNetB7 (ViT-MAENB7) model. This model utilizes a combination of advanced algorithms and deep learning techniques to accurately identify and locate tumors within the segmented medical image. By incorporating a multiscale adaptive approach, the ViT-MAENB7 model can analyze the image at various levels of detail, improving the overall accuracy of tumor detection. This crucial step in the medical image analysis process allows healthcare professionals to make more informed decisions regarding patient treatment and care. Here, the created MGSOA algorithm is used to optimize the parameters for enhancing the performance of the model. The suggested breast cancer diagnosis performance is compared to conventional cancer diagnosis models and it showed high accuracy. The accuracy of the developed MGSOA-ViT-MAENB7 is 96.6 %, and others model like RNN, LSTM, EffNet, and ViT-MAENet given the accuracy to be 90.31 %, 92.79 %, 94.46 % and 94.75 %. The developed model's ability to analyze images at multiple scales, combined with the optimization provided by the MGSOA algorithm, results in a highly accurate and efficient system for detecting tumors in medical images. This cutting-edge technology not only improves the accuracy of diagnosis but also helps healthcare professionals tailor treatment plans to individual patients, ultimately leading to better outcomes. By outperforming traditional cancer diagnosis models, the proposed model is revolutionizing the field of medical imaging and setting a new standard for precision and effectiveness in healthcare.

Dual attention model with reinforcement learning for classification of histology whole-slide images

Digital whole slide images (WSIs) are generally captured at microscopic resolution and encompass extensive spatial data (several billions of pixels per image). Directly feeding these images to deep learning models is computationally intractable due to memory constraints, while downsampling the WSIs risks incurring information loss. Alternatively, splitting the WSIs into smaller patches (or tiles) may result in a loss of important contextual information. In this paper, we propose a novel dual attention approach, consisting of two main components, both inspired by the visual examination process of a pathologist: The first soft attention model processes a low magnification view of the WSI to identify relevant regions of interest (ROIs), followed by a custom sampling method to extract diverse and spatially distinct image tiles from the selected ROIs. The second component, the hard attention classification model further extracts a sequence of multi-resolution glimpses from each tile for classification. Since hard attention is non-differentiable, we train this component using reinforcement learning to predict the location of the glimpses. This approach allows the model to focus on essential regions instead of processing the entire tile, thereby aligning with a pathologist's way of diagnosis. The two components are trained in an end-to-end fashion using a joint loss function to demonstrate the efficacy of the model. The proposed model was evaluated on two WSI-level classification problems: Human epidermal growth factor receptor 2 (HER2) scoring on breast cancer histology images and prediction of Intact/Loss status of two Mismatch Repair (MMR) biomarkers from colorectal cancer histology images. We show that the proposed model achieves performance better than or comparable to the state-of-the-art methods while processing less than 10% of the WSI at the highest magnification and reducing the time required to infer the WSI-level label by more than 75%. The code is available at github.

A survey on cancer detection via convolutional neural networks: Current challenges and future directions

Cancer is a condition in which abnormal cells uncontrollably split and damage the body tissues. Hence, detecting cancer at an early stage is highly essential. Currently, medical images play an indispensable role in detecting various cancers; however, manual interpretation of these images by radiologists is observer-dependent, time-consuming, and tedious. An automatic decision-making process is thus an essential need for cancer detection and diagnosis. This paper presents a comprehensive survey on automated cancer detection in various human body organs, namely, the breast, lung, liver, prostate, brain, skin, and colon, using convolutional neural networks (CNN) and medical imaging techniques. It also includes a brief discussion about deep learning based on state-of-the-art cancer detection methods, their outcomes, and the possible medical imaging data used. Eventually, the description of the dataset used for cancer detection, the limitations of the existing solutions, future trends, and challenges in this domain are discussed. The utmost goal of this paper is to provide a piece of comprehensive and insightful information to researchers who have a keen interest in developing CNN-based models for cancer detection.

A Hierarchical Graph V-Net With Semi-Supervised Pre-Training for Histological Image Based Breast Cancer Classification

Numerous patch-based methods have recently been proposed for histological image based breast cancer classification. However, their performance could be highly affected by ignoring spatial contextual information in the whole slide image (WSI). To address this issue, we propose a novel hierarchical Graph V-Net by integrating 1) patch-level pre-training and 2) context-based fine-tuning, with a hierarchical graph network. Specifically, a semi-supervised framework based on knowledge distillation is first developed to pre-train a patch encoder for extracting disease-relevant features. Then, a hierarchical Graph V-Net is designed to construct a hierarchical graph representation from neighboring/similar individual patches for coarse-to-fine classification, where each graph node (corresponding to one patch) is attached with extracted disease-relevant features and its target label during training is the average label of all pixels in the corresponding patch. To evaluate the performance of our proposed hierarchical Graph V-Net, we collect a large WSI dataset of 560 WSIs, with 30 labeled WSIs from the BACH dataset (through our further refinement), 30 labeled WSIs and 500 unlabeled WSIs from Yunnan Cancer Hospital. Those 500 unlabeled WSIs are employed for patch-level pre-training to improve feature representation, while 60 labeled WSIs are used to train and test our proposed hierarchical Graph V-Net. Both comparative assessment and ablation studies demonstrate the superiority of our proposed hierarchical Graph V-Net over state-of-the-art methods in classifying breast cancer from WSIs. The source code and our annotations for the BACH dataset have been released at https://github.com/lyhkevin/Graph-V-Net.

A collaborative multi-task learning method for BI-RADS category 4 breast lesion segmentation and classification of MRI images

BACKGROUND AND OBJECTIVE

The diagnosis of BI-RADS category 4 breast lesion is difficult because its probability of malignancy ranges from 2% to 95%. For BI-RADS category 4 breast lesions, MRI is one of the prominent noninvasive imaging techniques. In this paper, we research computer algorithms to segment lesions and classify the benign or malignant lesions in MRI images. However, this task is challenging because the BI-RADS category 4 lesions are characterized by irregular shape, imbalanced class, and low contrast.

METHODS

We fully utilize the intrinsic correlation between segmentation and classification tasks, where accurate segmentation will yield accurate classification results, and classification results will promote better segmentation. Therefore, we propose a collaborative multi-task algorithm (CMTL-SC). Specifically, a preliminary segmentation subnet is designed to identify the boundaries, locations and segmentation masks of lesions; a classification subnet, which combines the information provided by the preliminary segmentation, is designed to achieve benign or malignant classification; a repartition segmentation subnet which aggregates the benign or malignant results, is designed to refine the lesion segment. The three subnets work cooperatively so that the CMTL-SC can identify the lesions better which solves the three challenges.

RESULTS AND CONCLUSION

We collect MRI data from 248 patients in the Second Hospital of Dalian Medical University. The results show that the lesion boundaries delineated by the CMTL-SC are close to the boundaries delineated by the physicians. Moreover, the CMTL-SC yields better results than the single-task and multi-task state-of-the-art algorithms. Therefore, CMTL-SC can help doctors make precise diagnoses and refine treatments for patients.

Meta-adaptive-weighting-based bilateral multi-dimensional refined space feature attention network for imbalanced breast cancer histopathological image classification

Breast cancer histopathological image automatic classification can reduce pathologists workload and provide accurate diagnosis. However, one challenge is that empirical datasets are usually imbalanced, resulting in poorer classification quality compared with conventional methods based on balanced datasets. The recently proposed bilateral branch network (BBN) tackles this problem through considering both representation and classifier learning to improve classification performance. We firstly apply bilateral sampling strategy to imbalanced breast cancer histopathological image classification and propose a meta-adaptive-weighting-based bilateral multi-dimensional refined space feature attention network (MAW-BMRSFAN). The model is composed of BMRSFAN and MAWN. Specifically, the refined space feature attention module (RSFAM) is based on convolutional long short-term memories (ConvLSTMs). It is designed to extract refined spatial features of different dimensions for image classification and is inserted into different layers of classification model. Meanwhile, the MAWN is proposed to model the mapping from a balanced meta-dataset to imbalanced dataset. It finds suitable weighting parameter for BMRSFAN more flexibly through adaptively learning from a small amount of balanced dataset directly. The experiments show that MAW-BMRSFAN performs better than previous methods. The recognition accuracy of MAW-BMRSFAN under four different magnifications still is higher than 80% even when unbalance factor is 16, indicating that MAW-BMRSFAN can make ideal performance under extreme imbalanced conditions.

Learning how to detect: A deep reinforcement learning method for whole-slide melanoma histopathology images

Cutaneous melanoma represents one of the most life-threatening malignancies. Histopathological image analysis serves as a vital tool for early melanoma detection. Deep neural network (DNN) models are frequently employed to aid pathologists in enhancing the efficiency and accuracy of diagnoses. However, due to the paucity of well-annotated, high-resolution, whole-slide histopathology image (WSI) datasets, WSIs are typically fragmented into numerous patches during the model training and testing stages. This process disregards the inherent interconnectedness among patches, potentially impeding the models' performance. Additionally, the presence of excess, non-contributing patches extends processing times and introduces substantial computational burdens. To mitigate these issues, we draw inspiration from the clinical decision-making processes of dermatopathologists to propose an innovative, weakly supervised deep reinforcement learning framework, titled Fast medical decision-making in melanoma histopathology images (FastMDP-RL). This framework expedites model inference by reducing the number of irrelevant patches identified within WSIs. FastMDP-RL integrates two DNN-based agents: the search agent (SeAgent) and the decision agent (DeAgent). The SeAgent initiates actions, steered by the image features observed in the current viewing field at various magnifications. Simultaneously, the DeAgent provides labeling probabilities for each patch. We utilize multi-instance learning (MIL) to construct a teacher-guided model (MILTG), serving a dual purpose: rewarding the SeAgent and guiding the DeAgent. Our evaluations were conducted using two melanoma datasets: the publicly accessible TCIA-CM dataset and the proprietary MELSC dataset. Our experimental findings affirm FastMDP-RL's ability to expedite inference and accurately predict WSIs, even in the absence of pixel-level annotations. Moreover, our research investigates the WSI-based interactive environment, encompassing the design of agents, state and reward functions, and feature extractors suitable for melanoma tissue images. This investigation offers valuable insights and references for researchers engaged in related studies. The code is available at: https://github.com/titizheng/FastMDP-RL.

Improved breast ultrasound tumor classification using dual-input CNN with GAP-guided attention loss

Ultrasonography is a widely used medical imaging technique for detecting breast cancer. While manual diagnostic methods are subject to variability and time-consuming, computer-aided diagnostic (CAD) methods have proven to be more efficient. However, current CAD approaches neglect the impact of noise and artifacts on the accuracy of image analysis. To enhance the precision of breast ultrasound image analysis for identifying tissues, organs and lesions, we propose a novel approach for improved tumor classification through a dual-input model and global average pooling (GAP)-guided attention loss function. Our approach leverages a convolutional neural network with transformer architecture and modifies the single-input model for dual-input. This technique employs a fusion module and GAP operation-guided attention loss function simultaneously to supervise the extraction of effective features from the target region and mitigate the effect of information loss or redundancy on misclassification. Our proposed method has three key features: (i) ResNet and MobileViT are combined to enhance local and global information extraction. In addition, a dual-input channel is designed to include both attention images and original breast ultrasound images, mitigating the impact of noise and artifacts in ultrasound images. (ii) A fusion module and GAP operation-guided attention loss function are proposed to improve the fusion of dual-channel feature information, as well as supervise and constrain the weight of the attention mechanism on the fused focus region. (iii) Using the collected uterine fibroid ultrasound dataset to train ResNet18 and load the pre-trained weights, our experiments on the BUSI and BUSC public datasets demonstrate that the proposed method outperforms some state-of-the-art methods. The code will be publicly released at https://github.com/425877/Improved-Breast-Ultrasound-Tumor-Classification.

Classification of breast tumors by using a novel approach based on deep learning methods and feature selection

PURPOSE

Cancer is one of the most insidious diseases that the most important factor in overcoming the cancer is early diagnosis and detection. The histo-pathological images are used to determine whether the tissue is cancerous and the type of cancer. As the result of examination on tissue images by the expert personnel, the cancer type, and stage of the tissue can be determined. However, this situation can cause both time and energy loss as well as personnel-related inspection errors. By the increased usage of computer-based decision methods in the last decades, it would be more efficient and accurate to detect and classify the cancerous tissues with computer-aided systems.

METHODS

As classical image processing methods were used for cancer-type detection in early studies, advanced deep learning methods based on recurrent neural networks and convolutional neural networks have been used more recently. In this paper, popular deep learning methods such as ResNet-50, GoogLeNet, InceptionV3, and MobilNetV2 are employed by implementing novel feature selection method in order to classify cancer type on a local binary class dataset and multi-class BACH dataset.

RESULTS

The classification performance of the proposed feature selection implemented deep learning methods follows as for the local binary class dataset 98.89% and 92.17% for BACH dataset which is much better than most of the obtained results in literature.

CONCLUSION

The obtained findings on both datasets indicates that the proposed methods can detect and classify the cancerous type of a tissue with high accuracy and efficiency.

Predicting breast cancer types on and beyond molecular level in a multi-modal fashion

Accurately determining the molecular subtypes of breast cancer is important for the prognosis of breast cancer patients and can guide treatment selection. In this study, we develop a deep learning-based model for predicting the molecular subtypes of breast cancer directly from the diagnostic mammography and ultrasound images. Multi-modal deep learning with intra- and inter-modality attention modules (MDL-IIA) is proposed to extract important relations between mammography and ultrasound for this task. MDL-IIA leads to the best diagnostic performance compared to other cohort models in predicting 4-category molecular subtypes with Matthews correlation coefficient (MCC) of 0.837 (95% confidence interval [CI]: 0.803, 0.870). The MDL-IIA model can also discriminate between Luminal and Non-Luminal disease with an area under the receiver operating characteristic curve of 0.929 (95% CI: 0.903, 0.951). These results significantly outperform clinicians' predictions based on radiographic imaging. Beyond molecular-level test, based on gene-level ground truth, our method can bypass the inherent uncertainty from immunohistochemistry test. This work thus provides a noninvasive method to predict the molecular subtypes of breast cancer, potentially guiding treatment selection for breast cancer patients and providing decision support for clinicians.

Compatible-domain Transfer Learning for Breast Cancer Classification with Limited Annotated Data

Microscopic analysis of breast cancer images is the primary task in diagnosing cancer malignancy. Recent attempts to automate this task have employed deep learning models whose success has depended on large volumes of data, while acquiring annotated data in biomedical domains is time-consuming and may not always be feasible. A typical strategy to address this is to apply transfer learning using pre-trained models on a large natural image database (e.g., ImageNet) instead of training a model from scratch. This approach, however, has not been effective in several previous studies due to fundamental differences between natural and medical images. In this study, for the first time we proposed the idea of using a compatible data set of histopathological images to classify breast cancer cytological biopsy specimens. Despite intrinsic differences between histopathological and cytological images, we demonstrate that the features learned by deep networks during the pre-training procedure are compatible with those obtained throughout fine-tuning process. To thoroughly investigate this assertion, we explore three different strategies for training as well as two different approaches for fine-tuning deep learning models. By comparing the obtained results with those of previous state-of-the-art research conducted on the same data set, we demonstrate that the proposed method boasts of improved classification accuracy by 6% to 17% compared to the studies which were based on traditional machine learning techniques, and also enhanced accuracy by roughly 7% compared to those who utilized deep learning methods, eventually achieving 98.73% validation accuracy and 94.55% test accuracy. Exploring different training scenarios also revealed that using a compatible dataset has helped to elevate the classification accuracy by 3.0% compared to the typical approach of using ImageNet. Experimental results show that our approach, despite using a very small number of training images, has achieved performance comparable to that of experienced pathologists and has the potential to be applied in clinical settings.

Colorectal cancer lymph node metastasis prediction with weakly supervised transformer-based multi-instance learning

Lymph node metastasis examined by the resected lymph nodes is considered one of the most important prognostic factors for colorectal cancer (CRC). However, it requires careful and comprehensive inspection by expert pathologists. To relieve the pathologists' burden and speed up the diagnostic process, in this paper, we develop a deep learning system with the binary positive/negative labels of the lymph nodes to solve the CRC lymph node classification task. The multi-instance learning (MIL) framework is adopted in our method to handle the whole slide images (WSIs) of gigapixels in size at once and get rid of the labor-intensive and time-consuming detailed annotations. First, a transformer-based MIL model, DT-DSMIL, is proposed in this paper based on the deformable transformer backbone and the dual-stream MIL (DSMIL) framework. The local-level image features are extracted and aggregated with the deformable transformer, and the global-level image features are obtained with the DSMIL aggregator. The final classification decision is made based on both the local and the global-level features. After the effectiveness of our proposed DT-DSMIL model is demonstrated by comparing its performance with its predecessors, a diagnostic system is developed to detect, crop, and finally identify the single lymph nodes within the slides based on the DT-DSMIL and the Faster R-CNN model. The developed diagnostic model is trained and tested on a clinically collected CRC lymph node metastasis dataset composed of 843 slides (864 metastasis lymph nodes and 1415 non-metastatic lymph nodes), achieving the accuracy of 95.3% and the area under the receiver operating characteristic curve (AUC) of 0.9762 (95% confidence interval [CI]: 0.9607-0.9891) for the single lymph node classification. As for the lymph nodes with micro-metastasis and macro-metastasis, our diagnostic system achieves the AUC of 0.9816 (95% CI: 0.9659-0.9935) and 0.9902 (95% CI: 0.9787-0.9983), respectively. Moreover, the system shows reliable diagnostic region localizing performance: the model can always identify the most likely metastases, no matter the model's predictions or manual labels, showing great potential in avoiding false negatives and discovering incorrectly labeled slides in actual clinical use.

An Automatic Image Processing Method Based on Artificial Intelligence for Locating the Key Boundary Points in the Central Serous Chorioretinopathy Lesion Area

Accurately and rapidly measuring the diameter of central serous chorioretinopathy (CSCR) lesion area is the key to judge the severity of CSCR and evaluate the efficacy of the corresponding treatments. Currently, the manual measurement scheme based on a single or a small number of optical coherence tomography (OCT) B-scan images encounters the dilemma of incredibility. Although manually measuring the diameters of all OCT B-scan images of a single patient can alleviate the previous issue, the situation of inefficiency will thus arise. Additionally, manual operation is subject to subjective factors of ophthalmologists, resulting in unrepeatable measurement results. Therefore, an automatic image processing method (i.e., a joint framework) based on artificial intelligence (AI) is innovatively proposed for locating the key boundary points of CSCR lesion area to assist the diameter measurement. Firstly, the initial location module (ILM) benefiting from multitask learning is properly adjusted and tentatively achieves the preliminary location of key boundary points. Secondly, the location task is formulated as a Markov decision process, aiming at further improving the location accuracy by utilizing the single agent reinforcement learning module (SARLM). Finally, the joint framework based on the ILM and SARLM is skillfully established, in which ILM provides an initial starting point for SARLM to narrow the active region of agent, and SARLM makes up for the defect of low generalization of ILM by virtue of the independent exploration ability of agent. Experiments reveal the AI-based method which joins the multitask learning, and single agent reinforcement learning paradigms enable agents to work in local region, alleviating the time-consuming problem of SARLM, performing location task in a global scope, and improving the location accuracy of ILM, thus reflecting its effectiveness and clinical application value in the task of rapidly and accurately measuring the diameter of CSCR lesions.

Painless and accurate medical image analysis using deep reinforcement learning with task-oriented homogenized automatic pre-processing

Pre-processing is widely applied in medical image analysis to remove the interference information. However, the existing pre-processing solutions mainly encounter two problems: (i) it is heavily relied on the assistance of clinical experts, making it hard for intelligent CAD systems to deploy quickly; (ii) due to the personnel and information barriers, it is difficult for medical institutions to conduct the same pre-processing operations, making a deep model that performs well on a specific medical institution difficult to achieve similar performances on the same task in other medical institutions. To overcome these problems, we propose a deep-reinforcement-learning-based task-oriented homogenized automatic pre-processing (DRL-HAPre) framework to overcome these two problems. This framework utilizes deep reinforcement learning techniques to learn a policy network to automatically and adaptively select the optimal pre-processing operations for the input medical images according to different analysis tasks, thus helping the intelligent CAD system to achieve a rapid deployment (i.e., painless) and maintain a satisfactory performance (i.e., accurate) among different medical institutes. To verify the effectiveness and advantages of the proposed DRL-HAPre framework, we further develop a homogenized automatic pre-processing model based on the DRL-HAPre framework to realize the automatic pre-processing of key region selection (called HAPre-KRS) in the pneumonia image classification task. Extensive experimental studies are conducted on three pediatric pneumonia classification datasets with different image qualities, and the results show that: (i) There does exist a hard-to-reproduce problem in clinical practices and the fact that having different medical image qualities in different medical institutes is an important reason for the existing of hard-to-reproduce problem, so it is compelling to propose homogenized automatic pre-processing method. (ii) The proposed HAPre-KRS model and DRL-HAPre framework greatly outperform three kinds of state-of-the-art baselines (i.e., pre-processing, attention and pneumonia baseline), and the lower the medical image quality, the greater the improvements of using our HAPre-KRS model and DRL-HAPre framework. (iii) With the help of homogenized pre-processing, HAPre-KRS (and DRL-HAPre framework) can greatly avoid performance degradation in real-world cross-source applications (i.e., thus overcoming the hard-to-reproduce problem).

Reinforcement learning in medical image analysis: Concepts, applications, challenges, and future directions

MOTIVATION

Medical image analysis involves a series of tasks used to assist physicians in qualitative and quantitative analyses of lesions or anatomical structures which can significantly improve the accuracy and reliability of medical diagnoses and prognoses. Traditionally, these tedious tasks were finished by experienced physicians or medical physicists and were marred with two major problems, low efficiency and bias. In the past decade, many machine learning methods have been applied to accelerate and automate the image analysis process. Compared to the enormous deployments of supervised and unsupervised learning models, attempts to use reinforcement learning in medical image analysis are still scarce. We hope that this review article could serve as the stepping stone for related research in the future.

SIGNIFICANCE

We found that although reinforcement learning has gradually gained momentum in recent years, many researchers in the medical analysis field still find it hard to understand and deploy in clinical settings. One possible cause is a lack of well-organized review articles intended for readers without professional computer science backgrounds. Rather than to provide a comprehensive list of all reinforcement learning models applied in medical image analysis, the aim of this review is to help the readers formulate and solve their medical image analysis research through the lens of reinforcement learning.

APPROACH & RESULTS

We selected published articles from Google Scholar and PubMed. Considering the scarcity of related articles, we also included some outstanding newest preprints. The papers were carefully reviewed and categorized according to the type of image analysis task. In this article, we first reviewed the basic concepts and popular models of reinforcement learning. Then, we explored the applications of reinforcement learning models in medical image analysis. Finally, we concluded the article by discussing the reviewed reinforcement learning approaches' limitations and possible future improvements.

EHDFL: Evolutionary hybrid domain feature learning based on windowed fast Fourier convolution pyramid for medical image classification

With the development of modern medical technology, medical image classification has played an important role in medical diagnosis and clinical practice. Medical image classification algorithms based on deep learning emerge in endlessly, and have achieved amazing results. However, most of these methods ignore the feature representation based on frequency domain, and only focus on spatial features. To solve this problem, we propose a hybrid domain feature learning (HDFL) module based on windowed fast Fourier convolution pyramid, which combines the global features with a wide range of receptive fields in frequency domain and the local features with multiple scales in spatial domain. In order to prevent frequency leakage, we construct a Windowed Fast Fourier Convolution (WFFC) structure based on Fast Fourier Convolution (FFC). In order to learn hybrid domain features, we combine ResNet, FPN, and attention mechanism to construct a hybrid domain feature learning module. In addition, a super-parametric optimization algorithm is constructed based on genetic algorithm for our classification model, so as to realize the automation of our super-parametric optimization. We evaluated the newly published medical image classification dataset MedMNIST, and the experimental results show that our method can effectively learning the hybrid domain feature information of frequency domain and spatial domain.

BIAS-3D: Brain inspired attentional search model fashioned after what and where/how pathways for target search in 3D environment

We propose a brain inspired attentional search model for target search in a 3D environment, which has two separate channels—one for the object classification, analogous to the “what” pathway in the human visual system, and the other for prediction of the next location of the camera, analogous to the “where” pathway. To evaluate the proposed model, we generated 3D Cluttered Cube datasets that consist of an image on one vertical face, and clutter or background images on the other faces. The camera goes around each cube on a circular orbit and determines the identity of the image pasted on the face. The images pasted on the cube faces were drawn from: MNIST handwriting digit, QuickDraw, and RGB MNIST handwriting digit datasets. The attentional input of three concentric cropped windows resembling the high-resolution central fovea and low-resolution periphery of the retina, flows through a Classifier Network and a Camera Motion Network. The Classifier Network classifies the current view into one of the target classes or the clutter. The Camera Motion Network predicts the camera's next position on the orbit (varying the azimuthal angle or “θ”). Here the camera performs one of three actions: move right, move left, or do not move. The Camera-Position Network adds the camera's current position (θ) into the higher features level of the Classifier Network and the Camera Motion Network. The Camera Motion Network is trained using Q-learning where the reward is 1 if the classifier network gives the correct classification, otherwise 0. Total loss is computed by adding the mean square loss of temporal difference and cross entropy loss. Then the model is trained end-to-end by backpropagating the total loss using Adam optimizer. Results on two grayscale image datasets and one RGB image dataset show that the proposed model is successfully able to discover the desired search pattern to find the target face on the cube, and also classify the target face accurately.

Region-based feature enhancement using channel-wise attention for classification of breast histopathological images

Breast histopathological image analysis at 400x magnification is essential for the determination of malignant breast tumours. But manual analysis of these images is tedious, subjective, error-prone and requires domain knowledge. To this end, computer-aided tools are gaining much attention in the recent past as it aids pathologists and save time. Furthermore, advances in computational power have leveraged the usage of computer tools. Yet, usage of computer-aided tools to analyse these images is challenging due to various reasons such as heterogeneity of malignant tumours, colour variations and presence of artefacts. Moreover, these images are captured at high resolutions which pose a major challenge to designing deep learning models as it demands high computational requirements. In this context, the present work proposes a new approach to efficiently and effectively extract features from these high-resolution images. In addition, at 400x magnification, the characteristics and structure of nuclei play a prominent role in the decision of malignancy. In this regard, the study introduces a novel CNN architecture called as CWA-Net that uses a colour channel attention module to enhance the features of the potential regions of interest such as nuclei. The developed model is qualitatively and quantitatively evaluated on private and public datasets and achieved an accuracy of 0.95% and 0.96%, respectively. The experimental evaluation demonstrates that the proposed method outperforms state-of-the-art methods on both datasets.

MTRRE-Net: A deep learning model for detection of breast cancer from histopathological images

Histopathological image classification has become one of the most challenging tasks among researchers due to the fine-grained variability of the disease. However, the rapid development of deep learning-based models such as the Convolutional Neural Network (CNN) has propelled much attentiveness to the classification of complex biomedical images. In this work, we propose a novel end-to-end deep learning model, named Multi-scale Dual Residual Recurrent Network (MTRRE-Net), for breast cancer classification from histopathological images. This model introduces a contrasting approach of dual residual block combined with the recurrent network to overcome the vanishing gradient problem even if the network is significantly deep. The proposed model has been evaluated on a publicly available standard dataset, namely BreaKHis, and achieved impressive accuracy in overcoming state-of-the-art models on all the images considered at various magnification levels.

Classification of breast cancer histology images using MSMV-PFENet

Deep learning has been used extensively in histopathological image classification, but people in this field are still exploring new neural network architectures for more effective and efficient cancer diagnosis. Here, we propose multi-scale, multi-view progressive feature encoding network (MSMV-PFENet) for effective classification. With respect to the density of cell nuclei, we selected the regions potentially related to carcinogenesis at multiple scales from each view. The progressive feature encoding network then extracted the global and local features from these regions. A bidirectional long short-term memory analyzed the encoding vectors to get a category score, and finally the majority voting method integrated different views to classify the histopathological images. We tested our method on the breast cancer histology dataset from the ICIAR 2018 grand challenge. The proposed MSMV-PFENet achieved 93.0[Formula: see text] and 94.8[Formula: see text] accuracies at the patch and image levels, respectively. This method can potentially benefit the clinical cancer diagnosis.

A multi-level feature-fusion-based approach to breast histopathological image classification

Previously, convolutional neural networks mostly used deep semantic feature information obtained from several convolutions for image classification. Such deep semantic features have a larger receptive field, and the features extracted are more effective as the number of convolutions increases, which helps in the classification of targets. However, this method tends to lose the shallow local features, such as the spatial connectivity and correlation of tumor region texture and edge contours in breast histopathology images, which leads to its recognition accuracy not being high enough. To address this problem, we propose a multi-level feature fusion method for breast histopathology image classification. First, we fuse shallow features and deep semantic features by attention mechanism and convolutions. Then, a new weighted cross entropy loss function is used to deal with the misjudgment of false negative and false positive. And finally, the correlation of spatial information is used to correct the misjudgment of some patches. We have conducted experiments on our own datasets and compared with the base network Inception-ResNet-v2, which has a high accuracy. The proposed method achieves an accuracy of 99.0% and an AUC of 99.9%.

Multi-Classification of Breast Cancer Lesions in Histopathological Images Using DEEP_Pachi: Multiple Self-Attention Head

INTRODUCTION AND BACKGROUND

Despite fast developments in the medical field, histological diagnosis is still regarded as the benchmark in cancer diagnosis. However, the input image feature extraction that is used to determine the severity of cancer at various magnifications is harrowing since manual procedures are biased, time consuming, labor intensive, and error-prone. Current state-of-the-art deep learning approaches for breast histopathology image classification take features from entire images (generic features). Thus, they are likely to overlook the essential image features for the unnecessary features, resulting in an incorrect diagnosis of breast histopathology imaging and leading to mortality.

METHODS

This discrepancy prompted us to develop DEEP_Pachi for classifying breast histopathology images at various magnifications. The suggested DEEP_Pachi collects global and regional features that are essential for effective breast histopathology image classification. The proposed model backbone is an ensemble of DenseNet201 and VGG16 architecture. The ensemble model extracts global features (generic image information), whereas DEEP_Pachi extracts spatial information (regions of interest). Statistically, the evaluation of the proposed model was performed on publicly available dataset: BreakHis and ICIAR 2018 Challenge datasets.

RESULTS

A detailed evaluation of the proposed model's accuracy, sensitivity, precision, specificity, and f1-score metrics revealed the usefulness of the backbone model and the DEEP_Pachi model for image classifying. The suggested technique outperformed state-of-the-art classifiers, achieving an accuracy of 1.0 for the benign class and 0.99 for the malignant class in all magnifications of BreakHis datasets and an accuracy of 1.0 on the ICIAR 2018 Challenge dataset.

CONCLUSIONS

The acquired findings were significantly resilient and proved helpful for the suggested system to assist experts at big medical institutions, resulting in early breast cancer diagnosis and a reduction in the death rate.

Explanation and Use of Uncertainty Quantified by Bayesian Neural Network Classifiers for Breast Histopathology Images

Despite the promise of Convolutional neural network (CNN) based classification models for histopathological images, it is infeasible to quantify its uncertainties. Moreover, CNNs may suffer from overfitting when the data is biased. We show that Bayesian-CNN can overcome these limitations by regularizing automatically and by quantifying the uncertainty. We have developed a novel technique to utilize the uncertainties provided by the Bayesian-CNN that significantly improves the performance on a large fraction of the test data (about 6% improvement in accuracy on 77% of test data). Further, we provide a novel explanation for the uncertainty by projecting the data into a low dimensional space through a nonlinear dimensionality reduction technique. This dimensionality reduction enables interpretation of the test data through visualization and reveals the structure of the data in a low dimensional feature space. We show that the Bayesian-CNN can perform much better than the state-of-the-art transfer learning CNN (TL-CNN) by reducing the false negative and false positive by 11% and 7.7% respectively for the present data set. It achieves this performance with only 1.86 million parameters as compared to 134.33 million for TL-CNN. Besides, we modify the Bayesian-CNN by introducing a stochastic adaptive activation function. The modified Bayesian-CNN performs slightly better than Bayesian-CNN on all performance metrics and significantly reduces the number of false negatives and false positives (3% reduction for both). We also show that these results are statistically significant by performing McNemar's statistical significance test. This work shows the advantages of Bayesian-CNN against the state-of-the-art, explains and utilizes the uncertainties for histopathological images. It should find applications in various medical image classifications.

Learning Binary Semantic Embedding forLarge-Scale Breast Histology Image Analysis

With the progress of clinical imaging innovation and machine learning, the computer-assisted diagnosis of breast histology images has attracted broad attention. Nonetheless, the use of computer-assisted diagnoses has been blocked due to the incomprehensibility of customary classification models. In view of this question, we propose a novel method for Learning Binary Semantic Embedding (LBSE). In this study, bit balance and uncorrela-tion constraints, double supervision, discrete optimization and asymmetric pairwise similarity are seamlessly integrated for learning binary semantic-preserving embedding. Moreover, a fusion-based strategy is carefully designed to handle the intractable problem of parameter setting, saving huge amounts of time for boundary tuning. Based on the above-mentioned proficient and effective embedding, classification and retrieval are simultaneously performed to give interpretable image-based deduction and model helped conclusions for breast histology images. Extensive experiments are conducted on three benchmark datasets to approve the predominance of LBSE in different situations.

DRDA-Net: Dense residual dual-shuffle attention network for breast cancer classification using histopathological images

Breast cancer is caused by the uncontrolled growth and division of cells in the breast, whereby a mass of tissue called a tumor is created. Early detection of breast cancer can save many lives. Hence, many researchers worldwide have invested considerable effort in developing robust computer-aided tools for the classification of breast cancer using histopathological images. For this purpose, in this study we designed a dual-shuffle attention-guided deep learning model, called the dense residual dual-shuffle attention network (DRDA-Net). Inspired by the bottleneck unit of the ShuffleNet architecture, in our proposed model we incorporate a channel attention mechanism, which enhances the model's ability to learn the complex patterns of images. Moreover, the model's densely connected blocks address both the overfitting and the vanishing gradient problem, although the model is trained on a substantially small dataset. We have evaluated our proposed model on the publicly available BreaKHis dataset and achieved classification accuracies of 95.72%, 94.41%, 97.43% and 98.1% on four different magnification levels i.e., 40x, 1000x, 200x, 400x respectively which proves the supremacy of the proposed model. The relevant code of the proposed DRDA-Net model can be foundt at: https://github.com/SohamChattopadhyayEE/DRDA-Net.

Breast histopathological image analysis using image processing techniques for diagnostic puposes: A methodological review

Breast cancer in women is the second most common cancer worldwide. Early detection of breast cancer can reduce the risk of human life. Non-invasive techniques such as mammograms and ultrasound imaging are popularly used to detect the tumour. However, histopathological analysis is necessary to determine the malignancy of the tumour as it analyses the image at the cellular level. Manual analysis of these slides is time consuming, tedious, subjective and are susceptible to human errors. Also, at times the interpretation of these images are inconsistent between laboratories. Hence, a Computer-Aided Diagnostic system that can act as a decision support system is need of the hour. Moreover, recent developments in computational power and memory capacity led to the application of computer tools and medical image processing techniques to process and analyze breast cancer histopathological images. This review paper summarizes various traditional and deep learning based methods developed to analyze breast cancer histopathological images. Initially, the characteristics of breast cancer histopathological images are discussed. A detailed discussion on the various potential regions of interest is presented which is crucial for the development of Computer-Aided Diagnostic systems. We summarize the recent trends and choices made during the selection of medical image processing techniques. Finally, a detailed discussion on the various challenges involved in the analysis of BCHI is presented along with the future scope.

BCHisto-Net: Breast histopathological image classification by global and local feature aggregation

Breast cancer among women is the second most common cancer worldwide. Non-invasive techniques such as mammograms and ultrasound imaging are used to detect the tumor. However, breast histopathological image analysis is inevitable for the detection of malignancy of the tumor. Manual analysis of breast histopathological images is subjective, tedious, laborious and is prone to human errors. Recent developments in computational power and memory have made automation a popular choice for the analysis of these images. One of the key challenges of breast histopathological image classification at 100× magnification is to extract the features of the potential regions of interest to decide on the malignancy of the tumor. The current state-of-the-art CNN based methods for breast histopathological image classification extract features from the entire image (global features) and thus may overlook the features of the potential regions of interest. This can lead to inaccurate diagnosis of breast histopathological images. This research gap has motivated us to propose BCHisto-Net to classify breast histopathological images at 100× magnification. The proposed BCHisto-Net extracts both global and local features required for the accurate classification of breast histopathological images. The global features extract abstract image features while local features focus on potential regions of interest. Furthermore, a feature aggregation branch is proposed to combine these features for the classification of 100× images. The proposed method is quantitatively evaluated on red a private dataset and publicly available BreakHis dataset. An extensive evaluation of the proposed model showed the effectiveness of the local and global features for the classification of these images. The proposed method achieved an accuracy of 95% and 89% on KMC and BreakHis datasets respectively, outperforming state-of-the-art classifiers.

Domain Knowledge Powered Deep Learning for Breast Cancer Diagnosis Based on Contrast-Enhanced Ultrasound Videos

In recent years, deep learning has been widely used in breast cancer diagnosis, and many high-performance models have emerged. However, most of the existing deep learning models are mainly based on static breast ultrasound (US) images. In actual diagnostic process, contrast-enhanced ultrasound (CEUS) is a commonly used technique by radiologists. Compared with static breast US images, CEUS videos can provide more detailed blood supply information of tumors, and therefore can help radiologists make a more accurate diagnosis. In this paper, we propose a novel diagnosis model based on CEUS videos. The backbone of the model is a 3D convolutional neural network. More specifically, we notice that radiologists generally follow two specific patterns when browsing CEUS videos. One pattern is that they focus on specific time slots, and the other is that they pay attention to the differences between the CEUS frames and the corresponding US images. To incorporate these two patterns into our deep learning model, we design a domain-knowledge-guided temporal attention module and a channel attention module. We validate our model on our Breast-CEUS dataset composed of 221 cases. The result shows that our model can achieve a sensitivity of 97.2% and an accuracy of 86.3%. In particular, the incorporation of domain knowledge leads to a 3.5% improvement in sensitivity and a 6.0% improvement in specificity. Finally, we also prove the validity of two domain knowledge modules in the 3D convolutional neural network (C3D) and the 3D ResNet (R3D).

Deep reinforcement learning in medical imaging: A literature review

Deep reinforcement learning (DRL) augments the reinforcement learning framework, which learns a sequence of actions that maximizes the expected reward, with the representative power of deep neural networks. Recent works have demonstrated the great potential of DRL in medicine and healthcare. This paper presents a literature review of DRL in medical imaging. We start with a comprehensive tutorial of DRL, including the latest model-free and model-based algorithms. We then cover existing DRL applications for medical imaging, which are roughly divided into three main categories: (i) parametric medical image analysis tasks including landmark detection, object/lesion detection, registration, and view plane localization; (ii) solving optimization tasks including hyperparameter tuning, selecting augmentation strategies, and neural architecture search; and (iii) miscellaneous applications including surgical gesture segmentation, personalized mobile health intervention, and computational model personalization. The paper concludes with discussions of future perspectives.