Attention, please! A survey of neural attention models in deep learning

An interpretable framework for gastric cancer classification using multi-channel attention mechanisms and transfer learning approach on histopathology images

The importance of gastric cancer (GC) and the role of deep learning techniques in categorizing GC histopathology images have recently increased. Identifying the drawbacks of traditional deep learning models, including lack of interpretability, inability to capture complex patterns, lack of adaptability, and sensitivity to noise. A multi-channel attention mechanism-based framework is proposed that can overcome the limitations of conventional deep learning models by dynamically focusing on relevant features, enhancing extraction, and capturing complex relationships in medical data. The proposed framework uses three different attention mechanism channels and convolutional neural networks to extract multichannel features during the classification process. The proposed framework's strong performance is confirmed by competitive experiments conducted on a publicly available Gastric Histopathology Sub-size Image Database, which yielded remarkable classification accuracies of 99.07% and 98.48% on the validation and testing sets, respectively. Additionally, on the HCRF dataset, the framework achieved high classification accuracy of 99.84% and 99.65% on the validation and testing sets, respectively. The effectiveness and interchangeability of the three channels are further confirmed by ablation and interchangeability experiments, highlighting the remarkable performance of the framework in GC histopathological image classification tasks. This offers an advanced and pragmatic artificial intelligence solution that addresses challenges posed by unique medical image characteristics for intricate image analysis. The proposed approach in artificial intelligence medical engineering demonstrates significant potential for enhancing diagnostic precision by achieving high classification accuracy and treatment outcomes.

A Position- and Similarity-Aware Named Entity Recognition Model for Power Equipment Maintenance Work Orders

Power equipment maintenance work orders are vital in power equipment management because they contain detailed information such as equipment specifications, defect reports, and specific maintenance activities. However, due to limited research into automated information extraction, valuable operational and maintenance data remain underutilized. A key challenge is recognizing unstructured Chinese maintenance texts filled with specialized and abbreviated terms unique to the power sector. Existing named entity recognition (NER) solutions often fail to effectively manage these complexities. To tackle this, this paper proposes a NER model tailored to power equipment maintenance work orders. First, a dataset called power equipment maintenance work orders (PE-MWO) is constructed, which covers seven entity categories. Next, a novel position- and similarity-aware attention module is proposed, where an innovative position embedding method and attention score calculation are designed to improve the model's contextual understanding while keeping computational costs low. Further, with this module as the main body, combined with the BERT-wwm-ext and conditional random field (CRF) modules, an efficient NER model is jointly constructed. Finally, validated on the PE-MWO and five public datasets, our model shows high accuracy in recognizing power sector entities, outperforming comparative models on public datasets.

Quantitative Analysis of Perovskite Morphologies Employing Deep Learning Framework Enables Accurate Solar Cell Performance Prediction

In perovskite solar cells, grain boundaries are considered one of the major structural defect sites, and consequently affect solar cell performance. Therefore, a precise edge detection of perovskite grains may enable to predict resulting solar cell performance. Herein, a deep learning model, Self-UNet, is developed to extract and quantify morphological information such as grain boundary length (GBL), the number of grains (NG), and average grain surface area (AGSA) from scanning elecron microscope (SEM) images. The Self-UNet excels conventional Canny and UNet models in edge extraction; the Dice coefficient and F1-score exhibit as high as 91.22% and 93.58%, respectively. The high edge detection accuracy of Self-UNet allows for not only identifying tiny grains stuck between relatively large grains, but also distinguishing actual grain boundaries from grooves on grain surface from low quality SEM images, avoiding under- or over-estimation of grain information. Moreover, the gradient boosted decision tree (GBDT) regression integrated to the Self-UNet exhibits high accuracy in predicting solar cell efficiency with relative errors of less than 10% compared to the experimentally measured efficiencies, which is corroborated by results from the literature and the experiments. Additionally, the GBL can be verified in multiple ways as a new morphological feature.

Enhanced quasi-meshing hotspot effect integrated embedded attention residual network for culture-free SERS accurate determination of Fusarium spores

Determination of Fusarium spores is essential for precision control of Fusarium head blight and ensuring agri-food safety. As a highly sensitive real-time detection technique with rich fingerprint information and little influence from water, surface-enhanced Raman spectroscopy (SERS) has been widely applied in the determination of microorganisms. However, fungi determination faces significant challenges including low sensitivity, and poor specificity. The enhanced quasi-meshing hotspot effect (EQMHE) integrated with the embedded attention residual network (EARNet) was proposed to realize a label-free and accurate SERS determination of various Fusarium spores. The EQMHE can promote the binding of spores and nanoparticles to form numerous hotspots, significantly enhancing signal quality and improving the detection limit by at least four orders of magnitude. The key factors inducing EQMHE were validated through various characterization techniques. Moreover, due to its excellent feature extraction and recognition capabilities, EARNet successfully overcomes the limitations of spectral similarity, achieving determination accuracies of 100% in the training set, 98.33% in the validation set, and 100% in the prediction set for three Fusarium species from actual samples. EARNet requires only a small amount of training data and provides rapid and accurate diagnostics. Throughout the process, the spores do not require culturing or lysing, providing an effective determination method for the practical determination of mixed fungal spores. Overall, the proposed strategy effectively addresses challenges such as the need for fungal spore cultivation, difficulties in SERS hotspot formation, and suboptimal signal quality, and it holds significant promise for applications in disease control, food safety, and agricultural production.

An improved algorithm for salient object detection of microscope based on U2-Net

With the rapid advancement of modern medical technology, microscopy imaging systems have become one of the key technologies in medical image analysis. However, manual use of microscopes presents issues such as operator dependency, inefficiency, and time consumption. To enhance the efficiency and accuracy of medical image capture and reduce the burden of subsequent quantitative analysis, this paper proposes an improved microscope salient object detection algorithm based on U2-Net, incorporating deep learning technology. The improved algorithm first enhances the network's key information extraction capability by incorporating the Convolutional Block Attention Module (CBAM) into U2-Net. It then optimizes network complexity by constructing a Simple Pyramid Pooling Module (SPPM) and uses Ghost convolution to achieve model lightweighting. Additionally, data augmentation is applied to the slides to improve the algorithm's robustness and generalization. The experimental results show that the size of the improved algorithm model is 72.5 MB, which represents a 56.85% reduction compared to the original U2-Net model size of 168.0 MB. Additionally, the model's prediction accuracy has increased from 92.24 to 97.13%, providing an efficient means for subsequent image processing and analysis tasks in microscopy imaging systems.

Common issues of data science on the eco-environmental risks of emerging contaminants

Data-driven approaches (e.g., machine learning) are increasingly used to replace or assist laboratory studies in the study of emerging contaminants (ECs). In the past ten years, an increasing number of models or approaches have been applied to ECs, and the datasets used are continuously enriched. However, there are large knowledge gaps between what we have found and the natural eco-environmental meaning. For most published reviews, the contents are organized by the types of ECs, but the common issues of data science, regardless of the type of pollutant, are not sufficiently addressed. To close or narrow the knowledge gaps, we highlight the following issues ignored in the field of data-driven EC research. Complicated biological and ecological data and ensemble models revealing mechanisms and spatiotemporal trends with strong causal relationships and without data leakage deserve more attention in the future. In addition, the matrix influence, trace concentration, and complex scenario have often been ignored in previous works. Therefore, an integrated research framework related to natural fields, ecological systems, and large-scale environmental problems, rather than relying solely on laboratory data-related analysis, is urgently needed. Beyond the current prediction purposes, data science can inspire the discovery of scientific questions, and mutual inspiration among data science, process and mechanism models, and laboratory and field research is a critical direction. Focusing on the above urgent and common issues related to data, frameworks, and purposes, regardless of the type of pollutant, data science is expected to achieve great advancements in addressing the eco-environmental risks of ECs.

Artificial intelligence and machine learning in cell-free-DNA-based diagnostics

The discovery of circulating fetal and tumor cell-free DNA (cfDNA) molecules in plasma has opened up tremendous opportunities in noninvasive diagnostics such as the detection of fetal chromosomal aneuploidies and cancers and in posttransplantation monitoring. The advent of high-throughput sequencing technologies makes it possible to scrutinize the characteristics of cfDNA molecules, opening up the fields of cfDNA genetics, epigenetics, transcriptomics, and fragmentomics, providing a plethora of biomarkers. Machine learning (ML) and/or artificial intelligence (AI) technologies that are known for their ability to integrate high-dimensional features have recently been applied to the field of liquid biopsy. In this review, we highlight various AI and ML approaches in cfDNA-based diagnostics. We first introduce the biology of cell-free DNA and basic concepts of ML and AI technologies. We then discuss selected examples of ML- or AI-based applications in noninvasive prenatal testing and cancer liquid biopsy. These applications include the deduction of fetal DNA fraction, plasma DNA tissue mapping, and cancer detection and localization. Finally, we offer perspectives on the future direction of using ML and AI technologies to leverage cfDNA fragmentation patterns in terms of methylomic and transcriptional investigations.

Online comments of tourist attractions combining artificial intelligence text mining model and attention mechanism

This paper intends to solve the limitations of the existing methods to deal with the comments of tourist attractions. With the technical support of Artificial Intelligence (AI), an online comment method of tourist attractions based on text mining model and attention mechanism is proposed. In the process of text mining, the attention mechanism is used to calculate the contribution of each topic to text representation on the topic layer of Latent Dirichlet Allocation (LDA). The Bidirectional Recurrent Neural Network (BiGRU) can effectively capture the temporal relationship and semantic dependence in the text through its powerful sequence modeling ability, thus achieving a more accurate classification of emotional tendencies. In order to verify the performance of the proposed ATT-LDA- Bigelow model, online comments about tourist attractions are collected from Ctrip.com, and users' emotional tendencies towards different scenic spots are analyzed. The results show that this model has the best emotion classification effect in online comments of scenic spots, with the accuracy and F1 value reaching 93.85% and 93.68% respectively, which is superior to other emotion classification models. The proposed method not only improves the accuracy of sentiment analysis, but also provides strong support for the optimization of tourism recommendation system and provides more comprehensive, objective and accurate tourism information for scenic spot managers and tourism enterprises. This achievement is expected to bring new enlightenment and breakthrough to the research and practice in related fields.

ExGAT: Context extended graph attention neural network

As an essential concept in attention, context defines the overall scope under consideration. In attention-based GNNs, context becomes the set of representation nodes of graph embedding. Current approaches choose immediate neighbors of the target or its subset as the context, which limits the ability of attention to capture long-distance dependency. To address this deficiency, we propose a novel attention-based GNN framework with extended contexts. Concretely, multi-hop nodes are first selected for context expansion according to information transferability and the number of hops. Then, to reduce the computational cost and fit the graph representation learning process, two heuristic context refinement policies are designed by focusing on local graph structure. One is for the graphs with high degrees, multi-hop neighbors with fewer connections to the target are removed to acquire accurate diffused information. The other is for the graphs with low degrees or uniform degree distribution, low-transferability neighbors are dislodged to ensure the graph locality is not obscured by the global information induced by the extended context. Finally, multi-head attention is employed in the refined context. Numerical comparisons with 23 baselines demonstrate the superiority of our method. Extensive model analysis shows that extending context with the informative multi-hop neighbors properly indeed promotes the performance of attention-based GNNs.

Integrating a multi-variable scenario with Attention-LSTM model to forecast long-term coastal beach erosion

Beach erosion is an adverse impact of climate change and human development activities. Effective beach management necessitates integrating natural and anthropogenic factors to address future erosion trends, while most current prediction models focus only on natural factors, which may provide an incomplete and potentially inaccurate representation of erosion dynamics. This study enhances prediction methods by integrating both natural and anthropogenic factors, thereby enhancing the accuracy and reliability of erosion projections. By extracting historical shorelines through CoastSat model from 1986 to 2020, we develop multivariable scenarios with Attention-LSTM model to predict the regional impacts of natural and anthropogenic factors on erosion to sandy beaches along the typical shoreline of Shenzhen in China. Results reveal that Shenzhen's beaches experienced erosion up to 12 m over the past 35 years. Here we project a decrease in the mean erosion rate of the beaches, identifying population growth (21.0 %) as the main controlling factor before the mid-century in a range of scenarios. We find that Attention-LSTM multi-model ensemble approach can provide overall improved accuracy and reliability over a wide range of beach erosion compared to scenario prediction model of Attention-LSTM and statistical model of Digital Shoreline Analysis System (DSAS), yielding an average uncertainty of 10.99 compared to 13.29. These insights reveal policies to safeguard beaches because of the rising demand for beaches due to human factors, coupled with decreased impervious surfaces through ecological conservation, lead to mitigation for beach erosion. Accurate forecasts empower policymakers to implement effective coastal management strategies, safeguard resources, and mitigate erosion's adverse effects. Our study offers finely-tuned predictions of coastal erosion, providing crucial insights for future coastal conservation efforts and climate change adaptation along the shoreline, and serving as a foundation for further research aimed at understanding the evolving environmental impacts of beach erosion in Shenzhen.

Selective Auditory Attention Detection Using Combined Transformer and Convolutional Graph Neural Networks

Attention is one of many human cognitive functions that are essential in everyday life. Given our limited processing capacity, attention helps us focus only on what matters. Focusing attention on one speaker in an environment with many speakers is a critical ability of the human auditory system. This paper proposes a new end-to-end method based on the combined transformer and graph convolutional neural network (TraGCNN) that can effectively detect auditory attention from electroencephalograms (EEGs). This approach eliminates the need for manual feature extraction, which is often time-consuming and subjective. Here, the first EEG signals are converted to graphs. We then extract attention information from these graphs using spatial and temporal approaches. Finally, our models are trained with these data. Our model can detect auditory attention in both the spatial and temporal domains. Here, the EEG input is first processed by transformer layers to obtain a sequential representation of EEG based on attention onsets. Then, a family of graph convolutional layers is used to find the most active electrodes using the spatial position of electrodes. Finally, the corresponding EEG features of active electrodes are fed into the graph attention layers to detect auditory attention. The Fuglsang 2020 dataset is used in the experiments to train and test the proposed and baseline systems. The new TraGCNN approach, as compared with state-of-the-art attention classification methods from the literature, yields the highest performance in terms of accuracy (80.12%) as a classification metric. Additionally, the proposed model results in higher performance than our previously graph-based model for different lengths of EEG segments. The new TraGCNN approach is advantageous because attenuation detection is achieved from EEG signals of subjects without requiring speech stimuli, as is the case with conventional auditory attention detection methods. Furthermore, examining the proposed model for different lengths of EEG segments shows that the model is faster than our previous graph-based detection method in terms of computational complexity. The findings of this study have important implications for the understanding and assessment of auditory attention, which is crucial for many applications, such as brain-computer interface (BCI) systems, speech separation, and neuro-steered hearing aid development.

Automated classification of pollen grains microscopic images using cognitive attention based on human Two Visual Streams Hypothesis

Aerobiology is a branch of biology that studies microorganisms passively transferred by the air. Bacteria, viruses, fungal spores, tiny insects, and pollen grains are samples of microorganisms. Pollen grains classification is essential in medicine, agronomy, economy, etc. It is performed traditionally (manually) and automatically. The automated approach is faster, more accurate, cost-effective, and with less human intervention than the manual method. In this paper, we introduce a Residual Cognitive Attention Network (RCANet) for the automated classification of pollen grains microscopic images. The suggested attention block, Ventral-Dorsal Ateetntion Block (VDAB), is designed based on the ventral (temporal) and dorsal (parietal) pathways of the occipital lobe. It is embedded in each Basic Block of the architecture of ResNet18. The VDAB is composed of ventral and dorsal attention blocks. The ventral and dorsal streams detect the structure and location of the pollen grain, respectively. According to the mentioned pathways, the Ventral Attention Block (VAB) extracts the channels related to the shape of the pollen grain, and the Dorsal Attention Block (DAB) is focused on its position. Three publicly pollen grains datasets including the Cretan Pollen Dataset (CPD), Pollen13K, and Pollen23E are employed for experiments. The ResNet18 and the proposed method (RCANet) are trained on the datasets and the proposed RCANet obtained higher performance metrics than the ResNet18 in the test step. It achieved weighted F1-score values of 98.69%, 97.83%, and 98.24% with CPD, Pollen13K, and Pollen23E datasets, respectively.

Drilling Rate of Penetration Prediction Based on CBT-LSTM Neural Network

Due to the uncertainty of the subsurface environment and the complexity of parameters, particularly in feature extraction from input data and when seeking to understand bidirectional temporal information, the evaluation and prediction of the rate of penetration (ROP) in real-time drilling operations has remained a long-standing challenge. To address these issues, this study proposes an improved LSTM neural network model for ROP prediction (CBT-LSTM). This model integrates the capability of a two-dimensional convolutional neural network (2D-CNN) for multi-feature extraction, the advantages of bidirectional long short-term memory networks (BiLSTM) for processing bidirectional temporal information, and the dynamic weight adjustment of the time pattern attention mechanism (TPA) for extracting crucial information in BiLSTM, effectively capturing key features in temporal data. Initially, data are denoised using the Savitzky-Golay filter, and five correlation coefficient methods are employed to select input features, with principal component analysis (PCA) used to reduce model complexity. Subsequently, a sliding window approach transforms the time series into a two-dimensional structure to capture dynamic changes, constructing the model input. Finally, the ROP prediction model is established, and search methods are utilized to identify the optimal hyperparameter combinations. Compared with other neural networks, CBT-LSTM demonstrates superior performance metrics, with MAE, MAPE, RMSE, and R2 values of 0.0295, 0.0357, 9.3101%, and 0.9769, respectively, indicating the highest predictive capability. To validate the model's robustness, noise was introduced into the training data, and results show stable performance. Furthermore, the model's predictive results for other wells achieved R2 values of 0.95, confirming its strong generalization ability. This method provides a new solution for ROP prediction in real-time drilling operations, assisting drilling engineers in better planning their operations and reducing drilling cycles.

GPT-Driven Radiology Report Generation with Fine-Tuned Llama 3

The integration of deep learning into radiology has the potential to enhance diagnostic processes, yet its acceptance in clinical practice remains limited due to various challenges. This study aimed to develop and evaluate a fine-tuned large language model (LLM), based on Llama 3-8B, to automate the generation of accurate and concise conclusions in magnetic resonance imaging (MRI) and computed tomography (CT) radiology reports, thereby assisting radiologists and improving reporting efficiency. A dataset comprising 15,000 radiology reports was collected from the University of Medicine and Pharmacy of Craiova's Imaging Center, covering a diverse range of MRI and CT examinations made by four experienced radiologists. The Llama 3-8B model was fine-tuned using transfer-learning techniques, incorporating parameter quantization to 4-bit precision and low-rank adaptation (LoRA) with a rank of 16 to optimize computational efficiency on consumer-grade GPUs. The model was trained over five epochs using an NVIDIA RTX 3090 GPU, with intermediary checkpoints saved for monitoring. Performance was evaluated quantitatively using Bidirectional Encoder Representations from Transformers Score (BERTScore), Recall-Oriented Understudy for Gisting Evaluation (ROUGE), Bilingual Evaluation Understudy (BLEU), and Metric for Evaluation of Translation with Explicit Ordering (METEOR) metrics on a held-out test set. Additionally, a qualitative assessment was conducted, involving 13 independent radiologists who participated in a Turing-like test and provided ratings for the AI-generated conclusions. The fine-tuned model demonstrated strong quantitative performance, achieving a BERTScore F1 of 0.8054, a ROUGE-1 F1 of 0.4998, a ROUGE-L F1 of 0.4628, and a METEOR score of 0.4282. In the human evaluation, the artificial intelligence (AI)-generated conclusions were preferred over human-written ones in approximately 21.8% of cases, indicating that the model's outputs were competitive with those of experienced radiologists. The average rating of the AI-generated conclusions was 3.65 out of 5, reflecting a generally favorable assessment. Notably, the model maintained its consistency across various types of reports and demonstrated the ability to generalize to unseen data. The fine-tuned Llama 3-8B model effectively generates accurate and coherent conclusions for MRI and CT radiology reports. By automating the conclusion-writing process, this approach can assist radiologists in reducing their workload and enhancing report consistency, potentially addressing some barriers to the adoption of deep learning in clinical practice. The positive evaluations from independent radiologists underscore the model's potential utility. While the model demonstrated strong performance, limitations such as dataset bias, limited sample diversity, a lack of clinical judgment, and the need for large computational resources require further refinement and real-world validation. Future work should explore the integration of such models into clinical workflows, address ethical and legal considerations, and extend this approach to generate complete radiology reports.

Application of GWO-attention-ConvLSTM model in customer churn prediction and satisfaction analysis in customer relationship management

Customer Relationship Management (CRM) is vital in modern business, aiding in the management and analysis of customer interactions. However, existing methods struggle to capture the dynamic and complex nature of customer relationships, as traditional approaches fail to leverage time series data effectively. To address this, we propose a novel GWO-attention-ConvLSTM model, which offers more effective prediction of customer churn and analysis of customer satisfaction. This model utilizes an attention mechanism to focus on key information and integrates a ConvLSTM layer to capture spatiotemporal features, effectively modeling complex temporal patterns in customer data. We validate our proposed model on multiple real-world datasets, including the BigML Telco Churn dataset, IBM Telco dataset, Cell2Cell dataset, and Orange Telecom dataset. Experimental results demonstrate significant performance improvements of our model compared to existing baseline models across these datasets. For instance, on the BigML Telco Churn dataset, our model achieves an accuracy of 95.17%, a recall of 93.66%, an F1 score of 92.89%, and an AUC of 95.00%. Similar results are validated on other datasets. In conclusion, our proposed GWO-attention-ConvLSTM model makes significant advancements in the CRM domain, providing powerful tools for predicting customer churn and analyzing customer satisfaction. By addressing the limitations of existing methods and leveraging the capabilities of deep learning, attention mechanisms, and optimization algorithms, our model paves the way for improving customer relationship management practices and driving business success.

Enhancing Water-Deficient Potato Plant Identification: Assessing Realistic Performance of Attention-Based Deep Neural Networks and Hyperspectral Imaging for Agricultural Applications

Hyperspectral imaging has emerged as a pivotal technology in agricultural research, offering a powerful means to non-invasively monitor stress factors, such as drought, in crops like potato plants. In this context, the integration of attention-based deep learning models presents a promising avenue for enhancing the efficiency of stress detection, by enabling the identification of meaningful spectral channels. This study assesses the performance of deep learning models on two potato plant cultivars exposed to water-deficient conditions. It explores how various sampling strategies and biases impact the classification metrics by using a dual-sensor hyperspectral imaging systems (VNIR -Visible and Near-Infrared and SWIR-Short-Wave Infrared). Moreover, it focuses on pinpointing crucial wavelengths within the concatenated images indicative of water-deficient conditions. The proposed deep learning model yields encouraging results. In the context of binary classification, it achieved an area under the receiver operating characteristic curve (AUC-ROC-Area Under the Receiver Operating Characteristic Curve) of 0.74 (95% CI: 0.70, 0.78) and 0.64 (95% CI: 0.56, 0.69) for the KIS Krka and KIS Savinja varieties, respectively. Moreover, the corresponding F1 scores were 0.67 (95% CI: 0.64, 0.71) and 0.63 (95% CI: 0.56, 0.68). An evaluation of the performance of the datasets with deliberately introduced biases consistently demonstrated superior results in comparison to their non-biased equivalents. Notably, the ROC-AUC values exhibited significant improvements, registering a maximum increase of 10.8% for KIS Krka and 18.9% for KIS Savinja. The wavelengths of greatest significance were observed in the ranges of 475-580 nm, 660-730 nm, 940-970 nm, 1420-1510 nm, 1875-2040 nm, and 2350-2480 nm. These findings suggest that discerning between the two treatments is attainable, despite the absence of prominently manifested symptoms of drought stress in either cultivar through visual observation. The research outcomes carry significant implications for both precision agriculture and potato breeding. In precision agriculture, precise water monitoring enhances resource allocation, irrigation, yield, and loss prevention. Hyperspectral imaging holds potential to expedite drought-tolerant cultivar selection, thereby streamlining breeding for resilient potatoes adaptable to shifting climates.

Improved neural network with multi-task learning for Alzheimer's disease classification

Alzheimer's disease(AD) poses a significant challenge due to its widespread prevalence and the lack of effective treatments, highlighting the urgent need for early detection. This research introduces an enhanced neural network, named ADnet, which is based on the VGG16 model, to detect Alzheimer's disease using two-dimensional MRI slices. ADNet incorporates several key improvements: it replaces traditional convolution with depthwise separable convolution to reduce model parameters, replaces the ReLU activation function with ELU to address potential issues with exploding gradients, and integrates the SE(Squeeze-and-Excitation) module to enhance feature extraction efficiency. In addition to the primary task of MRI feature extraction, ADnet is simultaneously trained on two auxiliary tasks: clinical dementia score regression and mental state score regression. Experimental results demonstrate that compared to the baseline VGG16, ADNet achieves a 4.18% accuracy improvement for AD vs. CN classification and a 6% improvement for MCI vs. CN classification. These findings highlight the effectiveness of ADnet in classifying Alzheimer's disease, providing crucial support for early diagnosis and intervention by medical professionals. The proposed enhancements represent advancements in neural network architecture and training strategies for improved AD classification.

Human Pose Estimation Based on Efficient and Lightweight High-Resolution Network (EL-HRNet)

As an important direction in computer vision, human pose estimation has received extensive attention in recent years. A High-Resolution Network (HRNet) can achieve effective estimation results as a classical human pose estimation method. However, the complex structure of the model is not conducive to deployment under limited computer resources. Therefore, an improved Efficient and Lightweight HRNet (EL-HRNet) model is proposed. In detail, point-wise and grouped convolutions were used to construct a lightweight residual module, replacing the original 3 × 3 module to reduce the parameters. To compensate for the information loss caused by the network's lightweight nature, the Convolutional Block Attention Module (CBAM) is introduced after the new lightweight residual module to construct the Lightweight Attention Basicblock (LA-Basicblock) module to achieve high-precision human pose estimation. To verify the effectiveness of the proposed EL-HRNet, experiments were carried out using the COCO2017 and MPII datasets. The experimental results show that the EL-HRNet model requires only 5 million parameters and 2.0 GFlops calculations and achieves an AP score of 67.1% on the COCO2017 validation set. In addition, PCKh@0.5mean is 87.7% on the MPII validation set, and EL-HRNet shows a good balance between model complexity and human pose estimation accuracy.

Dense attention network identifies EEG abnormalities during working memory performance of patients with schizophrenia

Introduction

Patients with schizophrenia typically exhibit deficits in working memory (WM) associated with abnormalities in brain activity. Alterations in the encoding, maintenance and retrieval phases of sequential WM tasks are well established. However, due to the heterogeneity of symptoms and complexity of its neurophysiological underpinnings, differential diagnosis remains a challenge. We conducted an electroencephalographic (EEG) study during a visual WM task in fifteen schizophrenia patients and fifteen healthy controls. We hypothesized that EEG abnormalities during the task could be identified, and patients successfully classified by an interpretable machine learning algorithm.

Methods

We tested a custom dense attention network (DAN) machine learning model to discriminate patients from control subjects and compared its performance with simpler and more commonly used machine learning models. Additionally, we analyzed behavioral performance, event-related EEG potentials, and time-frequency representations of the evoked responses to further characterize abnormalities in patients during WM.

Results

The DAN model was significantly accurate in discriminating patients from healthy controls, ACC = 0.69, SD = 0.05. There were no significant differences between groups, conditions, or their interaction in behavioral performance or event-related potentials. However, patients showed significantly lower alpha suppression in the task preparation, memory encoding, maintenance, and retrieval phases F(1,28) = 5.93, p = 0.022, η2 = 0.149. Further analysis revealed that the two highest peaks in the attention value vector of the DAN model overlapped in time with the preparation and memory retrieval phases, as well as with two of the four significant time-frequency ROIs.

Discussion

These results highlight the potential utility of interpretable machine learning algorithms as an aid in diagnosis of schizophrenia and other psychiatric disorders presenting oscillatory abnormalities.

Attention-Enabled Ensemble Deep Learning Models and Their Validation for Depression Detection: A Domain Adoption Paradigm

Depression is increasingly prevalent, leading to higher suicide risk. Depression detection and sentimental analysis of text inputs in cross-domain frameworks are challenging. Solo deep learning (SDL) and ensemble deep learning (EDL) models are not robust enough. Recently, attention mechanisms have been introduced in SDL. We hypothesize that attention-enabled EDL (aeEDL) architectures are superior compared to attention-not-enabled SDL (aneSDL) or aeSDL models. We designed EDL-based architectures with attention blocks to build eleven kinds of SDL model and five kinds of EDL model on four domain-specific datasets. We scientifically validated our models by comparing "seen" and "unseen" paradigms (SUP). We benchmarked our results against the SemEval (2016) sentimental dataset and established reliability tests. The mean increase in accuracy for EDL over their corresponding SDL components was 4.49%. Regarding the effect of attention block, the increase in the mean accuracy (AUC) of aeSDL over aneSDL was 2.58% (1.73%), and the increase in the mean accuracy (AUC) of aeEDL over aneEDL was 2.76% (2.80%). When comparing EDL vs. SDL for non-attention and attention, the mean aneEDL was greater than aneSDL by 4.82% (3.71%), and the mean aeEDL was greater than aeSDL by 5.06% (4.81%). For the benchmarking dataset (SemEval), the best-performing aeEDL model (ALBERT+BERT-BiLSTM) was superior to the best aeSDL (BERT-BiLSTM) model by 3.86%. Our scientific validation and robust design showed a difference of only 2.7% in SUP, thereby meeting the regulatory constraints. We validated all our hypotheses and further demonstrated that aeEDL is a very effective and generalized method for detecting symptoms of depression in cross-domain settings.

A Review on Deep-Learning-Based Cyberbullying Detection

Bullying is described as an undesirable behavior by others that harms an individual physically, mentally, or socially. Cyberbullying is a virtual form (e.g., textual or image) of bullying or harassment, also known as online bullying. Cyberbullying detection is a pressing need in today’s world, as the prevalence of cyberbullying is continually growing, resulting in mental health issues. Conventional machine learning models were previously used to identify cyberbullying. However, current research demonstrates that deep learning surpasses traditional machine learning algorithms in identifying cyberbullying for several reasons, including handling extensive data, efficiently classifying text and images, extracting features automatically through hidden layers, and many others. This paper reviews the existing surveys and identifies the gaps in those studies. We also present a deep-learning-based defense ecosystem for cyberbullying detection, including data representation techniques and different deep-learning-based models and frameworks. We have critically analyzed the existing DL-based cyberbullying detection techniques and identified their significant contributions and the future research directions they have presented. We have also summarized the datasets being used, including the DL architecture being used and the tasks that are accomplished for each dataset. Finally, several challenges faced by the existing researchers and the open issues to be addressed in the future have been presented.

Breast Cancer Diagnosis in Thermography Using Pre-Trained VGG16 with Deep Attention Mechanisms

One of the most prevalent cancers in women is breast cancer. The mortality rate related to this disease can be decreased by early, accurate diagnosis to increase the chance of survival. Infrared thermal imaging is one of the breast imaging modalities in which the temperature of the breast tissue is measured using a screening tool. The previous studies did not use pre-trained deep learning (DL) with deep attention mechanisms (AMs) on thermographic images for breast cancer diagnosis. Using thermal images from the Database for Research Mastology with Infrared Image (DMR-IR), the study investigates the use of a pre-trained Visual Geometry Group with 16 layers (VGG16) with AMs that can produce good diagnosis performance utilizing the thermal images of breast cancer. The symmetry of the three models resulting from the combination of VGG16 with three types of AMs is evident in all its stages in methodology. The models were compared to state-of-art breast cancer diagnosis approaches and tested for accuracy, sensitivity, specificity, precision, F1-score, AUC score, and Cohen’s kappa. The test accuracy rates for the AMs using the VGG16 model on the breast thermal dataset were encouraging, at 99.80%, 99.49%, and 99.32%. Test accuracy for VGG16 without AMs was 99.18%, whereas test accuracy for VGG16 with AMs improved by 0.62%. The proposed approaches also performed better than previous approaches examined in the related studies.

Prediction of dissolved oxygen concentration in aquaculture based on attention mechanism and combined neural network

As an essential water quality parameter in aquaculture ponds, dissolved oxygen (DO) affects the growth and development of aquatic animals and their feeding and absorption. However, DO is easily influenced by external factors. It is not easy to make scientific and accurate predictions of DO concentration trends, especially in long-term predictions. This paper uses a one-dimensional convolutional neural network to extract the features of multidimensional input data. Bidirectional long and short-term memory neural network propagated forward and backward twice and thoroughly mined the before and after attribute relationship of each data of dissolved oxygen sequence. The attention mechanism focuses the model on the time series prediction step to improve long-term prediction accuracy. Finally, we built an integrated prediction model based on convolutional neural network (CNN), bidirectional long and short-term memory neural network (BiLSTM) and attention mechanism (AM), which is called CNN-BiLSTM-AM model. To determine the accuracy of the CNN-BiLSTM-AM model, we conducted short-term (30 minutes, one hour) and long-term (6 hours, 12 hours) experimental validation on real datasets monitored at two aquaculture farms in Yantai City, Shandong Province, China. Meanwhile, the performance was compared and visualized with support vector regression, recurrent neural network, long short-term memory neural network, CNN-LSTM model and CNN-BiLSTM model. The results show that compared with other comparative models, the proposed CNN-BiLSTM-AM model has an excellent performance in mean absolute error, root means square error, mean absolute percentage error and determination coefficient.

A cross-domain fruit classification method based on lightweight attention networks and unsupervised domain adaptation

Image-based fruit classification offers many useful applications in industrial production and daily life, such as self-checkout in the supermarket, automatic fruit sorting and dietary guidance. However, fruit classification task will have different data distributions due to different application scenarios. One feasible solution to solve this problem is to use domain adaptation that adapts knowledge from the original training data (source domain) to the new testing data (target domain). In this paper, we propose a novel deep learning-based unsupervised domain adaptation method for cross-domain fruit classification. A hybrid attention module is proposed and added to MobileNet V3 to construct the HAM-MobileNet that can suppress the impact of complex backgrounds and extract more discriminative features. A hybrid loss function combining subdomain alignment and implicit distribution metrics is used to reduce domain discrepancy during model training and improve model classification performance. Two fruit classification datasets covering several domains are established to simulate common industrial and daily life application scenarios. We validate the proposed method on our constructed grape classification dataset and general fruit classification dataset. The experimental results show that the proposed method achieves an average accuracy of 95.0% and 93.2% on the two datasets, respectively. The classification model after domain adaptation can well overcome the domain discrepancy brought by different fruit classification scenarios. Meanwhile, the proposed datasets and method can serve as a benchmark for future cross-domain fruit classification research.

Self-Relation Attention and Temporal Awareness for Emotion Recognition via Vocal Burst

Speech emotion recognition (SER) is one of the most exciting topics many researchers have recently been involved in. Although much research has been conducted recently on this topic, emotion recognition via non-verbal speech (known as the vocal burst) is still sparse. The vocal burst is concise and has meaningless content, which is harder to deal with than verbal speech. Therefore, in this paper, we proposed a self-relation attention and temporal awareness (SRA-TA) module to tackle this problem with vocal bursts, which could capture the dependency in a long-term period and focus on the salient parts of the audio signal as well. Our proposed method contains three main stages. Firstly, the latent features are extracted using a self-supervised learning model from the raw audio signal and its Mel-spectrogram. After the SRA-TA module is utilized to capture the valuable information from latent features, all features are concatenated and fed into ten individual fully-connected layers to predict the scores of 10 emotions. Our proposed method achieves a mean concordance correlation coefficient (CCC) of 0.7295 on the test set, which achieves the first ranking of the high-dimensional emotion task in the 2022 ACII Affective Vocal Burst Workshop & Challenge.

TSHVNet: Simultaneous Nuclear Instance Segmentation and Classification in Histopathological Images Based on Multiattention Mechanisms

Accurate nuclear instance segmentation and classification in histopathologic images are the foundation of cancer diagnosis and prognosis. Several challenges are restricting the development of accurate simultaneous nuclear instance segmentation and classification. Firstly, the visual appearances of different category nuclei could be similar, making it difficult to distinguish different types of nuclei. Secondly, it is thorny to separate highly clustering nuclear instances. Thirdly, rare current studies have considered the global dependencies among diverse nuclear instances. In this article, we propose a novel deep learning framework named TSHVNet which integrates multiattention modules (i.e., Transformer and SimAM) into the state-of-the-art HoVer-Net for the sake of a more accurate nuclear instance segmentation and classification. Specifically, the Transformer attention module is employed on the trunk of the HoVer-Net to model the long-distance relationships of diverse nuclear instances. The SimAM attention modules are deployed on both the trunk and branches to apply the 3D channel and spatial attention to assign neurons with appropriate weights. Finally, we validate the proposed method on two public datasets: PanNuke and CoNSeP. The comparison results have shown the outstanding performance of the proposed TSHVNet network among the state-of-art methods. Particularly, as compared to the original HoVer-Net, the performance of nuclear instance segmentation evaluated by the PQ index has shown 1.4% and 2.8% increases on the CoNSeP and PanNuke datasets, respectively, and the performance of nuclear classification measured by F1_score has increased by 2.4% and 2.5% on the CoNSeP and PanNuke datasets, respectively. Therefore, the proposed multiattention-based TSHVNet is of great potential in simultaneous nuclear instance segmentation and classification.