Multimodal Sparse Transformer Network for Audio-Visual Speech Recognition

Multimodal Dual-Embedding Networks for Malware Open-Set Recognition

Malware open-set recognition (MOSR) is an emerging research domain that aims at jointly classifying malware samples from known families and detecting the ones from novel unknown families, respectively. Existing works mostly rely on a well-trained classifier considering the predicted probabilities of each known family with a threshold-based detection to achieve the MOSR. However, our observation reveals that the feature distributions of malware samples are extremely similar to each other even between known and unknown families. Thus, the obtained classifier may produce overly high probabilities of testing unknown samples toward known families and degrade the model performance. In this article, we propose the multi\modal dual-embedding networks, dubbed MDENet, to take advantage of comprehensive malware features from different modalities to enhance the diversity of malware feature space, which is more representative and discriminative for down-stream recognition. Concretely, we first generate a malware image for each observed sample based on their numeric features using our proposed numeric encoder with a re- designed multiscale CNN structure, which can better explore their statistical and spatial correlations. Besides, we propose to organize tokenized malware features into a sentence for each sample considering its behaviors and dynamics, and utilize language models as the textual encoder to transform it into a representable and computable textual vector. Such parallel multimodal encoders can fuse the above two components to enhance the feature diversity. Last, to further guarantee the open-set recognition (OSR), we dually embed the fused multimodal representation into one primary space and an associated sub-space, i.e., discriminative and exclusive spaces, with contrastive sampling and -bounded enclosing sphere regularizations, which resort to classification and detection, respectively. Moreover, we also enrich our previously proposed large-scaled malware dataset MAL-100 with multimodal characteristics and contribute an improved version dubbed MAL-100+. Experimental results on the widely used malware dataset Mailing and the proposed MAL-100+ demonstrate the effectiveness of our method.

MH6D: Multi-Hypothesis Consistency Learning for Category-Level 6-D Object Pose Estimation

Six-degree-of-freedom (6DoF) object pose estimation is a crucial task for virtual reality and accurate robotic manipulation. Category-level 6DoF pose estimation has recently become popular as it improves generalization to a complete category of objects. However, current methods focus on data-driven differential learning, which makes them highly dependent on the quality of the real-world labeled data and limits their ability to generalize to unseen objects. To address this problem, we propose multi-hypothesis (MH) consistency learning (MH6D) for category-level 6-D object pose estimation without using real-world training data. MH6D uses a parallel consistency learning structure, alleviating the uncertainty problem of single-shot feature extraction and promoting self-adaptation of domain to reduce the synthetic-to-real domain gap. Specifically, three randomly sampled pose transformations are first performed in parallel on the input point cloud. An attention-guided category-level 6-D pose estimation network with channel attention (CA) and global feature cross-attention (GFCA) modules is then proposed to estimate the three hypothesized 6-D object poses by extracting and fusing the global and local features effectively. Finally, we propose a novel loss function that considers both the process and the final result information allowing MH6D to perform robust consistency learning. We conduct experiments under two different training data settings (i.e., only synthetic data and synthetic and real-world data) to verify the generalization ability of MH6D. Extensive experiments on benchmark datasets demonstrate that MH6D achieves state-of-the-art (SOTA) performance, outperforming most data-driven methods even without using any real-world data. The code is available at https://github.com/CNJianLiu/MH6D.

Deep Ensemble Transformers for Dimensionality Reduction

We propose deep ensemble transformers (DETs), a fast, scalable approach for dimensionality reduction problems. This method leverages the power of deep neural networks and employs cascade ensemble techniques as its fundamental feature extraction tool. To handle high-dimensional data, our approach employs a flexible number of intermediate layers sequentially. These layers progressively transform the input data into decision tree predictions. To further enhance prediction performance, the output from the final intermediate layer is fed through a feed-forward neural network architecture for final prediction. We derive an upper bound of the disparity between the generalization error and the empirical error and demonstrate that it converges to zero. This highlights the generalizability of our method to parameter estimation and feature selection problems. In our experimental evaluations, DETs outperform existing models in terms of prediction accuracy, representation learning ability, and computational time. Specifically, the method achieves over 95% accuracy in gene expression data and can be trained on average 50% faster than traditional artificial neural networks (ANNs).

BIGFormer: A Graph Transformer With Local Structure Awareness for Diagnosis and Pathogenesis Identification of Alzheimer's Disease Using Imaging Genetic Data

Alzheimer's disease (AD) is a highly inheritable neurological disorder, and brain imaging genetics (BIG) has become a rapidly advancing field for comprehensive understanding its pathogenesis. However, most of the existing approaches underestimate the complexity of the interactions among factors that cause AD. To take full appreciate of these complexity interactions, we propose BIGFormer, a graph Transformer with local structural awareness, for AD diagnosis and identification of pathogenic mechanisms. Specifically, the factors interaction graph is constructed with lesion brain regions and risk genes as nodes, where the connection between nodes intuitively represents the interaction between nodes. After that, a perception with local structure awareness is built to extract local structure around nodes, which is then injected into node representation. Then, the global reliance inference component assembles the local structure into higher-order structure, and multi-level interaction structures are jointly aggregated into a classification projection head for disease state prediction. Experimental results show that BIGFormer demonstrated superiority in four classification tasks on the AD neuroimaging initiative dataset and proved to identify biomarkers closely intimately related to AD.

Breast Tumor Diagnosis Based on Molecular Learning Vector Quantization Neural Networks

DNA nanotechnology plays a crucial role in precise cancer medicine. Currently, molecular logic circuits are applied to detect tumor-specific biomarkers and control the release of therapeutic drugs. However, these systems lack self-learning capabilities for intelligent diagnostics in biological samples, and their data processing capabilities are limited. Here, a molecular learning vector quantization neural network (LVQNN) model based on DNA strand displacement (DSD) technology for breast tumor diagnosis is developed. Compared to previous work, the molecular LVQNN boasts powerful computing abilities, handling high-dimensional data for intelligent cancer diagnosis. To verify the feasibility and versatility of the network, two distinct typical datasets are selected: one from a single source with cell morphology data from 569 cases, and a more extensive one spanning different populations and ages, with miRNA gene expression data from 1881 cases. By using the molecular LVQNN, diagnostic experiments are conducted on 50 and 120 public individuals from these two datasets, respectively, achieving accuracy rates of 94% and 97.5%. This study demonstrates that the LVQNN model exhibits significant advantages in breast cancer diagnosis and enhances diagnostic accuracy while introducing new approaches for intelligent cancer diagnosis, anticipated to bring significant breakthroughs and application prospects to precise cancer medicine.

Improving Speech Recognition Performance in Noisy Environments by Enhancing Lip Reading Accuracy

The current accuracy of speech recognition can reach over 97% on different datasets, but in noisy environments, it is greatly reduced. Improving speech recognition performance in noisy environments is a challenging task. Due to the fact that visual information is not affected by noise, researchers often use lip information to help to improve speech recognition performance. This is where the performance of lip recognition and the effect of cross-modal fusion are particularly important. In this paper, we try to improve the accuracy of speech recognition in noisy environments by improving the lip reading performance and the cross-modal fusion effect. First, due to the same lip possibly containing multiple meanings, we constructed a one-to-many mapping relationship model between lips and speech allowing for the lip reading model to consider which articulations are represented from the input lip movements. Audio representations are also preserved by modeling the inter-relationships between paired audiovisual representations. At the inference stage, the preserved audio representations could be extracted from memory by the learned inter-relationships using only video input. Second, a joint cross-fusion model using the attention mechanism could effectively exploit complementary intermodal relationships, and the model calculates cross-attention weights on the basis of the correlations between joint feature representations and individual modalities. Lastly, our proposed model achieved a 4.0% reduction in WER in a -15 dB SNR environment compared to the baseline method, and a 10.1% reduction in WER compared to speech recognition. The experimental results show that our method could achieve a significant improvement over speech recognition models in different noise environments.