1
|
Simar C, Colot M, Cebolla AM, Petieau M, Cheron G, Bontempi G. Machine learning for hand pose classification from phasic and tonic EMG signals during bimanual activities in virtual reality. Front Neurosci 2024; 18:1329411. [PMID: 38737097 PMCID: PMC11082314 DOI: 10.3389/fnins.2024.1329411] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2023] [Accepted: 04/12/2024] [Indexed: 05/14/2024] Open
Abstract
Myoelectric prostheses have recently shown significant promise for restoring hand function in individuals with upper limb loss or deficiencies, driven by advances in machine learning and increasingly accessible bioelectrical signal acquisition devices. Here, we first introduce and validate a novel experimental paradigm using a virtual reality headset equipped with hand-tracking capabilities to facilitate the recordings of synchronized EMG signals and hand pose estimation. Using both the phasic and tonic EMG components of data acquired through the proposed paradigm, we compare hand gesture classification pipelines based on standard signal processing features, convolutional neural networks, and covariance matrices with Riemannian geometry computed from raw or xDAWN-filtered EMG signals. We demonstrate the performance of the latter for gesture classification using EMG signals. We further hypothesize that introducing physiological knowledge in machine learning models will enhance their performances, leading to better myoelectric prosthesis control. We demonstrate the potential of this approach by using the neurophysiological integration of the "move command" to better separate the phasic and tonic components of the EMG signals, significantly improving the performance of sustained posture recognition. These results pave the way for the development of new cutting-edge machine learning techniques, likely refined by neurophysiology, that will further improve the decoding of real-time natural gestures and, ultimately, the control of myoelectric prostheses.
Collapse
Affiliation(s)
- Cédric Simar
- Machine Learning Group, Computer Science Department, Université Libre de Bruxelles, Brussels, Belgium
| | - Martin Colot
- Machine Learning Group, Computer Science Department, Université Libre de Bruxelles, Brussels, Belgium
| | - Ana-Maria Cebolla
- Laboratory of Neurophysiology and Movement Biomechanics, ULB Neuroscience Institute, Université Libre de Bruxelles, Brussels, Belgium
| | - Mathieu Petieau
- Laboratory of Neurophysiology and Movement Biomechanics, ULB Neuroscience Institute, Université Libre de Bruxelles, Brussels, Belgium
| | - Guy Cheron
- Laboratory of Neurophysiology and Movement Biomechanics, ULB Neuroscience Institute, Université Libre de Bruxelles, Brussels, Belgium
- Laboratory of Electrophysiology, Université de Mons-Hainaut, Mons, Belgium
| | - Gianluca Bontempi
- Machine Learning Group, Computer Science Department, Université Libre de Bruxelles, Brussels, Belgium
| |
Collapse
|
2
|
Li X, Hao J, Li J, Zhao Z, Shang X, Li M. Pathway Activation Analysis for Pan-Cancer Personalized Characterization Based on Riemannian Manifold. Int J Mol Sci 2024; 25:4411. [PMID: 38673997 PMCID: PMC11050713 DOI: 10.3390/ijms25084411] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2024] [Revised: 04/08/2024] [Accepted: 04/11/2024] [Indexed: 04/28/2024] Open
Abstract
The pathogenesis of carcinoma is believed to come from the combined effect of polygenic variation, and the initiation and progression of malignant tumors are closely related to the dysregulation of biological pathways. Quantifying the alteration in pathway activation and identifying coordinated patterns of pathway dysfunction are the imperative part of understanding the malignancy process and distinguishing different tumor stages or clinical outcomes of individual patients. In this study, we have conducted in silico pathway activation analysis using Riemannian manifold (RiePath) toward pan-cancer personalized characterization, which is the first attempt to apply the Riemannian manifold theory to measure the extent of pathway dysregulation in individual patient on the tangent space of the Riemannian manifold. RiePath effectively integrates pathway and gene expression information, not only generating a relatively low-dimensional and biologically relevant representation, but also identifying a robust panel of biologically meaningful pathway signatures as biomarkers. The pan-cancer analysis across 16 cancer types reveals the capability of RiePath to evaluate pathway activation accurately and identify clinical outcome-related pathways. We believe that RiePath has the potential to provide new prospects in understanding the molecular mechanisms of complex diseases and may find broader applications in predicting biomarkers for other intricate diseases.
Collapse
Affiliation(s)
- Xingyi Li
- School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, China; (X.L.); (J.H.); (X.S.)
| | - Jun Hao
- School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, China; (X.L.); (J.H.); (X.S.)
| | - Junming Li
- School of Software, Northwestern Polytechnical University, Xi’an 710072, China; (J.L.); (Z.Z.)
| | - Zhelin Zhao
- School of Software, Northwestern Polytechnical University, Xi’an 710072, China; (J.L.); (Z.Z.)
| | - Xuequn Shang
- School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, China; (X.L.); (J.H.); (X.S.)
| | - Min Li
- School of Computer Science and Engineering, Central South University, Changsha 410083, China
| |
Collapse
|
3
|
Cai G, Zhang F, Yang B, Huang S, Ma T. Manifold Learning-Based Common Spatial Pattern for EEG Signal Classification. IEEE J Biomed Health Inform 2024; 28:1971-1981. [PMID: 38265900 DOI: 10.1109/jbhi.2024.3357995] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/26/2024]
Abstract
EEG signal classification using Riemannian manifolds has shown great potential. However, the huge computational cost associated with Riemannian metrics poses challenges for applying Riemannian methods, particularly in high-dimensional feature data. To address these, we propose an efficient ensemble method called MLCSP-TSE-MLP, which aims to reduce the computational cost while achieving superior performance. MLCSP of the ensemble utilizes a Riemannian graph embedding strategy to learn intrinsic low-dimensional sub-manifolds, enhancing discrimination. TSE uses the Euclidean mean as the reference point for tangent space mapping and reducing computational cost. Finally, the ensemble incorporates the MLP classifier to offer improved classification performance. Classification results conducted on three datasets demonstrate that MLCSP-TSE-MLP achieves significant superior performance compared to various competing methods. Notably, the MLCSP-TSE module achieves a remarkable increase in training speed and exhibits much lower test time compared to traditional Riemannian methods. Based on these results, we believe that the proposed MLCSP-TSE-MLP is a powerful tool for handling high-dimensional data and holds great potential for practical applications.
Collapse
|
4
|
Benisty H, Barson D, Moberly AH, Lohani S, Tang L, Coifman RR, Crair MC, Mishne G, Cardin JA, Higley MJ. Rapid fluctuations in functional connectivity of cortical networks encode spontaneous behavior. Nat Neurosci 2024; 27:148-158. [PMID: 38036743 DOI: 10.1038/s41593-023-01498-y] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2022] [Accepted: 10/16/2023] [Indexed: 12/02/2023]
Abstract
Experimental work across species has demonstrated that spontaneously generated behaviors are robustly coupled to variations in neural activity within the cerebral cortex. Functional magnetic resonance imaging data suggest that temporal correlations in cortical networks vary across distinct behavioral states, providing for the dynamic reorganization of patterned activity. However, these data generally lack the temporal resolution to establish links between cortical signals and the continuously varying fluctuations in spontaneous behavior observed in awake animals. Here, we used wide-field mesoscopic calcium imaging to monitor cortical dynamics in awake mice and developed an approach to quantify rapidly time-varying functional connectivity. We show that spontaneous behaviors are represented by fast changes in both the magnitude and correlational structure of cortical network activity. Combining mesoscopic imaging with simultaneous cellular-resolution two-photon microscopy demonstrated that correlations among neighboring neurons and between local and large-scale networks also encode behavior. Finally, the dynamic functional connectivity of mesoscale signals revealed subnetworks not predicted by traditional anatomical atlas-based parcellation of the cortex. These results provide new insights into how behavioral information is represented across the neocortex and demonstrate an analytical framework for investigating time-varying functional connectivity in neural networks.
Collapse
Affiliation(s)
- Hadas Benisty
- Department of Neuroscience, Kavli Institute for Neuroscience, Yale University School of Medicine, New Haven, CT, USA
| | - Daniel Barson
- Department of Neuroscience, Kavli Institute for Neuroscience, Yale University School of Medicine, New Haven, CT, USA
| | - Andrew H Moberly
- Department of Neuroscience, Kavli Institute for Neuroscience, Yale University School of Medicine, New Haven, CT, USA
| | - Sweyta Lohani
- Department of Neuroscience, Kavli Institute for Neuroscience, Yale University School of Medicine, New Haven, CT, USA
| | - Lan Tang
- Department of Neuroscience, Kavli Institute for Neuroscience, Yale University School of Medicine, New Haven, CT, USA
| | - Ronald R Coifman
- Program in Applied Mathematics, Yale University, New Haven, CT, USA
| | - Michael C Crair
- Department of Neuroscience, Kavli Institute for Neuroscience, Yale University School of Medicine, New Haven, CT, USA
| | - Gal Mishne
- Halıcıoğlu Data Science Institute, University of California, San Diego, La Jolla, CA, USA
| | - Jessica A Cardin
- Department of Neuroscience, Kavli Institute for Neuroscience, Yale University School of Medicine, New Haven, CT, USA
| | - Michael J Higley
- Department of Neuroscience, Kavli Institute for Neuroscience, Yale University School of Medicine, New Haven, CT, USA.
| |
Collapse
|
5
|
Wilroth J, Bernhardsson B, Heskebeck F, Skoglund MA, Bergeling C, Alickovic E. Improving EEG-based decoding of the locus of auditory attention through domain adaptation . J Neural Eng 2023; 20:066022. [PMID: 37988748 DOI: 10.1088/1741-2552/ad0e7b] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2022] [Accepted: 11/21/2023] [Indexed: 11/23/2023]
Abstract
Objective.This paper presents a novel domain adaptation (DA) framework to enhance the accuracy of electroencephalography (EEG)-based auditory attention classification, specifically for classifying the direction (left or right) of attended speech. The framework aims to improve the performances for subjects with initially low classification accuracy, overcoming challenges posed by instrumental and human factors. Limited dataset size, variations in EEG data quality due to factors such as noise, electrode misplacement or subjects, and the need for generalization across different trials, conditions and subjects necessitate the use of DA methods. By leveraging DA methods, the framework can learn from one EEG dataset and adapt to another, potentially resulting in more reliable and robust classification models.Approach.This paper focuses on investigating a DA method, based on parallel transport, for addressing the auditory attention classification problem. The EEG data utilized in this study originates from an experiment where subjects were instructed to selectively attend to one of the two spatially separated voices presented simultaneously.Main results.Significant improvement in classification accuracy was observed when poor data from one subject was transported to the domain of good data from different subjects, as compared to the baseline. The mean classification accuracy for subjects with poor data increased from 45.84% to 67.92%. Specifically, the highest achieved classification accuracy from one subject reached 83.33%, a substantial increase from the baseline accuracy of 43.33%.Significance.The findings of our study demonstrate the improved classification performances achieved through the implementation of DA methods. This brings us a step closer to leveraging EEG in neuro-steered hearing devices.
Collapse
Affiliation(s)
- Johanna Wilroth
- Department of Electrical Engineering, Linkoping University, Linkoping, Sweden
| | - Bo Bernhardsson
- Department of Automatic Control, Lund University, Lund, Sweden
| | - Frida Heskebeck
- Department of Automatic Control, Lund University, Lund, Sweden
| | - Martin A Skoglund
- Department of Electrical Engineering, Linkoping University, Linkoping, Sweden
- Eriksholm Research Centre, Oticon A/S, Snekkersten, Denmark
| | - Carolina Bergeling
- Department of Mathematics and Natural Sciences, Blekinge Institute of Technology, Karlskrona, Sweden
| | - Emina Alickovic
- Department of Electrical Engineering, Linkoping University, Linkoping, Sweden
- Eriksholm Research Centre, Oticon A/S, Snekkersten, Denmark
| |
Collapse
|
6
|
Liu Z, Zhou X, Zhou T, Chen Y. Foreground Segmentation-Based Density Grading Networks for Crowd Counting. SENSORS (BASEL, SWITZERLAND) 2023; 23:8177. [PMID: 37837007 PMCID: PMC10575052 DOI: 10.3390/s23198177] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/16/2023] [Revised: 09/20/2023] [Accepted: 09/26/2023] [Indexed: 10/15/2023]
Abstract
Estimating object counts within a single image or video frame represents a challenging yet pivotal task in the field of computer vision. Its increasing significance arises from its versatile applications across various domains, including public safety and urban planning. Among the various object counting tasks, crowd counting is particularly notable for its critical role in social security and urban planning. However, intricate backgrounds in images often lead to misidentifications, wherein the complex background is mistaken as the foreground, thereby inflating forecasting errors. Additionally, the uneven distribution of crowd density within the foreground further exacerbates predictive errors of the network. This paper introduces a novel architecture with a three-branch structure aimed at synergistically incorporating hierarchical foreground information and global scale information into density map estimation, thereby achieving more precise counting results. Hierarchical foreground information guides the network to perform distinct operations on regions with varying densities, while global scale information evaluates the overall density level of the image and adjusts the model's global predictions accordingly. We also systematically investigate and compare three potential locations for integrating hierarchical foreground information into the density estimation network, ultimately determining the most effective placement.Through extensive comparative experiments across three datasets, we demonstrate the superior performance of our proposed method.
Collapse
Affiliation(s)
- Zelong Liu
- College of Computer Science, Sichuan University, Chengdu 610000, China; (Z.L.); (X.Z.)
| | - Xin Zhou
- College of Computer Science, Sichuan University, Chengdu 610000, China; (Z.L.); (X.Z.)
| | - Tao Zhou
- School of Automation Engineering, University of Electronic Science and Technology of China, Chengdu 610054, China;
| | - Yuanyuan Chen
- College of Computer Science, Sichuan University, Chengdu 610000, China; (Z.L.); (X.Z.)
| |
Collapse
|
7
|
Ding X, Yang L, Li C. Study of MI-BCI classification method based on the Riemannian transform of personalized EEG spatiotemporal features. MATHEMATICAL BIOSCIENCES AND ENGINEERING : MBE 2023; 20:12454-12471. [PMID: 37501450 DOI: 10.3934/mbe.2023554] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/29/2023]
Abstract
Motor imagery (MI) is a traditional paradigm of brain-computer interface (BCI) and can assist users in creating direct connections between their brains and external equipment. The common spatial patterns algorithm is the most popular spatial filtering technique for collecting EEG signal features in MI-based BCI systems. Due to the defect that it only considers the spatial information of EEG signals and is susceptible to noise interference and other issues, its performance is diminished. In this study, we developed a Riemannian transform feature extraction method based on filter bank fusion with a combination of multiple time windows. First, we proposed the multi-time window data segmentation and recombination method by combining it with a filter group to create new data samples. This approach could capture individual differences due to the variation in time-frequency patterns across different participants, thereby improving the model's generalization performance. Second, Riemannian geometry was used for feature extraction from non-Euclidean structured EEG data. Then, considering the non-Gaussian distribution of EEG signals, the neighborhood component analysis (NCA) algorithm was chosen for feature selection. Finally, to meet real-time requirements and a low complexity, we employed a Support Vector Machine (SVM) as the classification algorithm. The proposed model achieved improved accuracy and robustness. In this study, we proposed an algorithm with superior performance on the BCI Competition IV dataset 2a, achieving an accuracy of 89%, a kappa value of 0.73 and an AUC of 0.9, demonstrating advanced capabilities. Furthermore, we analyzed data collected in our laboratory, and the proposed method achieved an accuracy of 77.4%, surpassing other comparative models. This method not only significantly improved the classification accuracy of motor imagery EEG signals but also bore significant implications for applications in the fields of brain-computer interfaces and neural engineering.
Collapse
Affiliation(s)
- Xiaotong Ding
- China Academy of Information and Communications Technology, Beijing, China
| | - Lei Yang
- China Academy of Information and Communications Technology, Beijing, China
| | - Congsheng Li
- China Academy of Information and Communications Technology, Beijing, China
| |
Collapse
|
8
|
Yamin MA, Valsasina P, Tessadori J, Filippi M, Murino V, Rocca MA, Sona D. Discovering functional connectivity features characterizing multiple sclerosis phenotypes using explainable artificial intelligence. Hum Brain Mapp 2023; 44:2294-2306. [PMID: 36715247 PMCID: PMC10028625 DOI: 10.1002/hbm.26210] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2022] [Revised: 12/14/2022] [Accepted: 01/02/2023] [Indexed: 01/31/2023] Open
Abstract
Multiple sclerosis (MS) is a neurological condition characterized by severe structural brain damage and by functional reorganization of the main brain networks that try to limit the clinical consequences of structural burden. Resting-state (RS) functional connectivity (FC) abnormalities found in this condition were shown to be variable across different MS phases, according to the severity of clinical manifestations. The article describes a system exploiting machine learning on RS FC matrices to discriminate different MS phenotypes and to identify relevant functional connections for MS stage characterization. To this end, the system exploits some mathematical properties of covariance-based RS FC representation, which can be described by a Riemannian manifold. The classification performance of the proposed framework was significantly above the chance level for all MS phenotypes. Moreover, the proposed system was successful in identifying relevant RS FC alterations contributing to an accurate phenotype classification.
Collapse
Affiliation(s)
- Muhammad Abubakar Yamin
- Department of Electrical Engineering and Automation, Aalto University, Espoo, Finland
- Department of Neuroscience and Biomedical Engineering, Aalto University, Espoo, Finland
- Pattern Analysis and Computer Vision, Istituto Italiano di Tecnologia, Genova, Italy
- Center for Autism Research, Kessler Foundation, East Hanover, New Jersey, USA
| | - Paola Valsasina
- Neuroimaging Research Unit, Division of Neuroscience, IRCCS San Raffaele Scientific Institute, Milan, Italy
| | - Jacopo Tessadori
- Pattern Analysis and Computer Vision, Istituto Italiano di Tecnologia, Genova, Italy
- Dipartimento di Informatica, University of Verona, Verona, Italy
| | - Massimo Filippi
- Neuroimaging Research Unit, Division of Neuroscience, IRCCS San Raffaele Scientific Institute, Milan, Italy
- Neurology Unit, IRCCS San Raffaele Scientific Institute, Milan, Italy
- Neurophysiology Service, IRCCS San Raffaele Scientific Institute, Milan, Italy
- Neurorehabilitation Unit, IRCCS San Raffaele Scientific Institute, Milan, Italy
- Vita Salute San Raffaele University, Milan, Italy
| | - Vittorio Murino
- Pattern Analysis and Computer Vision, Istituto Italiano di Tecnologia, Genova, Italy
- Dipartimento di Informatica, University of Verona, Verona, Italy
| | - Maria A Rocca
- Neuroimaging Research Unit, Division of Neuroscience, IRCCS San Raffaele Scientific Institute, Milan, Italy
- Neurology Unit, IRCCS San Raffaele Scientific Institute, Milan, Italy
- Vita Salute San Raffaele University, Milan, Italy
| | - Diego Sona
- Pattern Analysis and Computer Vision, Istituto Italiano di Tecnologia, Genova, Italy
- Data Science for Health, Center for Digital Health and Wellbeing, Fondazione Bruno Kessler, Trento, Italy
| |
Collapse
|
9
|
Wang R, Wu XJ, Xu T, Hu C, Kittler J. U-SPDNet: An SPD manifold learning-based neural network for visual classification. Neural Netw 2023; 161:382-396. [PMID: 36780861 DOI: 10.1016/j.neunet.2022.11.030] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2022] [Revised: 11/07/2022] [Accepted: 11/27/2022] [Indexed: 12/15/2022]
Abstract
With the development of neural networking techniques, several architectures for symmetric positive definite (SPD) matrix learning have recently been put forward in the computer vision and pattern recognition (CV&PR) community for mining fine-grained geometric features. However, the degradation of structural information during multi-stage feature transformation limits their capacity. To cope with this issue, this paper develops a U-shaped neural network on the SPD manifolds (U-SPDNet) for visual classification. The designed U-SPDNet contains two subsystems, one of which is a shrinking path (encoder) making up of a prevailing SPD manifold neural network (SPDNet (Huang and Van Gool, 2017)) for capturing compact representations from the input data. Another is a constructed symmetric expanding path (decoder) to upsample the encoded features, trained by a reconstruction error term. With this design, the degradation problem will be gradually alleviated during training. To enhance the representational capacity of U-SPDNet, we also append skip connections from encoder to decoder, realized by manifold-valued geometric operations, namely Riemannian barycenter and Riemannian optimization. On the MDSD, Virus, FPHA, and UAV-Human datasets, the accuracy achieved by our method is respectively 6.92%, 8.67%, 1.57%, and 1.08% higher than SPDNet, certifying its effectiveness.
Collapse
Affiliation(s)
- Rui Wang
- School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China; Jiangsu Provincial Engineering Laboratory of Pattern Recognition and Computational Intelligence, Jiangnan University, Wuxi 214122, China
| | - Xiao-Jun Wu
- School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China; Jiangsu Provincial Engineering Laboratory of Pattern Recognition and Computational Intelligence, Jiangnan University, Wuxi 214122, China.
| | - Tianyang Xu
- School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China; Jiangsu Provincial Engineering Laboratory of Pattern Recognition and Computational Intelligence, Jiangnan University, Wuxi 214122, China
| | - Cong Hu
- School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China; Jiangsu Provincial Engineering Laboratory of Pattern Recognition and Computational Intelligence, Jiangnan University, Wuxi 214122, China
| | - Josef Kittler
- School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China; Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, Guildford GU2 7XH, UK
| |
Collapse
|
10
|
Dilek E, Dener M. Computer Vision Applications in Intelligent Transportation Systems: A Survey. SENSORS (BASEL, SWITZERLAND) 2023; 23:2938. [PMID: 36991649 PMCID: PMC10051529 DOI: 10.3390/s23062938] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/24/2023] [Revised: 03/03/2023] [Accepted: 03/06/2023] [Indexed: 06/19/2023]
Abstract
As technology continues to develop, computer vision (CV) applications are becoming increasingly widespread in the intelligent transportation systems (ITS) context. These applications are developed to improve the efficiency of transportation systems, increase their level of intelligence, and enhance traffic safety. Advances in CV play an important role in solving problems in the fields of traffic monitoring and control, incident detection and management, road usage pricing, and road condition monitoring, among many others, by providing more effective methods. This survey examines CV applications in the literature, the machine learning and deep learning methods used in ITS applications, the applicability of computer vision applications in ITS contexts, the advantages these technologies offer and the difficulties they present, and future research areas and trends, with the goal of increasing the effectiveness, efficiency, and safety level of ITS. The present review, which brings together research from various sources, aims to show how computer vision techniques can help transportation systems to become smarter by presenting a holistic picture of the literature on different CV applications in the ITS context.
Collapse
|
11
|
Zhao Z, Li X. Deformable Density Estimation via Adaptive Representation. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2023; PP:1134-1144. [PMID: 37022433 DOI: 10.1109/tip.2023.3240839] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Crowd counting is the basic task of crowd analysis and it is of great significance in the field of public safety. Therefore, it receives more and more attention recently. The common idea is to combine the crowd counting task with convolutional neural networks to predict the corresponding density map, which is generated by filtering the dot labels with specific Gaussian kernels. Although the counting performance is promoted by the newly proposed networks, they all suffer one conjunct problem, which is due to the perspective effect, there is significant scale contrast among targets in different positions within one scene, but the existing density maps can not represent this scale change well. To address the prediction difficulties caused by target scale variation, we propose a scale-sensitive crowd density map estimation framework, which focuses on dealing with target scale change from density map generation, network design, and model training stage. It consists of the Adaptive Density Map (ADM), Deformable Density Map Decoder (DDMD), and Auxiliary Branch. To be specific, the Gaussian kernel size variates adaptively based on target size to generate ADM that contains scale information for each specific target. DDMD introduces the deformable convolution to fit the Gaussian kernel variation and boosts the model's scale sensitivity. The Auxiliary Branch guides the learning of deformable convolution offsets during the training phase. Finally, we construct experiments on different large-scale datasets. The results show the effectiveness of the proposed ADM and DDMD. Furthermore, the visualization demonstrates that deformable convolution learns the target scale variation.
Collapse
|
12
|
Zois EN, Said S, Tsourounis D, Alexandridis A. Subscripto multiplex: a Riemannian symmetric positive definite strategy for offline signature verification. Pattern Recognit Lett 2023. [DOI: 10.1016/j.patrec.2023.02.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/05/2023]
|
13
|
Yacine F, Salah H, Amar K, Ahmad K. A novel ANN adaptive Riemannian-based kernel classification for motor imagery. Biomed Phys Eng Express 2022; 9. [PMID: 36535004 DOI: 10.1088/2057-1976/acaca2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2022] [Accepted: 12/19/2022] [Indexed: 12/23/2022]
Abstract
More recently, a number of studies show the interest of the use of the Riemannian geometry in EEG classification. The idea is to exploit the EEG covariance matrices, instead of the raw EEG data, and use the Riemannian geometry to directly classify these matrices. This paper presents a novel Artificial Neural Network approach based on an Adaptive Riemannian Kernel, named ARK-ANN, to classify Electroencephalographic (EEG) motor imaging signals in the context of Brain Computer Interface (BCI). A multilayer perceptron is used to classify the covariance matrices of Motor Imagery (MI) signals employing an adaptive optimization of the testing set. The contribution of a geodesic filter is also assessed for the ANN and the original method which uses an SVM classifier. The results demonstrate that the ARK-ANN performs better than the other methods and the geodesic filter gives slightly better results in the ARK-SVM, considered here as the reference method, in the case of inter-subject classification (accuracy of 87.4% and 86% for ARK-ANN and ARK-SVM, respectively). Regarding the cross-subject classification, the proposed method gives an accuracy of 77.3% and increases the precision by 8.2% in comparison to the SVM based method.
Collapse
Affiliation(s)
- Fodil Yacine
- Laboratoire d'Analyse et Modélisation des Phénomènes Aléatoires (LAMPA), University Mouloud Mammeri of Tizi-Ouzou (UMMTO), Algeria
| | - Haddab Salah
- Laboratoire d'Analyse et Modélisation des Phénomènes Aléatoires (LAMPA), University Mouloud Mammeri of Tizi-Ouzou (UMMTO), Algeria
| | - Kachenoura Amar
- Laboratoire Traitement du Signal et de l'Image (LTSI), University Rennes, Inserm, LTSI-UMR 1099, Rennes, France
| | - Karfoul Ahmad
- Laboratoire Traitement du Signal et de l'Image (LTSI), University Rennes, Inserm, LTSI-UMR 1099, Rennes, France
| |
Collapse
|
14
|
Peng W, Varanka T, Mostafa A, Shi H, Zhao G. Hyperbolic Deep Neural Networks: A Survey. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:10023-10044. [PMID: 34932472 DOI: 10.1109/tpami.2021.3136921] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Recently, hyperbolic deep neural networks (HDNNs) have been gaining momentum as the deep representations in the hyperbolic space provide high fidelity embeddings with few dimensions, especially for data possessing hierarchical structure. Such a hyperbolic neural architecture is quickly extended to different scientific fields, including natural language processing, single-cell RNA-sequence analysis, graph embedding, financial analysis, and computer vision. The promising results demonstrate its superior capability, significant compactness of the model, and a substantially better physical interpretability than its counterpart in the euclidean space. To stimulate future research, this paper presents a comprehensive review of the literature around the neural components in the construction of HDNN, as well as the generalization of the leading deep approaches to the hyperbolic space. It also presents current applications of various tasks, together with insightful observations and identifying open questions and promising future directions.
Collapse
|
15
|
Smith A, Laubach B, Castillo I, Zavala VM. Data analysis using Riemannian geometry and applications to chemical engineering. Comput Chem Eng 2022. [DOI: 10.1016/j.compchemeng.2022.108023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/03/2022]
|
16
|
Supervised and Semisupervised Manifold Embedded Knowledge Transfer in Motor Imagery-Based BCI. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2022; 2022:1603104. [PMID: 36299440 PMCID: PMC9592202 DOI: 10.1155/2022/1603104] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/13/2022] [Revised: 08/14/2022] [Accepted: 09/01/2022] [Indexed: 11/17/2022]
Abstract
A long calibration procedure limits the use in practice for a motor imagery (MI)-based brain-computer interface (BCI) system. To tackle this problem, we consider supervised and semisupervised transfer learning. However, it is a challenge for them to cope with high intersession/subject variability in the MI electroencephalographic (EEG) signals. Based on the framework of unsupervised manifold embedded knowledge transfer (MEKT), we propose a supervised MEKT algorithm (sMEKT) and a semisupervised MEKT algorithm (ssMEKT), respectively. sMEKT only has limited labelled samples from a target subject and abundant labelled samples from multiple source subjects. Compared to sMEKT, ssMEKT adds comparably more unlabelled samples from the target subject. After performing Riemannian alignment (RA) and tangent space mapping (TSM), both sMEKT and ssMEKT execute domain adaptation to shorten the differences among subjects. During domain adaptation, to make use of the available samples, two algorithms preserve the source domain discriminability, and ssMEKT preserves the geometric structure embedded in the labelled and unlabelled target domains. Moreover, to obtain a subject-specific classifier, sMEKT minimizes the joint probability distribution shift between the labelled target and source domains, whereas ssMEKT performs the joint probability distribution shift minimization between the unlabelled target domain and all labelled domains. Experimental results on two publicly available MI datasets demonstrate that our algorithms outperform the six competing algorithms, where the sizes of labelled and unlabelled target domains are variable. Especially for the target subjects with 10 labelled samples and 270/190 unlabelled samples, ssMEKT shows 5.27% and 2.69% increase in average accuracy on the two abovementioned datasets compared to the previous best semisupervised transfer learning algorithm (RA-regularized common spatial patterns-weighted adaptation regularization, RA-RCSP-wAR), respectively. Therefore, our algorithms can effectively reduce the need of labelled samples for the target subject, which is of importance for the MI-based BCI application.
Collapse
|
17
|
Dan T, Huang Z, Cai H, Laurienti PJ, Wu G. Learning Brain Dynamics of Evolving Manifold Functional MRI Data Using Geometric-Attention Neural Network. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:2752-2763. [PMID: 35452386 PMCID: PMC10505045 DOI: 10.1109/tmi.2022.3169640] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Functional connectivities (FC) of brain network manifest remarkable geometric patterns, which is the gateway to understanding brain dynamics. In this work, we present a novel geometric-attention neural network to characterize the time-evolving brain state change from the functional neuroimages by tracking the trajectory of functional dynamics on high-dimension Riemannian manifold of symmetric positive definite (SPD) matrices. Specifically, we put the spotlight on learning the common state-specific manifold signatures that represent the underlying cognition. In this context, the driving force of our neural network is tied up with the learning of the evolution functionals on the Riemannian manifold of SPD matrix that underlies the known evolving brain states. To do so, we train a convolution neural network (CNN) on the Riemannian manifold of SPD matrices to seek for the putative low-dimension feature representations, followed by an end-to-end recurrent neural network (RNN) to yield the time-varying mapping function of SPD matrices which fits the evolutionary trajectories of the underlying states. Furthermore, we devise a geometric attention mechanism in CNN, allowing us to discover the latent geometric patterns in SPD matrices that are associated with the underlying states. Notably, our work has the potential to understand how brain function emerges behavior by investigating the geometrical patterns from functional brain networks, which is essentially a correlation matrix of neuronal activity signals. Our proposed manifold-based neural network achieves promising results in predicting brain state changes on both simulated data and task functional neuroimaging data from Human Connectome Project, which implies great applicability in neuroscience studies.
Collapse
|
18
|
Qu T, Jin J, Xu R, Wang X, Cichocki A. Riemannian distance based channel selection and feature extraction combining discriminative time-frequency bands and Riemannian tangent space for MI-BCIs. J Neural Eng 2022; 19. [PMID: 36126643 DOI: 10.1088/1741-2552/ac9338] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2022] [Accepted: 09/20/2022] [Indexed: 11/12/2022]
Abstract
OBJECTIVE Motor imagery-based brain computer interfaces (MI-BCIs) have been widely researched because they do not demand external stimuli and have a high degree of maneuverability. In most scenarios, superabundant selected channels, fixed time windows, and frequency bands would certainly affect the performance of MI-BCIs due to the neurophysiological diversities between different individuals. In this study, we tended to effectively use the Riemannian geometry of spatial covariance matrix to extract more robust features and thus enhance the decoding efficiency. APPROACH First, we propose a Riemannian distance-based EEG channel selection method (RDCS), which preliminarily reduces the information redundancy in the first stage. Second, we extract discriminative Riemannian Tangent Space features of EEG signals of selected channels from the most discriminant time-frequency bands (DTFRTS) to further enhance decoding accuracy for MI-BCIs. Finally, we trained a support vector machine (SVM) model with a linear kernel to classify our extracted discriminative Riemannian features and evaluated our proposed method using publicly available BCI Competition Ⅳ dataset Ⅰ (DS1) and Competition Ⅲ dataset Ⅲa (DS2). MAIN RESULTS The experimental results showed that the average classification accuracy with the selected 10-channel EEG signals of our method is 88.1% and 91.6% in DS1 and DS2 respectively. The average improvements are 24.3% & 27.1% on DS1 and 4.4% & 14.2% on DS2 for 10 & 20 selected channels, respectively. SIGNIFICANCE These results showed that our proposed method is a promising candidate for performance improvement of MI-BCIs.
Collapse
Affiliation(s)
- Tingnan Qu
- Key Laboratory of Advanced Control and Optimization for Chemical Processes Ministry of Education, East China University of Science and Technology, 130 Meilong Road, Xuhui District, Shanghai, 200237, CHINA
| | - Jing Jin
- Key Laboratory of Advanced Control and Optimization for Chemical Processes Ministry of Education, East China University of Science and Technology, 130 Meilong Road, Xuhui District, Shanghai 200237, Shanghai, Shanghai, 200237, CHINA
| | - Ren Xu
- Guger Technologies OG, Research and Software Developmentg.tec - Guger Technologies Sierningstrasse 14, 4521 Schiedlberg, Graz, 8020, AUSTRIA
| | - Xingyu Wang
- Key Laboratory of Advanced Control and Optimization for Chemical Processes Ministry of Education, East China University of Science and Technology, 130 Meilong Road, Shanghai, Shanghai, 200237, CHINA
| | - Andrzej Cichocki
- Skolkovo Institute of Science and Technology, Bolshoy Boulevard 30, bld. 1 Moscow, Russia 121205, Skolkovo, Moskovskaâ, 121205, RUSSIAN FEDERATION
| |
Collapse
|
19
|
Convolutional Neural Networks and Heuristic Methods for Crowd Counting: A Systematic Review. SENSORS 2022; 22:s22145286. [PMID: 35890966 PMCID: PMC9315600 DOI: 10.3390/s22145286] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 06/21/2022] [Revised: 07/13/2022] [Accepted: 07/14/2022] [Indexed: 11/27/2022]
Abstract
The crowd counting task has become a pillar for crowd control as it provides information concerning the number of people in a scene. It is helpful in many scenarios such as video surveillance, public safety, and future event planning. To solve such tasks, researchers have proposed different solutions. In the beginning, researchers went with more traditional solutions, while recently the focus is on deep learning methods and, more specifically, on Convolutional Neural Networks (CNNs), because of their efficiency. This review explores these methods by focusing on their key differences, advantages, and disadvantages. We have systematically analyzed algorithms and works based on the different models suggested and the problems they are trying to solve. The main focus is on the shift made in the history of crowd counting methods, moving from the heuristic models to CNN models by identifying each category and discussing its different methods and architectures. After a deep study of the literature on crowd counting, the survey partitions current datasets into sparse and crowded ones. It discusses the reviewed methods by comparing their results on the different datasets. The findings suggest that the heuristic models could be even more effective than the CNN models in sparse scenarios.
Collapse
|
20
|
Improved YOLOv4 for Pedestrian Detection and Counting in UAV Images. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2022; 2022:6106853. [PMID: 35875752 PMCID: PMC9303083 DOI: 10.1155/2022/6106853] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/03/2022] [Revised: 05/09/2022] [Accepted: 06/06/2022] [Indexed: 11/17/2022]
Abstract
UAV (unmanned aerial vehicle) captured images have small pedestrian targets and loss of key information after multiple down sampling, which are difficult to overcome by existing methods. We propose an improved YOLOv4 model for pedestrian detection and counting in UAV images, named YOLO-CC. We used the lightweight YOLOv4 for pedestrian detection, which replaces the backbone with CSPDarknet-34, and two feature layers are fused by FPN (Feature Pyramid Networks). We expanded the perception field using multiscale convolution based on the high-level feature map and generated the population density map by feature dimension reduction. By embedding the density map generation method into the network for end-to-end training, our model can effectively improve the accuracy of detection and counting and make feature extraction more focused on small targets. Our experiments demonstrate that YOLO-CC achieves 21.76 points AP50 higher than that of the original YOLOv4 on the VisDrone2021-counting data set while running faster than the original YOLOv4.
Collapse
|
21
|
Zou J, Zhang Y, Liu H, Ma L. Monogenic features based single sample face recognition by kernel sparse representation on multiple Riemannian manifolds. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2022.06.113] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
22
|
Xiong D, Zhang D, Zhao X, Chu Y, Zhao Y. Learning Non-Euclidean Representations With SPD Manifold for Myoelectric Pattern Recognition. IEEE Trans Neural Syst Rehabil Eng 2022; 30:1514-1524. [PMID: 35622796 DOI: 10.1109/tnsre.2022.3178384] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
How to learn informative representations from Electromyography (EMG) signals is of vital importance for myoelectric control systems. Traditionally, hand-crafted features are extracted from individual EMG channels and combined together for pattern recognition. The spatial topological information between different channels can also be informative, which is seldom considered. This paper presents a radically novel approach to extract spatial structural information within diverse EMG channels based on the symmetric positive definite (SPD) manifold. The object is to learn non-Euclidean representations inside EMG signals for myoelectric pattern recognition. The performance is compared with two classical feature sets using accuracy and F1-score. The algorithm is tested on eleven gestures collected from ten subjects, and the best accuracy reaches 84.85%±5.15% with an improvement of 4.04%~20.25%, which outperforms the contrast method, and reaches a significant improvement with the Wilcoxon signed-rank test. Eleven gestures from three public databases involving Ninapro DB2, DB4, and DB5 are also evaluated, and better performance is observed. Furthermore, the computational cost is less than the contrast method, making it more suitable for low-cost systems. It shows the effectiveness of the presented approach and contributes a new way for myoelectric pattern recognition.
Collapse
|
23
|
A Framework for Short Video Recognition Based on Motion Estimation and Feature Curves on SPD Manifolds. APPLIED SCIENCES-BASEL 2022. [DOI: 10.3390/app12094669] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Given the prosperity of video media such as TikTok and YouTube, the requirement of short video recognition is becoming more and more urgent. A significant feature of short video is that there are few switches of scenes in short video, and the target (e.g., the face of the key person in the short video) often runs through the short video. This paper presents a new short video recognition algorithm framework that transforms a short video into a family of feature curves on symmetric positive definite (SPD) manifold as the basis of recognition. Thus far, no similar algorithm has been reported. The results of experiments suggest that our method performs better on three changeling databases than seven other related algorithms published in the top issues.
Collapse
|
24
|
Domain adaptation for epileptic EEG classification using adversarial learning and Riemannian manifold. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103555] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/24/2023]
|
25
|
Li M, Wang Y, Lopez-Naranjo C, Hu S, Reyes RCG, Paz-Linares D, Areces-Gonzalez A, Hamid AIA, Evans AC, Savostyanov AN, Calzada-Reyes A, Villringer A, Tobon-Quintero CA, Garcia-Agustin D, Yao D, Dong L, Aubert-Vazquez E, Reza F, Razzaq FA, Omar H, Abdullah JM, Galler JR, Ochoa-Gomez JF, Prichep LS, Galan-Garcia L, Morales-Chacon L, Valdes-Sosa MJ, Tröndle M, Zulkifly MFM, Abdul Rahman MRB, Milakhina NS, Langer N, Rudych P, Koenig T, Virues-Alba TA, Lei X, Bringas-Vega ML, Bosch-Bayard JF, Valdes-Sosa PA. Harmonized-Multinational qEEG norms (HarMNqEEG). Neuroimage 2022; 256:119190. [PMID: 35398285 DOI: 10.1016/j.neuroimage.2022.119190] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2022] [Revised: 03/23/2022] [Accepted: 04/05/2022] [Indexed: 12/14/2022] Open
Abstract
This paper extends frequency domain quantitative electroencephalography (qEEG) methods pursuing higher sensitivity to detect Brain Developmental Disorders. Prior qEEG work lacked integration of cross-spectral information omitting important functional connectivity descriptors. Lack of geographical diversity precluded accounting for site-specific variance, increasing qEEG nuisance variance. We ameliorate these weaknesses. (i) Create lifespan Riemannian multinational qEEG norms for cross-spectral tensors. These norms result from the HarMNqEEG project fostered by the Global Brain Consortium. We calculate the norms with data from 9 countries, 12 devices, and 14 studies, including 1564 subjects. Instead of raw data, only anonymized metadata and EEG cross-spectral tensors were shared. After visual and automatic quality control, developmental equations for the mean and standard deviation of qEEG traditional and Riemannian DPs were calculated using additive mixed-effects models. We demonstrate qEEG "batch effects" and provide methods to calculate harmonized z-scores. (ii) We also show that harmonized Riemannian norms produce z-scores with increased diagnostic accuracy predicting brain dysfunction produced by malnutrition in the first year of life and detecting COVID induced brain dysfunction. (iii) We offer open code and data to calculate different individual z-scores from the HarMNqEEG dataset. These results contribute to developing bias-free, low-cost neuroimaging technologies applicable in various health settings.
Collapse
Affiliation(s)
- Min Li
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China
| | - Ying Wang
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China
| | - Carlos Lopez-Naranjo
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China
| | - Shiang Hu
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China; Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, Key Laboratory of Intelligent Computing & Signal Processing of Ministry of Education, School of Computer Science and Technology, Anhui University, Hefei 230601, China
| | | | - Deirel Paz-Linares
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China; Cuban Center for Neurocience, La Habana, Cuba
| | - Ariosky Areces-Gonzalez
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China; University of Pinar del Río "Hermanos Saiz Montes de Oca", Pinar del Río, Cuba
| | - Aini Ismafairus Abd Hamid
- Brain and Behaviour Cluster, School of Medical Sciences, Universiti Sains Malaysia, Health Campus, Kota Bharu, Kelantan 16150, Malaysia; Hospital Universiti Sains Malaysia, Universiti Sains Malaysia, Health Campus, Kota Bharu, Kelantan 16150, Malaysia; McGill Centre for Integrative Neuroscience, Ludmer Centre for Neuroinformatics and Mental Health, Montreal Neurological Institute, Canada
| | - Alan C Evans
- McGill Centre for Integrative Neuroscience, Ludmer Centre for Neuroinformatics and Mental Health, Montreal Neurological Institute, Canada
| | - Alexander N Savostyanov
- Humanitarian Institute, Novosibirsk State University, Novosibirsk 630090, Russia; Laboratory of Psychophysiology of Individual Differences, Federal State Budgetary Scientific Institution Scientific Research Institute of Neurosciences and Medicine, Novosibirsk 630117, Russia; Laboratory of Psychological Genetics at the Institute of Cytology and Genetics Siberian Branch of the Russian Academy of Sciences, Novosibirsk 630090, Russia
| | | | - Arno Villringer
- Department of Neurology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany; Department of Cognitive Neurology, University Hospital Leipzig, Leipzig, Germany; Center for Stroke Research, Charité-Universitätsmedizin Berlin, Berlin, Germany
| | - Carlos A Tobon-Quintero
- Grupo Neuropsicología y Conducta - GRUNECO, Faculty of Medicine, Universidad de Antioquia, Colombia; Research Department, Institución Prestadora de Servicios de Salud IPS Universitaria, Colombia
| | - Daysi Garcia-Agustin
- Cuban Center for Neurocience, La Habana, Cuba; The Cuban center aging longevity and health, Havana Cuba
| | - Dezhong Yao
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China; Research Unit of NeuroInformation, Chinese Academy of Medical Sciences, Chengdu 2019RU035, China; School of Electrical Engineering, Zhengzhou University, Zhengzhou 450001, China
| | - Li Dong
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China; Research Unit of NeuroInformation, Chinese Academy of Medical Sciences, Chengdu 2019RU035, China; Sichuan Institute for Brain Science and Brain-Inspired Intelligence, Chengdu 611731, China
| | | | - Faruque Reza
- Brain and Behaviour Cluster, School of Medical Sciences, Universiti Sains Malaysia, Health Campus, Kota Bharu, Kelantan 16150, Malaysia; Hospital Universiti Sains Malaysia, Universiti Sains Malaysia, Health Campus, Kota Bharu, Kelantan 16150, Malaysia; McGill Centre for Integrative Neuroscience, Ludmer Centre for Neuroinformatics and Mental Health, Montreal Neurological Institute, Canada
| | - Fuleah Abdul Razzaq
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China
| | - Hazim Omar
- Brain and Behaviour Cluster, School of Medical Sciences, Universiti Sains Malaysia, Health Campus, Kota Bharu, Kelantan 16150, Malaysia; Hospital Universiti Sains Malaysia, Universiti Sains Malaysia, Health Campus, Kota Bharu, Kelantan 16150, Malaysia; McGill Centre for Integrative Neuroscience, Ludmer Centre for Neuroinformatics and Mental Health, Montreal Neurological Institute, Canada
| | - Jafri Malin Abdullah
- Department of Neurosciences, School of Medical Sciences, Universiti Sains Malaysia, Universiti Sains Malaysia Health Campus, Kota Bharu, Kelantan 16150, Malaysia; Hospital Universiti Sains Malaysia, Universiti Sains Malaysia, Health Campus, Kota Bharu, Kelantan 16150, Malaysia; McGill Centre for Integrative Neuroscience, Ludmer Centre for Neuroinformatics and Mental Health, Montreal Neurological Institute, Canada
| | - Janina R Galler
- Division of Pediatric Gastroenterology and Nutrition, Massachusetts General Hospital for Children, Boston, MA, United States Massachusetts General Hospital for Children, Boston, MA, United States
| | - John F Ochoa-Gomez
- Grupo Neuropsicología y Conducta - GRUNECO, Faculty of Medicine, Universidad de Antioquia, Colombia; Grupo de Neurociencias de Antioquia, Universidad de Antioquia, School of Medicine. Medellín, Colombia
| | - Leslie S Prichep
- Research & Development, BrainScope Company, Inc. Bethesda, MD, United States; Department of Psychiatry (Ret.), Brain Research Laboratories, NYU School of Medicine, New York, NY, United States
| | | | - Lilia Morales-Chacon
- Department of Clinical Neurophysiology, International Center for Neurological Restoration, Playa, Havana 11300, Cuba
| | | | - Marius Tröndle
- Department of Methods of Plasticity Research, Institute of Psychology, University of Zurich, Zurich, Switzerland; University Research Priority Program "Dynamic of Healthy Aging", University of Zurich, Zurich, Switzerland; Neuroscience Center Zurich (ZNZ), Zurich, Switzerland
| | - Mohd Faizal Mohd Zulkifly
- Brain and Behaviour Cluster, School of Medical Sciences, Universiti Sains Malaysia, Health Campus, Kota Bharu, Kelantan 16150, Malaysia; Hospital Universiti Sains Malaysia, Universiti Sains Malaysia, Health Campus, Kota Bharu, Kelantan 16150, Malaysia; McGill Centre for Integrative Neuroscience, Ludmer Centre for Neuroinformatics and Mental Health, Montreal Neurological Institute, Canada
| | - Muhammad Riddha Bin Abdul Rahman
- Brain and Behaviour Cluster, School of Medical Sciences, Universiti Sains Malaysia, Health Campus, Kota Bharu, Kelantan 16150, Malaysia; Hospital Universiti Sains Malaysia, Universiti Sains Malaysia, Health Campus, Kota Bharu, Kelantan 16150, Malaysia; School of Medical Imaging, Faculty of Health Sciences, Universiti Sultan Zainal Abidin, Kuala Nerus 21300, Malaysia
| | - Natalya S Milakhina
- Laboratory of Psychophysiology of Individual Differences, Federal State Budgetary Scientific Institution Scientific Research Institute of Neurosciences and Medicine, Novosibirsk 630117, Russia; Laboratory of Psychological Genetics at the Institute of Cytology and Genetics Siberian Branch of the Russian Academy of Sciences, Novosibirsk 630090, Russia
| | - Nicolas Langer
- Department of Methods of Plasticity Research, Institute of Psychology, University of Zurich, Zurich, Switzerland; University Research Priority Program "Dynamic of Healthy Aging", University of Zurich, Zurich, Switzerland; Neuroscience Center Zurich (ZNZ), Zurich, Switzerland
| | - Pavel Rudych
- Laboratory of Psychophysiology of Individual Differences, Federal State Budgetary Scientific Institution Scientific Research Institute of Neurosciences and Medicine, Novosibirsk 630117, Russia; Department of Information Technologies Novosibirsk State University, Novosibirsk 630090, Russia; Federal Research Center for Information and Computational Technologies, Biomedical Data Processing Lab, Novosibirsk 630090, Russia
| | - Thomas Koenig
- University Hospital of Psychiatry and Psychotherapy, University of Bern, Bern, Switzerland
| | | | - Xu Lei
- Sleep and NeuroImaging Center, Faculty of Psychology, Southwest University, Chongqing 400715, China
| | - Maria L Bringas-Vega
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China; Cuban Center for Neurocience, La Habana, Cuba.
| | - Jorge F Bosch-Bayard
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China; Cuban Center for Neurocience, La Habana, Cuba; McGill Centre for Integrative Neuroscience, Ludmer Centre for Neuroinformatics and Mental Health, Montreal Neurological Institute, Canada.
| | - Pedro Antonio Valdes-Sosa
- The Clinical Hospital of Chengdu Brain Science Institute, MOE Key Lab for Neuroinformation, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China; Cuban Center for Neurocience, La Habana, Cuba.
| |
Collapse
|
26
|
Liu Y, Chen Y, Lasang P, Sun Q. Covariance Attention for Semantic Segmentation. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:1805-1818. [PMID: 32976093 DOI: 10.1109/tpami.2020.3026069] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
The dependency between global and local information can provide important contextual cues for semantic segmentation. Existing attention methods capture this dependency by calculating the pixel wise correlation between the learnt feature maps, which is of high space and time complexity. In this article, a new attention module, covariance attention, is presented, and which is interesting in the following aspects: 1) covariance matrix is used as a new attention module to model the global and local dependency for the feature maps and the local-global dependency is formulated as a simple matrix projection process; 2) since covariance matrix can encode the joint distribution information for the heterogeneous yet complementary statistics, the hand-engineered features are combined with the learnt features effectively using covariance matrix to boost the segmentation performance; 3) a covariance attention mechanism based semantic segmentation framework, CANet, is proposed and very competitive performance has been obtained. Comparisons with the state-of-the-art methods reveal the superiority of the proposed method.
Collapse
|
27
|
Minh HQ. Entropic Regularization of Wasserstein Distance Between Infinite-Dimensional Gaussian Measures and Gaussian Processes. J THEOR PROBAB 2022. [DOI: 10.1007/s10959-022-01165-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
|
28
|
Li H, Wang S, Wan R, Kot AC. GMFAD: Towards Generalized Visual Recognition via Multilayer Feature Alignment and Disentanglement. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:1289-1303. [PMID: 32870783 DOI: 10.1109/tpami.2020.3020554] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
The deep learning based approaches which have been repeatedly proven to bring benefits to visual recognition tasks usually make a strong assumption that the training and test data are drawn from similar feature spaces and distributions. However, such an assumption may not always hold in various practical application scenarios on visual recognition tasks. Inspired by the hierarchical organization of deep feature representation that progressively leads to more abstract features at higher layers of representations, we propose to tackle this problem with a novel feature learning framework, which is called GMFAD, with better generalization capability in a multilayer perceptron manner. We first learn feature representations at the shallow layer where shareable underlying factors among domains (e.g., a subset of which could be relevant for each particular domain) can be explored. In particular, we propose to align the domain divergence between domain pair(s) by considering both inter-dimension and inter-sample correlations, which have been largely ignored by many cross-domain visual recognition methods. Subsequently, to learn more abstract information which could further benefit transferability, we propose to conduct feature disentanglement at the deep feature layer. Extensive experiments based on different visual recognition tasks demonstrate that our proposed framework can learn better transferable feature representation compared with state-of-the-art baselines.
Collapse
|
29
|
Koniusz P, Zhang H. Power Normalizations in Fine-Grained Image, Few-Shot Image and Graph Classification. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:591-609. [PMID: 34428137 DOI: 10.1109/tpami.2021.3107164] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Power Normalizations (PN) are useful non-linear operators which tackle feature imbalances in classification problems. We study PNs in the deep learning setup via a novel PN layer pooling feature maps. Our layer combines the feature vectors and their respective spatial locations in the feature maps produced by the last convolutional layer of CNN into a positive definite matrix with second-order statistics to which PN operators are applied, forming so-called Second-order Pooling (SOP). As the main goal of this paper is to study Power Normalizations, we investigate the role and meaning of MaxExp and Gamma, two popular PN functions. To this end, we provide probabilistic interpretations of such element-wise operators and discover surrogates with well-behaved derivatives for end-to-end training. Furthermore, we look at the spectral applicability of MaxExp and Gamma by studying Spectral Power Normalizations (SPN). We show that SPN on the autocorrelation/covariance matrix and the Heat Diffusion Process (HDP) on a graph Laplacian matrix are closely related, thus sharing their properties. Such a finding leads us to the culmination of our work, a fast spectral MaxExp which is a variant of HDP for covariances/autocorrelation matrices. We evaluate our ideas on fine-grained recognition, scene recognition, and material classification, as well as in few-shot learning and graph classification.
Collapse
|
30
|
Fan Z, Zhang H, Zhang Z, Lu G, Zhang Y, Wang Y. A survey of crowd counting and density estimation based on convolutional neural network. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2021.02.103] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
|
31
|
Fang H, Jin J, Daly I, Wang X. Feature Extraction Method Based on Filter Banks and Riemannian Tangent Space in Motor-Imagery BCI. IEEE J Biomed Health Inform 2022; 26:2504-2514. [PMID: 35085095 DOI: 10.1109/jbhi.2022.3146274] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
Optimal feature extraction for multi-category motor imagery brain-computer interfaces (MI-BCIs) is a research hotspot. The common spatial pattern (CSP) algorithm is one of the most widely used methods in MI-BCIs. However, its performance is adversely affected by variance in the operational frequency band and noise interference. Furthermore, the performance of CSP is not satisfactory when addressing multi-category classification problems. In this work, we propose a fusion method combining Filter Banks and Riemannian Tangent Space (FBRTS) in multiple time windows. FBRTS uses multiple filter banks to overcome the problem of variance in the operational frequency band. It also applies the Riemannian method to the covariance matrix extracted by the spatial filter to obtain more robust features in order to overcome the problem of noise interference. In addition, we use a One-Versus-Rest support vector machine (OVR-SVM) model to classify multi-category features. We evaluate our FBRTS method using BCI competition IV dataset 2a and 2b. The experimental results show that the average classification accuracy of our FBRTS method is 77.7% and 86.9% in datasets 2a and 2b respectively. By analyzing the influence of the different numbers of filter banks and time windows on the performance of our FBRTS method, we can identify the optimal number of filter banks and time windows. Additionally, our FBRTS method can obtain more distinctive features than the filter banks common spatial pattern (FBCSP) method in two-dimensional embedding space. These results show that our proposed method can improve the performance of MI-BCIs.
Collapse
|
32
|
Simar C, Petit R, Bozga N, Leroy A, Cebolla AM, Petieau M, Bontempi G, Cheron G. Riemannian classification of single-trial surface EEG and sources during checkerboard and navigational images in humans. PLoS One 2022; 17:e0262417. [PMID: 35030232 PMCID: PMC8759639 DOI: 10.1371/journal.pone.0262417] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/05/2021] [Accepted: 12/23/2021] [Indexed: 11/19/2022] Open
Abstract
OBJECTIVE Different visual stimuli are classically used for triggering visual evoked potentials comprising well-defined components linked to the content of the displayed image. These evoked components result from the average of ongoing EEG signals in which additive and oscillatory mechanisms contribute to the component morphology. The evoked related potentials often resulted from a mixed situation (power variation and phase-locking) making basic and clinical interpretations difficult. Besides, the grand average methodology produced artificial constructs that do not reflect individual peculiarities. This motivated new approaches based on single-trial analysis as recently used in the brain-computer interface field. APPROACH We hypothesize that EEG signals may include specific information about the visual features of the displayed image and that such distinctive traits can be identified by state-of-the-art classification algorithms based on Riemannian geometry. The same classification algorithms are also applied to the dipole sources estimated by sLORETA. MAIN RESULTS AND SIGNIFICANCE We show that our classification pipeline can effectively discriminate between the display of different visual items (Checkerboard versus 3D navigational image) in single EEG trials throughout multiple subjects. The present methodology reaches a single-trial classification accuracy of about 84% and 93% for inter-subject and intra-subject classification respectively using surface EEG. Interestingly, we note that the classification algorithms trained on sLORETA sources estimation fail to generalize among multiple subjects (63%), which may be due to either the average head model used by sLORETA or the subsequent spatial filtering failing to extract discriminative information, but reach an intra-subject classification accuracy of 82%.
Collapse
Affiliation(s)
- Cédric Simar
- Machine Learning Group, Computer Science Department, Faculty of Sciences, Université Libre de Bruxelles (ULB), Brussels, Belgium
| | - Robin Petit
- Machine Learning Group, Computer Science Department, Faculty of Sciences, Université Libre de Bruxelles (ULB), Brussels, Belgium
- Interuniversity Institute of Bioinformatics in Brussels, Université Libre de Bruxelles- Vrije Universiteit Brussel, Brussels, Belgium
| | - Nichita Bozga
- Machine Learning Group, Computer Science Department, Faculty of Sciences, Université Libre de Bruxelles (ULB), Brussels, Belgium
| | - Axelle Leroy
- Laboratory of Neurophysiology and Movement Biomechanics, Neuroscience Institute, Université Libre de Bruxelles (ULB), Brussels, Belgium
| | - Ana-Maria Cebolla
- Laboratory of Neurophysiology and Movement Biomechanics, Neuroscience Institute, Université Libre de Bruxelles (ULB), Brussels, Belgium
| | - Mathieu Petieau
- Laboratory of Neurophysiology and Movement Biomechanics, Neuroscience Institute, Université Libre de Bruxelles (ULB), Brussels, Belgium
| | - Gianluca Bontempi
- Machine Learning Group, Computer Science Department, Faculty of Sciences, Université Libre de Bruxelles (ULB), Brussels, Belgium
| | - Guy Cheron
- Laboratory of Neurophysiology and Movement Biomechanics, Neuroscience Institute, Université Libre de Bruxelles (ULB), Brussels, Belgium
- Laboratory of Electrophysiology, Université de Mons-Hainaut, Mons, Belgium
| |
Collapse
|
33
|
Abstract
AbstractIn motor imagery-based brain-computer interfaces (BCIs), the spatial covariance features of electroencephalography (EEG) signals that lie on Riemannian manifolds are used to enhance the classification performance of motor imagery BCIs. However, the problem of subject-specific bandpass frequency selection frequently arises in Riemannian manifold-based methods. In this study, we propose a multiple Riemannian graph fusion (MRGF) model to optimize the subject-specific frequency band for a Riemannian manifold. After constructing multiple Riemannian graphs corresponding to multiple bandpass frequency bands, graph embedding based on bilinear mapping and graph fusion based on mutual information were applied to simultaneously extract the spatial and spectral features of the EEG signals from Riemannian graphs. Furthermore, with a support vector machine (SVM) classifier performed on learned features, we obtained an efficient algorithm, which achieves higher classification performance on various datasets, such as BCI competition IIa and in-house BCI datasets. The proposed methods can also be used in other classification problems with sample data in the form of covariance matrices.
Collapse
|
34
|
|
35
|
Wang S, Yin Y, Wang D, Lv Z, Wang Y, Jin Y. An interpretable deep neural network for colorectal polyp diagnosis under colonoscopy. Knowl Based Syst 2021. [DOI: 10.1016/j.knosys.2021.107568] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
|
36
|
Xu Y, Huang X, Lan Q. Selective Cross-Subject Transfer Learning Based on Riemannian Tangent Space for Motor Imagery Brain-Computer Interface. Front Neurosci 2021; 15:779231. [PMID: 34803600 PMCID: PMC8595943 DOI: 10.3389/fnins.2021.779231] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2021] [Accepted: 10/14/2021] [Indexed: 11/13/2022] Open
Abstract
A motor imagery (MI) brain-computer interface (BCI) plays an important role in the neurological rehabilitation training for stroke patients. Electroencephalogram (EEG)-based MI BCI has high temporal resolution, which is convenient for real-time BCI control. Therefore, we focus on EEG-based MI BCI in this paper. The identification of MI EEG signals is always quite challenging. Due to high inter-session/subject variability, each subject should spend long and tedious calibration time in collecting amounts of labeled samples for a subject-specific model. To cope with this problem, we present a supervised selective cross-subject transfer learning (sSCSTL) approach which simultaneously makes use of the labeled samples from target and source subjects based on Riemannian tangent space. Since the covariance matrices representing the multi-channel EEG signals belong to the smooth Riemannian manifold, we perform the Riemannian alignment to make the covariance matrices from different subjects close to each other. Then, all aligned covariance matrices are converted into the Riemannian tangent space features to train a classifier in the Euclidean space. To investigate the role of unlabeled samples, we further propose semi-supervised and unsupervised versions which utilize the total samples and unlabeled samples from target subject, respectively. Sequential forward floating search (SFFS) method is executed for source selection. All our proposed algorithms transfer the labeled samples from most suitable source subjects into the feature space of target subject. Experimental results on two publicly available MI datasets demonstrated that our algorithms outperformed several state-of-the-art algorithms using small number of the labeled samples from target subject, especially for good target subjects.
Collapse
Affiliation(s)
- Yilu Xu
- School of Software, Jiangxi Agricultural University, Nanchang, China
| | - Xin Huang
- Software College, Jiangxi Normal University, Nanchang, China
| | - Quan Lan
- Department of Neurology, First Affiliated Hospital of Xiamen University, Xiamen, China
| |
Collapse
|
37
|
Tayanov V, Krzyżak A, Suen CY. Ensemble Learning Using Matrices of Classifier Interactions and Decision Profiles on Riemannian and Grassmann Manifolds. INT J PATTERN RECOGN 2021. [DOI: 10.1142/s0218001421600119] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
This paper introduces a new topic and research of geometric classifier ensemble learning using two types of objects: classifier prediction pairwise matrix (CPPM) and decision profiles (DPs). Learning from CPPM requires using Riemannian manifolds (R-manifolds) of symmetric positive definite (SPD) matrices. DPs can be used to build a Grassmann manifold (G-manifold). Experimental results show that classifier ensembles and their cascades built using R-manifolds are less dependent on some properties of individual classifiers (e.g. depth of decision trees in random forests (RFs) or extra trees (ETs)) in comparison to G-manifolds and Euclidean geometry. More independent individual classifiers allow obtaining R-manifolds with better properties for classification. Generally, the accuracy of classification in nonlinear geometry is higher than in Euclidean one. For multi-class problems, G-manifolds perform similarly to stacking-based classifiers built on R-manifolds of SPD matrices in terms of classification accuracy.
Collapse
Affiliation(s)
- Vitaliy Tayanov
- Department of Computer Science and Software Engineering, Concordia University, 1455 De Maisonneuve Blvd. West, Montreal, Quebec H3G 1M8, Canada
| | - Adam Krzyżak
- Department of Computer Science and Software Engineering, Concordia University, 1455 De Maisonneuve Blvd. West, Montreal, Quebec H3G 1M8, Canada
| | - Ching Y. Suen
- Department of Computer Science and Software Engineering, Concordia University, 1455 De Maisonneuve Blvd. West, Montreal, Quebec H3G 1M8, Canada
| |
Collapse
|
38
|
Xu S, Qi M, Gao F. A Semi-supervised Riemannian Kernel Dictionary Learning Algorithm Based on Locality-Constrained for Image Classification. ARABIAN JOURNAL FOR SCIENCE AND ENGINEERING 2021. [DOI: 10.1007/s13369-021-06129-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
39
|
Xia Y, He Y, Peng S, Hao X, Yang Q, Yin B. EDENet: Elaborate density estimation network for crowd counting. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2021.06.086] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
40
|
Roy D, Fernando B. Action Anticipation Using Pairwise Human-Object Interactions and Transformers. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2021; 30:8116-8129. [PMID: 34550884 DOI: 10.1109/tip.2021.3113114] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
The ability to anticipate future actions of humans is useful in application areas such as automated driving, robot-assisted manufacturing, and smart homes. These applications require representing and anticipating human actions involving the use of objects. Existing methods that use human-object interactions for anticipation require object affordance labels for every relevant object in the scene that match the ongoing action. Hence, we propose to represent every pairwise human-object (HO) interaction using only their visual features. Next, we use cross-correlation to capture the second-order statistics across human-object pairs in a frame. Cross-correlation produces a holistic representation of the frame that can also handle a variable number of human-object pairs in every frame of the observation period. We show that cross-correlation based frame representation is more suited for action anticipation than attention-based and other second-order approaches. Furthermore, we observe that using a transformer model for temporal aggregation of frame-wise HO representations results in better action anticipation than other temporal networks. So, we propose two approaches for constructing an end-to-end trainable multi-modal transformer (MM-Transformer; code at https://github.com/debadityaroy/MM-Transformer_ActAnt) model that combines the evidence across spatio-temporal, motion, and HO representations. We show the performance of MM-Transformer on procedural datasets like 50 Salads and Breakfast, and an unscripted dataset like EPIC-KITCHENS55. Finally, we demonstrate that the combination of human-object representation and MM-Transformers is effective even for long-term anticipation.
Collapse
|
41
|
Gao W, Ma Z, Gan W, Liu S. Dimensionality Reduction of SPD Data Based on Riemannian Manifold Tangent Spaces and Isometry. ENTROPY 2021; 23:e23091117. [PMID: 34573742 PMCID: PMC8471569 DOI: 10.3390/e23091117] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/07/2021] [Revised: 08/19/2021] [Accepted: 08/23/2021] [Indexed: 11/16/2022]
Abstract
Symmetric positive definite (SPD) data have become a hot topic in machine learning. Instead of a linear Euclidean space, SPD data generally lie on a nonlinear Riemannian manifold. To get over the problems caused by the high data dimensionality, dimensionality reduction (DR) is a key subject for SPD data, where bilinear transformation plays a vital role. Because linear operations are not supported in nonlinear spaces such as Riemannian manifolds, directly performing Euclidean DR methods on SPD matrices is inadequate and difficult in complex models and optimization. An SPD data DR method based on Riemannian manifold tangent spaces and global isometry (RMTSISOM-SPDDR) is proposed in this research. The main contributions are listed: (1) Any Riemannian manifold tangent space is a Hilbert space isomorphic to a Euclidean space. Particularly for SPD manifolds, tangent spaces consist of symmetric matrices, which can greatly preserve the form and attributes of original SPD data. For this reason, RMTSISOM-SPDDR transfers the bilinear transformation from manifolds to tangent spaces. (2) By log transformation, original SPD data are mapped to the tangent space at the identity matrix under the affine invariant Riemannian metric (AIRM). In this way, the geodesic distance between original data and the identity matrix is equal to the Euclidean distance between corresponding tangent vector and the origin. (3) The bilinear transformation is further determined by the isometric criterion guaranteeing the geodesic distance on high-dimensional SPD manifold as close as possible to the Euclidean distance in the tangent space of low-dimensional SPD manifold. Then, we use it for the DR of original SPD data. Experiments on five commonly used datasets show that RMTSISOM-SPDDR is superior to five advanced SPD data DR algorithms.
Collapse
Affiliation(s)
- Wenxu Gao
- School of Electronics and Information Technology, Sun Yat-Sen University, Guangzhou 510006, China; (W.G.); (W.G.)
| | - Zhengming Ma
- School of Electronics and Information Technology, Sun Yat-Sen University, Guangzhou 510006, China; (W.G.); (W.G.)
- Correspondence:
| | - Weichao Gan
- School of Electronics and Information Technology, Sun Yat-Sen University, Guangzhou 510006, China; (W.G.); (W.G.)
| | - Shuyu Liu
- Public Experimental Teaching Center, Sun Yat-sen University, Guangzhou 510006, China;
| |
Collapse
|
42
|
Peng S, Yin B, Hao X, Yang Q, Kumar A, Wang L. Depth and edge auxiliary learning for still image crowd density estimation. Pattern Anal Appl 2021. [DOI: 10.1007/s10044-021-01017-4] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
43
|
Latent Network Construction for Univariate Time Series Based on Variational Auto-Encode. ENTROPY 2021; 23:e23081071. [PMID: 34441211 PMCID: PMC8394240 DOI: 10.3390/e23081071] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/29/2021] [Revised: 08/12/2021] [Accepted: 08/16/2021] [Indexed: 11/27/2022]
Abstract
Time series analysis has been an important branch of information processing, and the conversion of time series into complex networks provides a new means to understand and analyze time series. In this work, using Variational Auto-Encode (VAE), we explored the construction of latent networks for univariate time series. We first trained the VAE to obtain the space of latent probability distributions of the time series and then decomposed the multivariate Gaussian distribution into multiple univariate Gaussian distributions. By measuring the distance between univariate Gaussian distributions on a statistical manifold, the latent network construction was finally achieved. The experimental results show that the latent network can effectively retain the original information of the time series and provide a new data structure for the downstream tasks.
Collapse
|
44
|
Shariat A, Zarei A, Karvigh SA, Asl BM. Automatic detection of epileptic seizures using Riemannian geometry from scalp EEG recordings. Med Biol Eng Comput 2021; 59:1431-1445. [PMID: 34128177 DOI: 10.1007/s11517-021-02385-z] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2020] [Accepted: 05/15/2021] [Indexed: 11/30/2022]
Abstract
This paper proposes a new framework for epileptic seizure detection using non-invasive scalp electroencephalogram (sEEG) signals. The major innovation of the current study is using the Riemannian geometry for transforming the covariance matrices estimated from the EEG channels into a feature vector. The spatial covariance matrices are considered as features in order to extract the spatial information of the sEEG signals without applying any spatial filtering. Since these matrices are symmetric and positive definite (SPD), they belong to a special manifold called the Riemannian manifold. Furthermore, a kernel based on Riemannian geometry is proposed. This kernel maps the SPD matrices onto the Riemannian tangent space. The SPD matrices, obtained from all channels of the segmented sEEG signals, have high dimensions and extra information. For these reasons, the sequential forward feature selection method is applied to select the best features and reduce the computational burden in the classification step. The selected features are fed into a support vector machine (SVM) with an RBF kernel to classify the feature vectors into seizure and non-seizure classes. The performance of the proposed method is evaluated using two long-term scalp EEG (CHB-MIT benchmark and private) databases. Experimental results on all 23 subjects of the CHB-MIT database reveal an accuracy of 99.87%, a sensitivity of 99.91%, and a specificity of 99.82%. In addition, the introduced algorithm is tested on the private sEEG signals recorded from 20 patients, having 1380 seizures. The proposed approach achieves an accuracy, a sensitivity, and a specificity of 98.14%, 98.16%, and 98.12%, respectively. The experimental results on both sEEG databases demonstrate the effectiveness of the proposed method for automated epileptic seizure detection, especially for the private database which has noisier signals in comparison to the CHB-MIT database. Graphical Abstract Block diagram of the proposed epileptic seizure detection algorithm.
Collapse
Affiliation(s)
- Atefeh Shariat
- Department of Biomedical Engineering, Tarbiat Modares University, Tehran, Iran
| | - Asghar Zarei
- Department of Biomedical Engineering, Tarbiat Modares University, Tehran, Iran
| | - Sanaz Ahmadi Karvigh
- Department of Neurology, Sina Hospital, Tehran University of Medical Sciences, Tehran, Iran
| | | |
Collapse
|
45
|
Kumar S, Tsunoda T, Sharma A. SPECTRA: a tool for enhanced brain wave signal recognition. BMC Bioinformatics 2021; 22:195. [PMID: 34078274 PMCID: PMC8170968 DOI: 10.1186/s12859-021-04091-x] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2021] [Accepted: 03/21/2021] [Indexed: 12/31/2022] Open
Abstract
Background Brain wave signal recognition has gained increased attention in neuro-rehabilitation applications. This has driven the development of brain–computer interface (BCI) systems. Brain wave signals are acquired using electroencephalography (EEG) sensors, processed and decoded to identify the category to which the signal belongs. Once the signal category is determined, it can be used to control external devices. However, the success of such a system essentially relies on significant feature extraction and classification algorithms. One of the commonly used feature extraction technique for BCI systems is common spatial pattern (CSP). Results The performance of the proposed spatial-frequency-temporal feature extraction (SPECTRA) predictor is analysed using three public benchmark datasets. Our proposed predictor outperformed other competing methods achieving lowest average error rates of 8.55%, 17.90% and 20.26%, and highest average kappa coefficient values of 0.829, 0.643 and 0.595 for BCI Competition III dataset IVa, BCI Competition IV dataset I and BCI Competition IV dataset IIb, respectively.
Conclusions Our proposed SPECTRA predictor effectively finds features that are more separable and shows improvement in brain wave signal recognition that can be instrumental in developing improved real-time BCI systems that are computationally efficient.
Collapse
Affiliation(s)
- Shiu Kumar
- School of Electrical and Electronics Engineering, Fiji National University, Suva, Fiji.
| | - Tatsuhiko Tsunoda
- Laboratory for Medical Science Mathematics, RIKEN Center for Integrative Medical Sciences, Yokohama, 230-0045, Japan.,Department of Medical Science Mathematics, Medical Research Institute, Tokyo Medical and Dental University (TMDU), Tokyo, 113-8510, Japan.,Laboratory for Medical Science Mathematics, Department of Biological Sciences, Graduate School of Science, University of Tokyo, Tokyo, 113-0033, Japan
| | - Alok Sharma
- Laboratory for Medical Science Mathematics, RIKEN Center for Integrative Medical Sciences, Yokohama, 230-0045, Japan.,Department of Medical Science Mathematics, Medical Research Institute, Tokyo Medical and Dental University (TMDU), Tokyo, 113-8510, Japan.,School of Engineering and Physics, The University of the South Pacific, Suva, Fiji.,Institute for Integrated and Intelligent Systems, Griffith University, Nathan, Brisbane, QLD, Australia
| |
Collapse
|
46
|
Tang F, Feng H, Tino P, Si B, Ji D. Probabilistic learning vector quantization on manifold of symmetric positive definite matrices. Neural Netw 2021; 142:105-118. [PMID: 33984734 DOI: 10.1016/j.neunet.2021.04.024] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2020] [Revised: 04/07/2021] [Accepted: 04/20/2021] [Indexed: 10/21/2022]
Abstract
In this paper, we develop a new classification method for manifold-valued data in the framework of probabilistic learning vector quantization. In many classification scenarios, the data can be naturally represented by symmetric positive definite matrices, which are inherently points that live on a curved Riemannian manifold. Due to the non-Euclidean geometry of Riemannian manifolds, traditional Euclidean machine learning algorithms yield poor results on such data. In this paper, we generalize the probabilistic learning vector quantization algorithm for data points living on the manifold of symmetric positive definite matrices equipped with Riemannian natural metric (affine-invariant metric). By exploiting the induced Riemannian distance, we derive the probabilistic learning Riemannian space quantization algorithm, obtaining the learning rule through Riemannian gradient descent. Empirical investigations on synthetic data, image data , and motor imagery electroencephalogram (EEG) data demonstrate the superior performance of the proposed method.
Collapse
Affiliation(s)
- Fengzhen Tang
- State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, 110016, China; Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang, 110169, China.
| | - Haifeng Feng
- State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, 110016, China; Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Shenyang, 110169, China; University of Chinese Academy of Sciences, Beijing, 100049, China.
| | - Peter Tino
- School of computer Science, University of Birmingham, Birmingham, B15 2TT, UK.
| | - Bailu Si
- School of Systems Science, Beijing Normal University, Beijing, 100875, China.
| | - Daxiong Ji
- Institute of Marine Electronics and Intelligent Systems, Ocean College, Zhejiang University, The Key Laboratory of Ocean Observation-Imaging Testbed of Zhejiang Province, The Engineering Research Center of Oceanic Sensing Technology and Equipment, Ministry of Education, Zhoushan, 316021, China.
| |
Collapse
|
47
|
Wu F, Gong A, Li H, Zhao L, Zhang W, Fu Y. A New Subject- Specific Discriminative and Multi- Scale Filter Bank Tangent Space Mapping Method for Recognition of Multiclass Motor Imagery. Front Hum Neurosci 2021; 15:595723. [PMID: 33762911 PMCID: PMC7982728 DOI: 10.3389/fnhum.2021.595723] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2020] [Accepted: 02/15/2021] [Indexed: 11/13/2022] Open
Abstract
Objective: Tangent Space Mapping (TSM) using the geometric structure of the covariance matrices is an effective method to recognize multiclass motor imagery (MI). Compared with the traditional CSP method, the Riemann geometric method based on TSM takes into account the nonlinear information contained in the covariance matrix, and can extract more abundant and effective features. Moreover, the method is an unsupervised operation, which can reduce the time of feature extraction. However, EEG features induced by MI mental activities of different subjects are not the same, so selection of subject-specific discriminative EEG frequency components play a vital role in the recognition of multiclass MI. In order to solve the problem, a discriminative and multi-scale filter bank tangent space mapping (DMFBTSM) algorithm is proposed in this article to design the subject-specific Filter Bank (FB) so as to effectively recognize multiclass MI tasks. Methods: On the 4-class BCI competition IV-2a dataset, first, a non-parametric method of multivariate analysis of variance (MANOVA) based on the sum of squared distances is used to select discriminative frequency bands for a subject; next, a multi-scale FB is generated according to the range of these frequency bands, and then decompose multi-channel EEG of the subject into multiple sub-bands combined with several time windows. Then TSM algorithm is used to estimate Riemannian tangent space features in each sub-band and finally a liner Support Vector Machines (SVM) is used for classification. Main Results: The analysis results show that the proposed discriminative FB enhances the multi-scale TSM algorithm, improves the classification accuracy and reduces the execution time during training and testing. On the 4-class BCI competition IV-2a dataset, the average session to session classification accuracy of nine subjects reached 77.33 ± 12.3%. When the training time and the test time are similar, the average classification accuracy is 2.56% higher than the latest TSM method based on multi-scale filter bank analysis technology. When the classification accuracy is similar, the training speed is increased by more than three times, and the test speed is increased two times more. Compared with Supervised Fisher Geodesic Minimum Distance to the Mean (Supervised FGMDRM), another new variant based on Riemann geometry classifier, the average accuracy is 3.36% higher, we also compared with the latest Deep Learning method, and the average accuracy of 10-fold cross validation improved by 2.58%. Conclusion: Research shows that the proposed DMFBTSM algorithm can improve the classification accuracy of MI tasks. Significance: Compared with the MFBTSM algorithm, the algorithm proposed in this article is expected to select frequency bands with good separability for specific subject to improve the classification accuracy of multiclass MI tasks and reduce the feature dimension to reduce training time and testing time.
Collapse
Affiliation(s)
- Fan Wu
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China.,Brain Cognition and Brain-Computer Intelligence Fusion Innovation Group, Kunming University of Science and Technology, Kunming, China
| | - Anmin Gong
- Department of Information Engineering, Engineering University of PAP, Xi'an, China
| | - Hongyun Li
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China.,Brain Cognition and Brain-Computer Intelligence Fusion Innovation Group, Kunming University of Science and Technology, Kunming, China
| | - Lei Zhao
- Brain Cognition and Brain-Computer Intelligence Fusion Innovation Group, Kunming University of Science and Technology, Kunming, China.,College of Science, Kunming University of Science and Technology, Kunming, China
| | - Wei Zhang
- Kunming Medical University, Kunming, China
| | - Yunfa Fu
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, China.,Brain Cognition and Brain-Computer Intelligence Fusion Innovation Group, Kunming University of Science and Technology, Kunming, China.,Yunnan Provincial Key Laboratory of Computer Technology Application, Kunming, China.,School of Medicine, Kunming University of Science and Technology, Kunming, China
| |
Collapse
|
48
|
Sharma K, Rameshan R. Image Set Classification Using a Distance-Based Kernel Over Affine Grassmann Manifold. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2021; 32:1082-1095. [PMID: 32275625 DOI: 10.1109/tnnls.2020.2980059] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Modeling image sets or videos as linear subspaces is quite popular for classification problems in machine learning. However, affine subspace modeling has not been explored much. In this article, we address the image sets classification problem by modeling them as affine subspaces. Affine subspaces are linear subspaces shifted from origin by an offset. The collection of the same dimensional affine subspaces of [Formula: see text] is known as affine Grassmann manifold (AGM) or affine Grassmannian that is a smooth and noncompact manifold. The non-Euclidean geometry of AGM and the nonunique representation of an affine subspace in AGM make the classification task in AGM difficult. In this article, we propose a novel affine subspace-based kernel that maps the points in AGM to a finite-dimensional Hilbert space. For this, we embed the AGM in a higher dimensional Grassmann manifold (GM) by embedding the offset vector in the Stiefel coordinates. The projection distance between two points in AGM is the measure of similarity obtained by the kernel function. The obtained kernel-gram matrix is further diagonalized to generate low-dimensional features in the Euclidean space corresponding to the points in AGM. Distance-preserving constraint along with sparsity constraint is used for minimum residual error classification by keeping the locally Euclidean structure of AGM in mind. Experimentation performed over four data sets for gait, object, hand, and body gesture recognition shows promising results compared with state-of-the-art techniques.
Collapse
|
49
|
|
50
|
Cheng Z, Chen C, Chen Z, Fang K, Jin X. Robust and high-order correlation alignment for unsupervised domain adaptation. Neural Comput Appl 2021. [DOI: 10.1007/s00521-020-05465-7] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|