1
|
Mark JA, Curtin A, Kraft AE, Ziegler MD, Ayaz H. Mental workload assessment by monitoring brain, heart, and eye with six biomedical modalities during six cognitive tasks. FRONTIERS IN NEUROERGONOMICS 2024; 5:1345507. [PMID: 38533517 PMCID: PMC10963413 DOI: 10.3389/fnrgo.2024.1345507] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/27/2023] [Accepted: 02/15/2024] [Indexed: 03/28/2024]
Abstract
Introduction The efficiency and safety of complex high precision human-machine systems such as in aerospace and robotic surgery are closely related to the cognitive readiness, ability to manage workload, and situational awareness of their operators. Accurate assessment of mental workload could help in preventing operator error and allow for pertinent intervention by predicting performance declines that can arise from either work overload or under stimulation. Neuroergonomic approaches based on measures of human body and brain activity collectively can provide sensitive and reliable assessment of human mental workload in complex training and work environments. Methods In this study, we developed a new six-cognitive-domain task protocol, coupling it with six biomedical monitoring modalities to concurrently capture performance and cognitive workload correlates across a longitudinal multi-day investigation. Utilizing two distinct modalities for each aspect of cardiac activity (ECG and PPG), ocular activity (EOG and eye-tracking), and brain activity (EEG and fNIRS), 23 participants engaged in four sessions over 4 weeks, performing tasks associated with working memory, vigilance, risk assessment, shifting attention, situation awareness, and inhibitory control. Results The results revealed varying levels of sensitivity to workload within each modality. While certain measures exhibited consistency across tasks, neuroimaging modalities, in particular, unveiled meaningful differences between task conditions and cognitive domains. Discussion This is the first comprehensive comparison of these six brain-body measures across multiple days and cognitive domains. The findings underscore the potential of wearable brain and body sensing methods for evaluating mental workload. Such comprehensive neuroergonomic assessment can inform development of next generation neuroadaptive interfaces and training approaches for more efficient human-machine interaction and operator skill acquisition.
Collapse
Affiliation(s)
- Jesse A. Mark
- School of Biomedical Engineering, Science, and Health Systems, Drexel University, Philadelphia, PA, United States
| | - Adrian Curtin
- School of Biomedical Engineering, Science, and Health Systems, Drexel University, Philadelphia, PA, United States
| | - Amanda E. Kraft
- Advanced Technology Laboratories, Lockheed Martin, Arlington, VA, United States
| | - Matthias D. Ziegler
- Advanced Technology Laboratories, Lockheed Martin, Arlington, VA, United States
| | - Hasan Ayaz
- School of Biomedical Engineering, Science, and Health Systems, Drexel University, Philadelphia, PA, United States
- Department of Psychological and Brain Sciences, College of Arts and Sciences, Drexel University, Philadelphia, PA, United States
- Drexel Solutions Institute, Drexel University, Philadelphia, PA, United States
- A. J. Drexel Autism Institute, Drexel University, Philadelphia, PA, United States
- Department of Family and Community Health, University of Pennsylvania, Philadelphia, PA, United States
- Center for Injury Research and Prevention, Children's Hospital of Philadelphia, Philadelphia, PA, United States
| |
Collapse
|
2
|
Clemente L, La Rocca M, Quaranta N, Iannuzzi L, Vecchio E, Brunetti A, Gentile E, Dibattista M, Lobasso S, Bevilacqua V, Stramaglia S, de Tommaso M. Prefrontal dysfunction in post-COVID-19 hyposmia: an EEG/fNIRS study. Front Hum Neurosci 2023; 17:1240831. [PMID: 37829821 PMCID: PMC10564993 DOI: 10.3389/fnhum.2023.1240831] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/27/2023] [Accepted: 08/29/2023] [Indexed: 10/14/2023] Open
Abstract
Introduction Subtle cognitive dysfunction and mental fatigue are frequent after severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, characterizing the so-called long COVID-19 syndrome. This study aimed to correlate cognitive, neurophysiological, and olfactory function in a group of subjects who experienced acute SARS-CoV-2 infection with persistent hyposmia at least 12 weeks before the observation. Methods For each participant (32 post-COVID-19 patients and 16 controls), electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) data were acquired using an integrated EEG-fNIRS system during the execution of a P300 odd-ball task and a Stroop test. The Sniffin' Sticks test was conducted to assess subjects' olfactory performance. The Montreal Cognitive Assessment (MoCA) and the Frontal Assessment Battery (FAB) were also administered. Results The post-COVID-19 group consisted of 32 individuals (20 women and 12 men) with an average education level of 12.9 ± 3.12 years, while the control group consisted of 16 individuals (10 women and 6 men) with an average education level of 14.9 ± 3.2 years. There were no significant differences in gender (X2 = 0, p = 1) or age between the two groups (age 44.81 ± 13.9 vs. 36.62 ± 11.4, p = 0.058). We identified a lower concentration of oxyhemoglobin (p < 0.05) at the prefrontal cortical level in post-COVID-19 subjects during the execution of the Stroop task, as well as a reduction in the amplitude of the P3a response. Moreover, we found that post-COVID-19 subjects performed worst at the MoCA screening test (p = 0.001), Sniffin's Sticks test (p < 0.001), and Stroop task response latency test (p < 0.001). Conclusions This study showed that post-COVID-19 patients with persistent hyposmia present mild deficits in prefrontal function, even 4 months after the end of the infection. These deficits, although subtle, could have long-term implications for quality of life and cognitive wellbeing. It is essential to continue monitoring and evaluating these patients to better understand the extent and duration of cognitive impairments associated with long COVID-19.
Collapse
Affiliation(s)
- Livio Clemente
- Department of Translational Biomedicine and Neuroscience (DiBraiN), University of Bari Aldo Moro, Bari, Italy
| | - Marianna La Rocca
- M. Merlin Inter-university Physics Department, University of Bari, Bari, Italy
- Laboratory of Neuroimaging, Keck School of Medicine of USC, USC Stevens Neuroimaging and Informatics Institute, University of Southern California, Los Angeles, CA, United States
| | - Nicola Quaranta
- Department of Translational Biomedicine and Neuroscience (DiBraiN), University of Bari Aldo Moro, Bari, Italy
| | - Lucia Iannuzzi
- Department of Translational Biomedicine and Neuroscience (DiBraiN), University of Bari Aldo Moro, Bari, Italy
| | - Eleonora Vecchio
- Department of Translational Biomedicine and Neuroscience (DiBraiN), University of Bari Aldo Moro, Bari, Italy
| | - Antonio Brunetti
- Department of Electrical and Information Engineering, Polytechnic University of Bari, Bari, Italy
| | - Eleonora Gentile
- Department of Translational Biomedicine and Neuroscience (DiBraiN), University of Bari Aldo Moro, Bari, Italy
| | - Michele Dibattista
- Department of Translational Biomedicine and Neuroscience (DiBraiN), University of Bari Aldo Moro, Bari, Italy
| | - Simona Lobasso
- Department of Translational Biomedicine and Neuroscience (DiBraiN), University of Bari Aldo Moro, Bari, Italy
| | - Vitoantonio Bevilacqua
- Department of Electrical and Information Engineering, Polytechnic University of Bari, Bari, Italy
| | | | - Marina de Tommaso
- Department of Translational Biomedicine and Neuroscience (DiBraiN), University of Bari Aldo Moro, Bari, Italy
| |
Collapse
|
3
|
Betts K, Reddy P, Galoyan T, Delaney B, McEachron DL, Izzetoglu K, Shewokis PA. An Examination of the Effects of Virtual Reality Training on Spatial Visualization and Transfer of Learning. Brain Sci 2023; 13:890. [PMID: 37371368 DOI: 10.3390/brainsci13060890] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2023] [Accepted: 05/23/2023] [Indexed: 06/29/2023] Open
Abstract
Spatial visualization ability (SVA) has been identified as a potential key factor for academic achievement and student retention in Science, Technology, Engineering, and Mathematics (STEM) in higher education, especially for engineering and related disciplines. Prior studies have shown that training using virtual reality (VR) has the potential to enhance learning through the use of more realistic and/or immersive experiences. The aim of this study was to investigate the effect of VR-based training using spatial visualization tasks on participant performance and mental workload using behavioral (i.e., time spent) and functional near infrared spectroscopy (fNIRS) brain-imaging-technology-derived measures. Data were collected from 10 first-year biomedical engineering students, who engaged with a custom-designed spatial visualization gaming application over a six-week training protocol consisting of tasks and procedures that varied in task load and spatial characteristics. Findings revealed significant small (Cohen's d: 0.10) to large (Cohen's d: 2.40) effects of task load and changes in the spatial characteristics of the task, such as orientation or position changes, on time spent and oxygenated hemoglobin (HbO) measures from all the prefrontal cortex (PFC) areas. Transfer had a large (d = 1.37) significant effect on time spent and HbO measures from right anterior medial PFC (AMPFC); while training had a moderate (d = 0.48) significant effect on time spent and HbR measures from left AMPFC. The findings from this study have important implications for VR training, research, and instructional design focusing on enhancing the learning, retention, and transfer of spatial skills within and across various VR-based training scenarios.
Collapse
Affiliation(s)
- Kristen Betts
- School of Education, Drexel University, Philadelphia, PA 19104, USA
| | - Pratusha Reddy
- School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA 19104, USA
| | - Tamara Galoyan
- School of Education, Drexel University, Philadelphia, PA 19104, USA
| | - Brian Delaney
- School of Communication and Journalism, Auburn University, Auburn, AL 36849, USA
| | - Donald L McEachron
- School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA 19104, USA
| | - Kurtulus Izzetoglu
- School of Education, Drexel University, Philadelphia, PA 19104, USA
- School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA 19104, USA
| | - Patricia A Shewokis
- School of Education, Drexel University, Philadelphia, PA 19104, USA
- School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA 19104, USA
- College of Nursing & Health Professions, Drexel University, Philadelphia, PA 19104, USA
| |
Collapse
|
4
|
Samima S, Sarma M. Mental workload level assessment based on compounded hysteresis effect. Cogn Neurodyn 2023; 17:357-372. [PMID: 37007201 PMCID: PMC10050634 DOI: 10.1007/s11571-022-09830-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2021] [Revised: 05/02/2022] [Accepted: 05/27/2022] [Indexed: 11/27/2022] Open
Abstract
In the domain of neuroergonomics, cognitive workload estimation has taken a significant concern among the researchers. This is because the knowledge gathered from its estimation is useful for distributing tasks among the operators, understanding human capability and intervening operators at times of havoc. Brain signals give a promising prospective for understanding cognitive workload. For this, electroencephalography (EEG) is by far the most efficient modality in interpreting the covert information arising in the brain. The present work explores the feasibility of EEG rhythms for monitoring continuous change occurring in a person's cognitive workload. This continuous monitoring is achieved by graphicallyinterpreting the cumulative effect of changes in EEG rhythms observed in the current instance and the former instance based on the hysteresis effect. In this work, classification is done to predict the data class label using an artificial neural network (ANN) architecture. The proposed model gives a classification accuracy of 98.66%.
Collapse
Affiliation(s)
- Shabnam Samima
- Subir Chowdhury School of Quality and Reliability, Indian Institute of Technology Kharagpur, Kharagpur, West Bengal India
| | - Monalisa Sarma
- Subir Chowdhury School of Quality and Reliability, Indian Institute of Technology Kharagpur, Kharagpur, West Bengal India
| |
Collapse
|
5
|
Cao J, Garro EM, Zhao Y. EEG/fNIRS Based Workload Classification Using Functional Brain Connectivity and Machine Learning. SENSORS (BASEL, SWITZERLAND) 2022; 22:s22197623. [PMID: 36236725 PMCID: PMC9571712 DOI: 10.3390/s22197623] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/31/2022] [Revised: 10/03/2022] [Accepted: 10/06/2022] [Indexed: 05/07/2023]
Abstract
There is high demand for techniques to estimate human mental workload during some activities for productivity enhancement or accident prevention. Most studies focus on a single physiological sensing modality and use univariate methods to analyse multi-channel electroencephalography (EEG) data. This paper proposes a new framework that relies on the features of hybrid EEG-functional near-infrared spectroscopy (EEG-fNIRS), supported by machine-learning features to deal with multi-level mental workload classification. Furthermore, instead of the well-used univariate power spectral density (PSD) for EEG recording, we propose using bivariate functional brain connectivity (FBC) features in the time and frequency domains of three bands: delta (0.5-4 Hz), theta (4-7 Hz) and alpha (8-15 Hz). With the assistance of the fNIRS oxyhemoglobin and deoxyhemoglobin (HbO and HbR) indicators, the FBC technique significantly improved classification performance at a 77% accuracy for 0-back vs. 2-back and 83% for 0-back vs. 3-back using a public dataset. Moreover, topographic and heat-map visualisation indicated that the distinguishing regions for EEG and fNIRS showed a difference among the 0-back, 2-back and 3-back test results. It was determined that the best region to assist the discrimination of the mental workload for EEG and fNIRS is different. Specifically, the posterior area performed the best for the posterior midline occipital (POz) EEG in the alpha band and fNIRS had superiority in the right frontal region (AF8).
Collapse
|
6
|
Aygun A, Nguyen T, Haga Z, Aeron S, Scheutz M. Investigating Methods for Cognitive Workload Estimation for Assistive Robots. SENSORS (BASEL, SWITZERLAND) 2022; 22:6834. [PMID: 36146189 PMCID: PMC9505485 DOI: 10.3390/s22186834] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/28/2022] [Revised: 08/29/2022] [Accepted: 09/05/2022] [Indexed: 06/16/2023]
Abstract
Robots interacting with humans in assistive contexts have to be sensitive to human cognitive states to be able to provide help when it is needed and not overburden the human when the human is busy. Yet, it is currently still unclear which sensing modality might allow robots to derive the best evidence of human workload. In this work, we analyzed and modeled data from a multi-modal simulated driving study specifically designed to evaluate different levels of cognitive workload induced by various secondary tasks such as dialogue interactions and braking events in addition to the primary driving task. Specifically, we performed statistical analyses of various physiological signals including eye gaze, electroencephalography, and arterial blood pressure from the healthy volunteers and utilized several machine learning methodologies including k-nearest neighbor, naive Bayes, random forest, support-vector machines, and neural network-based models to infer human cognitive workload levels. Our analyses provide evidence for eye gaze being the best physiological indicator of human cognitive workload, even when multiple signals are combined. Specifically, the highest accuracy (in %) of binary workload classification based on eye gaze signals is 80.45 ∓ 3.15 achieved by using support-vector machines, while the highest accuracy combining eye gaze and electroencephalography is only 77.08 ∓ 3.22 achieved by a neural network-based model. Our findings are important for future efforts of real-time workload estimation in the multimodal human-robot interactive systems given that eye gaze is easy to collect and process and less susceptible to noise artifacts compared to other physiological signal modalities.
Collapse
Affiliation(s)
- Ayca Aygun
- Department of Computer Science, Tufts University, Medford, MA 02155, USA
| | - Thuan Nguyen
- Department of Computer Science, Tufts University, Medford, MA 02155, USA
| | - Zachary Haga
- Department of Computer Science, Tufts University, Medford, MA 02155, USA
| | - Shuchin Aeron
- Department of Electrical and Computer Engineering, Tufts University, Medford, MA 02155, USA
| | - Matthias Scheutz
- Department of Computer Science, Tufts University, Medford, MA 02155, USA
| |
Collapse
|
7
|
Li R, Yang D, Fang F, Hong KS, Reiss AL, Zhang Y. Concurrent fNIRS and EEG for Brain Function Investigation: A Systematic, Methodology-Focused Review. SENSORS (BASEL, SWITZERLAND) 2022; 22:s22155865. [PMID: 35957421 PMCID: PMC9371171 DOI: 10.3390/s22155865] [Citation(s) in RCA: 31] [Impact Index Per Article: 15.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/20/2022] [Revised: 07/27/2022] [Accepted: 07/30/2022] [Indexed: 05/29/2023]
Abstract
Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) stand as state-of-the-art techniques for non-invasive functional neuroimaging. On a unimodal basis, EEG has poor spatial resolution while presenting high temporal resolution. In contrast, fNIRS offers better spatial resolution, though it is constrained by its poor temporal resolution. One important merit shared by the EEG and fNIRS is that both modalities have favorable portability and could be integrated into a compatible experimental setup, providing a compelling ground for the development of a multimodal fNIRS-EEG integration analysis approach. Despite a growing number of studies using concurrent fNIRS-EEG designs reported in recent years, the methodological reference of past studies remains unclear. To fill this knowledge gap, this review critically summarizes the status of analysis methods currently used in concurrent fNIRS-EEG studies, providing an up-to-date overview and guideline for future projects to conduct concurrent fNIRS-EEG studies. A literature search was conducted using PubMed and Web of Science through 31 August 2021. After screening and qualification assessment, 92 studies involving concurrent fNIRS-EEG data recordings and analyses were included in the final methodological review. Specifically, three methodological categories of concurrent fNIRS-EEG data analyses, including EEG-informed fNIRS analyses, fNIRS-informed EEG analyses, and parallel fNIRS-EEG analyses, were identified and explained with detailed description. Finally, we highlighted current challenges and potential directions in concurrent fNIRS-EEG data analyses in future research.
Collapse
Affiliation(s)
- Rihui Li
- Center for Interdisciplinary Brain Sciences Research, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA 94305, USA
- Department of Biomedical Engineering, University of Houston, Houston, TX 77004, USA
| | - Dalin Yang
- School of Mechanical Engineering, Pusan National University, Pusan 43241, Korea
- Mallinckrodt Institute of Radiology, Washington University School of Medicine in St. Louis, 4515 McKinley Avenue, St. Louis, MO 63110, USA
| | - Feng Fang
- Department of Biomedical Engineering, University of Houston, Houston, TX 77004, USA
| | - Keum-Shik Hong
- School of Mechanical Engineering, Pusan National University, Pusan 43241, Korea
| | - Allan L. Reiss
- Center for Interdisciplinary Brain Sciences Research, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA 94305, USA
| | - Yingchun Zhang
- Department of Biomedical Engineering, University of Houston, Houston, TX 77004, USA
| |
Collapse
|
8
|
Sensitive Channel Selection for Mental Workload Classification. MATHEMATICS 2022. [DOI: 10.3390/math10132266] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/10/2022]
Abstract
Mental workload (MW) assessment has been widely studied in various human–machine interaction tasks. The existing researches on MW classification mostly use non-invasive electroencephalography (EEG) caps to collect EEG signals and identify MW levels. However, the activation region of the brain stimulated by MW tasks is not the same for every subject. It may be inappropriate to use EEG signals from all electrode channels to identify MW. In this paper, an EEG rhythm energy heatmap is first established to visually show the change trends in the energy of four EEG rhythms with time, EEG channels and MW levels. It can be concluded from the presented heatmaps that this change trend varies with subjects, rhythms and channels. Based on the analysis, a double threshold method is proposed to select sensitive channels for MW assessment. The EEG signals of personalized selected channels, named positive sensitive channels (PSCs) and negative sensitive channels (NSCs), are used for MW classification using the Support Vector Machine (SVM) algorithm. The results show that the selection of personalized sensitive channels generally contributes to improving the performance of MW classification.
Collapse
|
9
|
Chu H, Cao Y, Jiang J, Yang J, Huang M, Li Q, Jiang C, Jiao X. Optimized electroencephalogram and functional near-infrared spectroscopy-based mental workload detection method for practical applications. Biomed Eng Online 2022; 21:9. [PMID: 35109879 PMCID: PMC8812267 DOI: 10.1186/s12938-022-00980-1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2021] [Accepted: 01/21/2022] [Indexed: 11/14/2022] Open
Abstract
Background Mental workload is a critical consideration in complex man–machine systems design. Among various mental workload detection techniques, multimodal detection techniques integrating electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) signals have attracted considerable attention. However, existing EEG–fNIRS-based mental workload detection methods have certain defects, such as complex signal acquisition channels and low detection accuracy, which restrict their practical application. Methods The signal acquisition configuration was optimized by analyzing the feature importance in mental workload recognition model and a more accurate and convenient EEG–fNIRS-based mental workload detection method was constructed. A classical Multi-Task Attribute Battery (MATB) task was conducted with 20 participating volunteers. Subjective scale data, 64-channel EEG data, and two-channel fNIRS data were collected. Results A higher number of EEG channels correspond to higher detection accuracy. However, there is no obvious improvement in accuracy once the number of EEG channels reaches 26, with a four-level mental workload detection accuracy of 76.25 ± 5.21%. Partial results of physiological analysis verify the results of previous studies, such as that the θ power of EEG and concentration of O2Hb in the prefrontal region increase while the concentration of HHb decreases with task difficulty. It was further observed, for the first time, that the energy of each band of EEG signals was significantly different in the occipital lobe region, and the power of \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$\beta_{1}$$\end{document}β1 and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$\beta_{2}$$\end{document}β2 bands in the occipital region increased significantly with task difficulty. The changing range and the mean amplitude of O2Hb in high-difficulty tasks were significantly higher compared with those in low-difficulty tasks. Conclusions The channel configuration of EEG–fNIRS-based mental workload detection was optimized to 26 EEG channels and two frontal fNIRS channels. A four-level mental workload detection accuracy of 76.25 ± 5.21% was obtained, which is higher than previously reported results. The proposed configuration can promote the application of mental workload detection technology in military, driving, and other complex human–computer interaction systems.
Collapse
Affiliation(s)
- Hongzuo Chu
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China.,Space Engineering University, Beijing, China
| | - Yong Cao
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China
| | - Jin Jiang
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China
| | - Jiehong Yang
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China.,Space Engineering University, Beijing, China
| | - Mengyin Huang
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China.,Space Engineering University, Beijing, China
| | - Qijie Li
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China
| | - Changhua Jiang
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China.
| | - Xuejun Jiao
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing, China. .,Space Engineering University, Beijing, China.
| |
Collapse
|
10
|
|
11
|
Maggi P, Di Nocera F. Sensitivity of the Spatial Distribution of Fixations to Variations in the Type of Task Demand and Its Relation to Visual Entropy. Front Hum Neurosci 2021; 15:642535. [PMID: 34168543 PMCID: PMC8217447 DOI: 10.3389/fnhum.2021.642535] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2020] [Accepted: 04/26/2021] [Indexed: 11/30/2022] Open
Abstract
Ocular activity is known to be sensitive to variations in mental workload, and recent studies have successfully related the distribution of eye fixations to the mental load. This study aimed to verify the effectiveness of the spatial distribution of fixations as a measure of mental workload and its sensitivity to different types of demands imposed by the task: mental, temporal, and physical. To test the research hypothesis, two experimental studies were run: Experiment 1 evaluated the sensitivity of an index of spatial distribution (Nearest Neighbor Index; NNI) to changes in workload. A sample of 30 participants participated in a within-subject design with different types of task demands (mental, temporal, physical) applied to Tetris game; Experiment 2 investigated the accuracy of the index through the analysis of 1-min epochs during the execution of a visual-spatial task (the “spot the differences” puzzle game). Additionally, NNI was compared to a better-known ocular mental workload index, the entropy rate. The data analysis showed a relation between the NNI and the different workload levels imposed by the tasks. In particular: Experiment 1 demonstrated that increased difficulty, due to higher temporal demand, led to a more dispersed pattern with respect to the baseline, whereas the mental demand led to a more grouped pattern of fixations with respect to the baseline; Experiment 2 indicated that the entropy rate and the NNI show a similar pattern over time, indicating high mental workload after the first minute of activity. That suggests that NNI highlights the greater presence of fixation groups and, accordingly, the entropy indicates a more regular and orderly scanpath. Both indices are sensitive to changes in workload and they seem to anticipate the drop in performance. However, the entropy rate is limited by the use of the areas of interest, making it impossible to apply it in dynamic contexts. Conversely, NNI works with the entire scanpath and it shows sensitivity to different types of task demands. These results confirm the NNI as a measure applicable to different contexts and its potential use as a trigger in adaptive systems implemented in high-risk settings, such as control rooms and transportation systems.
Collapse
Affiliation(s)
- Piero Maggi
- Department of Psychology, Sapienza University of Rome, Rome, Italy
| | | |
Collapse
|
12
|
Rosanne O, Albuquerque I, Cassani R, Gagnon JF, Tremblay S, Falk TH. Adaptive Filtering for Improved EEG-Based Mental Workload Assessment of Ambulant Users. Front Neurosci 2021; 15:611962. [PMID: 33897342 PMCID: PMC8058356 DOI: 10.3389/fnins.2021.611962] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2020] [Accepted: 03/08/2021] [Indexed: 11/13/2022] Open
Abstract
Recently, due to the emergence of mobile electroencephalography (EEG) devices, assessment of mental workload in highly ecological settings has gained popularity. In such settings, however, motion and other common artifacts have been shown to severely hamper signal quality and to degrade mental workload assessment performance. Here, we show that classical EEG enhancement algorithms, conventionally developed to remove ocular and muscle artifacts, are not optimal in settings where participant movement (e.g., walking or running) is expected. As such, an adaptive filter is proposed that relies on an accelerometer-based referential signal. We show that when combined with classical algorithms, accurate mental workload assessment is achieved. To test the proposed algorithm, data from 48 participants was collected as they performed the Revised Multi-Attribute Task Battery-II (MATB-II) under a low and a high workload setting, either while walking/jogging on a treadmill, or using a stationary exercise bicycle. Accuracy as high as 95% could be achieved with a random forest based mental workload classifier with ambulant users. Moreover, an increase in gamma activity was found in the parietal cortex, suggesting a connection between sensorimotor integration, attention, and workload in ambulant users.
Collapse
Affiliation(s)
- Olivier Rosanne
- Institut National de la Recherche Scientifique - Centre Énergie, Matériaux et Télécomunication, Université du Québec, Montréal, QC, Canada
| | - Isabela Albuquerque
- Institut National de la Recherche Scientifique - Centre Énergie, Matériaux et Télécomunication, Université du Québec, Montréal, QC, Canada
| | - Raymundo Cassani
- Institut National de la Recherche Scientifique - Centre Énergie, Matériaux et Télécomunication, Université du Québec, Montréal, QC, Canada
| | | | | | - Tiago H Falk
- Institut National de la Recherche Scientifique - Centre Énergie, Matériaux et Télécomunication, Université du Québec, Montréal, QC, Canada
| |
Collapse
|
13
|
Deligani RJ, Borgheai SB, McLinden J, Shahriari Y. Multimodal fusion of EEG-fNIRS: a mutual information-based hybrid classification framework. BIOMEDICAL OPTICS EXPRESS 2021; 12:1635-1650. [PMID: 33796378 PMCID: PMC7984774 DOI: 10.1364/boe.413666] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/28/2020] [Revised: 02/15/2021] [Accepted: 02/16/2021] [Indexed: 05/26/2023]
Abstract
Multimodal data fusion is one of the current primary neuroimaging research directions to overcome the fundamental limitations of individual modalities by exploiting complementary information from different modalities. Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) are especially compelling modalities due to their potentially complementary features reflecting the electro-hemodynamic characteristics of neural responses. However, the current multimodal studies lack a comprehensive systematic approach to properly merge the complementary features from their multimodal data. Identifying a systematic approach to properly fuse EEG-fNIRS data and exploit their complementary potential is crucial in improving performance. This paper proposes a framework for classifying fused EEG-fNIRS data at the feature level, relying on a mutual information-based feature selection approach with respect to the complementarity between features. The goal is to optimize the complementarity, redundancy and relevance between multimodal features with respect to the class labels as belonging to a pathological condition or healthy control. Nine amyotrophic lateral sclerosis (ALS) patients and nine controls underwent multimodal data recording during a visuo-mental task. Multiple spectral and temporal features were extracted and fed to a feature selection algorithm followed by a classifier, which selected the optimized subset of features through a cross-validation process. The results demonstrated considerably improved hybrid classification performance compared to the individual modalities and compared to conventional classification without feature selection, suggesting a potential efficacy of our proposed framework for wider neuro-clinical applications.
Collapse
Affiliation(s)
- Roohollah Jafari Deligani
- Department of Electrical, Computer and
Biomedical Engineering; University of Rhode
Island, Kingston, RI 02881, USA
| | - Seyyed Bahram Borgheai
- Department of Electrical, Computer and
Biomedical Engineering; University of Rhode
Island, Kingston, RI 02881, USA
| | - John McLinden
- Department of Electrical, Computer and
Biomedical Engineering; University of Rhode
Island, Kingston, RI 02881, USA
| | - Yalda Shahriari
- Department of Electrical, Computer and
Biomedical Engineering; University of Rhode
Island, Kingston, RI 02881, USA
- Interdisciplinary Neuroscience Program;
University of Rhode Island, Kingston, RI
02881, USA
| |
Collapse
|
14
|
Debie E, Fernandez Rojas R, Fidock J, Barlow M, Kasmarik K, Anavatti S, Garratt M, Abbass HA. Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review. IEEE TRANSACTIONS ON CYBERNETICS 2021; 51:1542-1555. [PMID: 31545761 DOI: 10.1109/tcyb.2019.2939399] [Citation(s) in RCA: 30] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/28/2023]
Abstract
Considerable progress has been made in improving the estimation accuracy of cognitive workload using various sensor technologies. However, the overall performance of different algorithms and methods remain suboptimal in real-world applications. Some studies in the literature demonstrate that a single modality is sufficient to estimate cognitive workload. These studies are limited to controlled settings, a scenario that is significantly different from the real world where data gets corrupted, interrupted, and delayed. In such situations, the use of multiple modalities is needed. Multimodal fusion approaches have been successful in other domains, such as wireless-sensor networks, in addressing single-sensor weaknesses and improving information quality/accuracy. These approaches are inherently more reliable when a data source is lost. In the cognitive workload literature, sensors, such as electroencephalography (EEG), electrocardiography (ECG), and eye tracking, have shown success in estimating the aspects of cognitive workload. Multimodal approaches that combine data from several sensors together can be more robust for real-time measurement of cognitive workload. In this article, we review the published studies related to multimodal data fusion to estimate the cognitive workload and synthesize their main findings. We identify the opportunities for designing better multimodal fusion systems for cognitive workload modeling.
Collapse
|
15
|
Galoyan T, Betts K, Abramian H, Reddy P, Izzetoglu K, Shewokis PA. Examining Mental Workload in a Spatial Navigation Transfer Game via Functional near Infrared Spectroscopy. Brain Sci 2021; 11:brainsci11010045. [PMID: 33406711 PMCID: PMC7824704 DOI: 10.3390/brainsci11010045] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2020] [Revised: 12/24/2020] [Accepted: 12/29/2020] [Indexed: 12/12/2022] Open
Abstract
The goal of this study was to examine the effects of task-related variables, such as the difficulty level, problem scenario, and experiment week, on performance and mental workload of 27 healthy adult subjects during problem solving within the spatial navigation transfer (SNT) game. The study reports task performance measures such as total time spent on a task (TT) and reaction time (RT); neurophysiological measures involving the use of functional near-infrared spectroscopy (fNIRS); and a subjective rating scale for self-assessment of mental workload (NASA TLX) to test the related hypothesis. Several within-subject repeated-measures factorial ANOVA models were developed to test the main hypothesis. The results revealed a number of interaction effects for the dependent measures of TT, RT, fNIRS, and NASA TLX. The results showed (1) a decrease in TT and RT across the three levels of difficulty from Week 1 to Week 2; (2) an increase in TT and RT for high and medium cognitive load tasks as compared to low cognitive load tasks in both Week 1 and Week 2; (3) an overall increase in oxygenation from Week 1 to Week 2. These findings confirmed that both the behavioral performance and mental workload were sensitive to task manipulations.
Collapse
Affiliation(s)
- Tamara Galoyan
- Department of Educational Psychology, College of Education, The University of Utah, Salt Lake City, UT 84112, USA
- Correspondence:
| | - Kristen Betts
- School of Education, Drexel University, Philadelphia, PA 19104, USA;
| | - Hovag Abramian
- College of Science and Engineering, American University of Armenia, Yerevan 0019, Armenia;
| | - Pratusha Reddy
- School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA 19104, USA; (P.R.); (K.I.); (P.A.S.)
| | - Kurtulus Izzetoglu
- School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA 19104, USA; (P.R.); (K.I.); (P.A.S.)
| | - Patricia A. Shewokis
- School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA 19104, USA; (P.R.); (K.I.); (P.A.S.)
| |
Collapse
|
16
|
Nuamah JK, Seong Y, Jiang S, Park E, Mountjoy D. Evaluating effectiveness of information visualizations using cognitive fit theory: A neuroergonomics approach. APPLIED ERGONOMICS 2020; 88:103173. [PMID: 32678781 DOI: 10.1016/j.apergo.2020.103173] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/24/2019] [Revised: 05/04/2020] [Accepted: 05/20/2020] [Indexed: 06/11/2023]
Abstract
Information visualizations may be evaluated from the perspective of how they match tasks that must be performed with them, a cognitive fit perspective. However, there is a gap between the high-level references made to cognitive fit and the low-level ability to identify and measure it during human interaction with visualizations. We bridge this gap by using an electroencephalography metric derived from frontal midline theta power and parietal alpha power, known as the task load index, to determine if cognitive effort measured at the level of cortical activity is less when cognitive fit is present compared to when cognitive fit is not. We found that when there is cognitive fit between the type of problem to be solved and the information displayed by a system, the task load index is lower compared to when cognitive fit is not present. We support this finding with subjective (NASA task load index) and performance (response time and accuracy) measures. Our approach, using electroencephalography, provides supplemental information to self-report and performance measures. Findings from this study are important because they (1) provide more validity to the cognitive fit theory using a neurophysiological measure, and (2) use the electroencephalography task load index metric as a means to assess cognitive workload and effort in general.
Collapse
Affiliation(s)
- Joseph K Nuamah
- Division of Healthcare Engineering, Department of Radiation Oncology, UNC School of Medicine, Chapel Hill, NC, 27599, United States.
| | - Younho Seong
- Industrial & Systems Engineering Department, North Carolina A&T State University, 1601 East Market Street, Greensboro, NC, 27411, United States.
| | - Steven Jiang
- Industrial & Systems Engineering Department, North Carolina A&T State University, 1601 East Market Street, Greensboro, NC, 27411, United States.
| | - Eui Park
- Industrial & Systems Engineering Department, North Carolina A&T State University, 1601 East Market Street, Greensboro, NC, 27411, United States.
| | - Daniel Mountjoy
- Human Systems Integration Directorate, US Air Force Research Laboratory's 711, th Human Performance Wing, United States.
| |
Collapse
|
17
|
Brain–machine interfaces using functional near-infrared spectroscopy: a review. ARTIFICIAL LIFE AND ROBOTICS 2020. [DOI: 10.1007/s10015-020-00592-9] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
|
18
|
Anderson A, Gropman A, Le Mons C, Stratakis C, Gandjbakhche A. Evaluation of neurocognitive function of prefrontal cortex in ornithine transcarbamylase deficiency. Mol Genet Metab 2020; 129:207-212. [PMID: 31952925 PMCID: PMC7416502 DOI: 10.1016/j.ymgme.2019.12.014] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/24/2019] [Revised: 12/30/2019] [Accepted: 12/30/2019] [Indexed: 02/02/2023]
Abstract
Hyperammonia due to ornithine transcarbamylase deficiency (OTCD) can cause a range of deficiencies in domains of executive function and working memory. Only a few fMRI studies have focused on neuroimaging data in a population with OTCD. Yet, there is a need for monitoring the disease progression and neurocognitive function in this population. In this study, we used a non-invasive neuroimaging technique, functional Near Infrared Spectroscopy (fNIRS), to examine the hemodynamics of prefrontal cortex (PFC) based on neural activation in an OTCD population. Using fNIRS, we measured the activation in PFC of the participants while performing the Stroop task. Behavioral assessment such as reaction time and correct response were recorded. We investigated the difference in behavioral measures as well as brain activation in left and right PFC in patients with OTCD and controls. Results revealed a distinction in left PFC activation between controls and patients with OTCD, where control subjects showed higher task related activation increase. Subjects with OTCD also exhibited bilateral increase in PFC activation. There was no significant difference in response time or correct response between the two groups. Our findings suggest the alterations in neurocognitive function of PFC in OTCD compared to the controls despite the behavioral profiles exhibiting no such differences. This is a first study using fNIRS to examine a neurocognitive function in OTCD population and can provide a novel insight into the screening of OTCD progression and examining neurocognitive changes.
Collapse
Affiliation(s)
- Afrouz Anderson
- NIH, National Institute of Child Health and Human Development, Bethesda, MD 20892, United States of America
| | - Andrea Gropman
- Children's National Medical Center, Division of Neurogenetics and Neurodevelopmental Pediatrics, Washington, DC 20010, United States of America
| | - Cynthia Le Mons
- National Urea Cycle Disorders Foundation, Pasadena, California 91105
| | - Constantine Stratakis
- NIH, National Institute of Child Health and Human Development, Bethesda, MD 20892, United States of America
| | - Amir Gandjbakhche
- NIH, National Institute of Child Health and Human Development, Bethesda, MD 20892, United States of America.
| |
Collapse
|
19
|
Parent M, Peysakhovich V, Mandrick K, Tremblay S, Causse M. The diagnosticity of psychophysiological signatures: Can we disentangle mental workload from acute stress with ECG and fNIRS? Int J Psychophysiol 2019; 146:139-147. [DOI: 10.1016/j.ijpsycho.2019.09.005] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2018] [Revised: 08/09/2019] [Accepted: 09/12/2019] [Indexed: 01/10/2023]
|
20
|
İşbilir E, Çakır MP, Acartürk C, Tekerek AŞ. Towards a Multimodal Model of Cognitive Workload Through Synchronous Optical Brain Imaging and Eye Tracking Measures. Front Hum Neurosci 2019; 13:375. [PMID: 31708760 PMCID: PMC6820355 DOI: 10.3389/fnhum.2019.00375] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/17/2019] [Accepted: 10/03/2019] [Indexed: 01/05/2023] Open
Abstract
Recent advances in neuroimaging technologies have rendered multimodal analysis of operators’ cognitive processes in complex task settings and environments increasingly more practical. In this exploratory study, we utilized optical brain imaging and mobile eye tracking technologies to investigate the behavioral and neurophysiological differences among expert and novice operators while they operated a human-machine interface in normal and adverse conditions. In congruence with related work, we observed that experts tended to have lower prefrontal oxygenation and exhibit gaze patterns that are better aligned with the optimal task sequence with shorter fixation durations as compared to novices. These trends reached statistical significance only in the adverse condition where the operators were prompted with an unexpected error message. Comparisons between hemodynamic and gaze measures before and after the error message indicated that experts’ neurophysiological response to the error involved a systematic increase in bilateral dorsolateral prefrontal cortex (dlPFC) activity accompanied with an increase in fixation durations, which suggests a shift in their attentional state, possibly from routine process execution to problem detection and resolution. The novices’ response was not as strong as that of experts, including a slight increase only in the left dlPFC with a decreasing trend in fixation durations, which is indicative of visual search behavior for possible cues to make sense of the unanticipated situation. A linear discriminant analysis model capitalizing on the covariance structure among hemodynamic and eye movement measures could distinguish experts from novices with 91% accuracy. Despite the small sample size, the performance of the linear discriminant analysis combining eye fixation and dorsolateral oxygenation measures before and after an unexpected event suggests that multimodal approaches may be fruitful for distinguishing novice and expert performance in similar neuroergonomic applications in the field.
Collapse
Affiliation(s)
- Erdinç İşbilir
- Advanced Technologies Directorate, Guidance and Photonics Division, Roketsan Missiles Industries Inc., Ankara, Turkey
| | - Murat Perit Çakır
- Department of Cognitive Science, Informatics Institute, Middle East Technical University, Ankara, Turkey
| | - Cengiz Acartürk
- Department of Cognitive Science, Informatics Institute, Middle East Technical University, Ankara, Turkey
| | - Ali Şimşek Tekerek
- Advanced Technologies Directorate, Guidance and Photonics Division, Roketsan Missiles Industries Inc., Ankara, Turkey
| |
Collapse
|
21
|
Combining functional near-infrared spectroscopy and EEG measurements for the diagnosis of attention-deficit hyperactivity disorder. Neural Comput Appl 2019. [DOI: 10.1007/s00521-019-04294-7] [Citation(s) in RCA: 16] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/26/2022]
|
22
|
Posada-Quintero HF, Bolkhovsky JB. Machine Learning models for the Identification of Cognitive Tasks using Autonomic Reactions from Heart Rate Variability and Electrodermal Activity. Behav Sci (Basel) 2019; 9:bs9040045. [PMID: 31027251 PMCID: PMC6523197 DOI: 10.3390/bs9040045] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2019] [Revised: 04/19/2019] [Accepted: 04/23/2019] [Indexed: 11/28/2022] Open
Abstract
Indices of heart rate variability (HRV) and electrodermal activity (EDA), in conjunction with machine learning models, were used to identify one of three tasks a subject is performing based on autonomic response elicited by the specific task. Using non-invasive measures to identify the task performed by a subject can help to provide individual monitoring and guidance to avoid the consequences of reduced performance due to fatigue or other stressors. In the present study, sixteen subjects were enrolled to undergo three tasks: The psychomotor vigilance task (PVT), an auditory working memory task (the n-back paradigm), and a visual search (ship search, SS). Electrocardiogram (ECG) (for HRV analysis) and EDA data were collected during the tests. For task-classification, we tested four machine learning classification tools: k-nearest neighbor classifier (KNN), support vector machines (SVM), decision trees, and discriminant analysis (DA). Leave-one-subject-out cross-validation was used to evaluate the performance of the constructed models to prevent overfitting. The most accurate models were the KNN (66%), linear SVM (62%), and linear DA (62%). The results of this study showed that it is possible to identify the task a subject is performing based on the subject’s autonomic reactions (from HRV and EDA). This information can be used to monitor individuals within a larger group to assist in reducing errors caused by uncoordinated or poor performance by allowing for automated tracking of and communication between individuals.
Collapse
|
23
|
Liu Y, Ayaz H. Speech Recognition via fNIRS Based Brain Signals. Front Neurosci 2018; 12:695. [PMID: 30356771 PMCID: PMC6189799 DOI: 10.3389/fnins.2018.00695] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2018] [Accepted: 09/18/2018] [Indexed: 11/13/2022] Open
Abstract
In this paper, we present the first evidence that perceived speech can be identified from the listeners' brain signals measured via functional-near infrared spectroscopy (fNIRS)—a non-invasive, portable, and wearable neuroimaging technique suitable for ecologically valid settings. In this study, participants listened audio clips containing English stories while prefrontal and parietal cortices were monitored with fNIRS. Machine learning was applied to train predictive models using fNIRS data from a subject pool to predict which part of a story was listened by a new subject not in the pool based on the brain's hemodynamic response as measured by fNIRS. fNIRS signals can vary considerably from subject to subject due to the different head size, head shape, and spatial locations of brain functional regions. To overcome this difficulty, a generalized canonical correlation analysis (GCCA) was adopted to extract latent variables that are shared among the listeners before applying principal component analysis (PCA) for dimension reduction and applying logistic regression for classification. A 74.7% average accuracy has been achieved for differentiating between two 50 s. long story segments and a 43.6% average accuracy has been achieved for differentiating four 25 s. long story segments. These results suggest the potential of an fNIRS based-approach for building a speech decoding brain-computer-interface for developing a new type of neural prosthetic system.
Collapse
Affiliation(s)
- Yichuan Liu
- School of Biomedical Engineering, Drexel University, Science and Health Systems, Philadelphia, PA, United States.,Cognitive Neuroengineering and Quantitative Experimental Research (CONQUER) Collaborative, Drexel University, Philadelphia, PA, United States
| | - Hasan Ayaz
- School of Biomedical Engineering, Drexel University, Science and Health Systems, Philadelphia, PA, United States.,Cognitive Neuroengineering and Quantitative Experimental Research (CONQUER) Collaborative, Drexel University, Philadelphia, PA, United States.,Department of Family and Community Health, University of Pennsylvania, Philadelphia, PA, United States.,The Division of General Pediatrics, Children's Hospital of Philadelphia, Philadelphia, PA, United States
| |
Collapse
|
24
|
Curtin A, Ayaz H. The Age of Neuroergonomics: Towards Ubiquitous and Continuous Measurement of Brain Function with fNIRS. JAPANESE PSYCHOLOGICAL RESEARCH 2018. [DOI: 10.1111/jpr.12227] [Citation(s) in RCA: 52] [Impact Index Per Article: 8.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
25
|
Omurtag A, Aghajani H, Keles HO. Decoding human mental states by whole-head EEG+fNIRS during category fluency task performance. J Neural Eng 2018; 14:066003. [PMID: 28730995 DOI: 10.1088/1741-2552/aa814b] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/19/2022]
Abstract
OBJECTIVE Concurrent scalp electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), which we refer to as EEG+fNIRS, promises greater accuracy than the individual modalities while remaining nearly as convenient as EEG. We sought to quantify the hybrid system's ability to decode mental states and compare it with its unimodal components. APPROACH We recorded from healthy volunteers taking the category fluency test and applied machine learning techniques to the data. MAIN RESULTS EEG+fNIRS's decoding accuracy was greater than that of its subsystems, partly due to the new type of neurovascular features made available by hybrid data. SIGNIFICANCE Availability of an accurate and practical decoding method has potential implications for medical diagnosis, brain-computer interface design, and neuroergonomics.
Collapse
Affiliation(s)
- Ahmet Omurtag
- Engineering Department, Nottingham Trent University, Nottingham, United Kingdom
| | | | | |
Collapse
|
26
|
Anderson AA, Parsa K, Geiger S, Zaragoza R, Kermanian R, Miguel H, Dashtestani H, Chowdhry FA, Smith E, Aram S, Gandjbakhche AH. Exploring the role of task performance and learning style on prefrontal hemodynamics during a working memory task. PLoS One 2018; 13:e0198257. [PMID: 29870536 PMCID: PMC5988299 DOI: 10.1371/journal.pone.0198257] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/02/2017] [Accepted: 05/16/2018] [Indexed: 11/19/2022] Open
Abstract
Existing literature outlines the quality and location of activation in the prefrontal cortex (PFC) during working memory (WM) tasks. However, the effects of individual differences on the underlying neural process of WM tasks are still unclear. In this functional near infrared spectroscopy study, we administered a visual and auditory n-back task to examine activation in the PFC while considering the influences of task performance, and preferred learning strategy (VARK score). While controlling for age, results indicated that high performance (HP) subjects (accuracy > 90%) showed task dependent lower activation compared to normal performance subjects in PFC region Specifically HP groups showed lower activation in left dorsolateral PFC (DLPFC) region during performance of auditory task whereas during visual task they showed lower activation in the right DLPFC. After accounting for learning style, we found a correlation between visual and aural VARK score and level of activation in the PFC. Subjects with higher visual VARK scores displayed lower activation during auditory task in left DLPFC, while those with higher visual scores exhibited higher activation during visual task in bilateral DLPFC. During performance of auditory task, HP subjects had higher visual VARK scores compared to NP subjects indicating an effect of learning style on the task performance and activation. The results of this study show that learning style and task performance can influence PFC activation, with applications toward neurological implications of learning style and populations with deficits in auditory or visual processing.
Collapse
Affiliation(s)
- Afrouz A. Anderson
- National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, MD, United States of America
| | - Kian Parsa
- National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, MD, United States of America
| | - Sydney Geiger
- St. Olaf College, Northfield, MN, United States of America
| | - Rachel Zaragoza
- National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, MD, United States of America
| | - Riley Kermanian
- National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, MD, United States of America
| | - Helga Miguel
- National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, MD, United States of America
| | - Hadis Dashtestani
- National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, MD, United States of America
| | - Fatima A. Chowdhry
- National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, MD, United States of America
| | - Elizabeth Smith
- National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, MD, United States of America
| | - Siamak Aram
- Analytics Department, Harrisburg University of Science and Technology, Harrisburg, PA, United States of America
| | - Amir H. Gandjbakhche
- National Institute of Child Health and Human Development, National Institutes of Health, Bethesda, MD, United States of America
- * E-mail:
| |
Collapse
|
27
|
Liu Y, Ayaz H, Shewokis PA. Multisubject "Learning" for Mental Workload Classification Using Concurrent EEG, fNIRS, and Physiological Measures. Front Hum Neurosci 2017; 11:389. [PMID: 28798675 PMCID: PMC5529418 DOI: 10.3389/fnhum.2017.00389] [Citation(s) in RCA: 46] [Impact Index Per Article: 6.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2017] [Accepted: 07/12/2017] [Indexed: 11/13/2022] Open
Abstract
An accurate measure of mental workload level has diverse neuroergonomic applications ranging from brain computer interfacing to improving the efficiency of human operators. In this study, we integrated electroencephalogram (EEG), functional near-infrared spectroscopy (fNIRS), and physiological measures for the classification of three workload levels in an n-back working memory task. A significantly better than chance level classification was achieved by EEG-alone, fNIRS-alone, physiological alone, and EEG+fNIRS based approaches. The results confirmed our previous finding that integrating EEG and fNIRS significantly improved workload classification compared to using EEG-alone or fNIRS-alone. The inclusion of physiological measures, however, does not significantly improves EEG-based or fNIRS-based workload classification. A major limitation of currently available mental workload assessment approaches is the requirement to record lengthy calibration data from the target subject to train workload classifiers. We show that by learning from the data of other subjects, workload classification accuracy can be improved especially when the amount of data from the target subject is small.
Collapse
Affiliation(s)
- Yichuan Liu
- School of Biomedical Engineering, Science and Health Systems, Drexel UniversityPhiladelphia, PA, United States.,Cognitive Neuroengineering and Quantitative Experimental Research Collaborative, Drexel UniversityPhiladelphia, PA, United States
| | - Hasan Ayaz
- School of Biomedical Engineering, Science and Health Systems, Drexel UniversityPhiladelphia, PA, United States.,Cognitive Neuroengineering and Quantitative Experimental Research Collaborative, Drexel UniversityPhiladelphia, PA, United States.,Department of Family and Community Health, University of PennsylvaniaPhiladelphia, PA, United States.,Division of General Pediatrics, Children's Hospital of PhiladelphiaPhiladelphia, PA, United States
| | - Patricia A Shewokis
- School of Biomedical Engineering, Science and Health Systems, Drexel UniversityPhiladelphia, PA, United States.,Cognitive Neuroengineering and Quantitative Experimental Research Collaborative, Drexel UniversityPhiladelphia, PA, United States.,Nutrition Sciences Department, College of Nursing and Health Professions, Drexel UniversityPhiladelphia, PA, United States
| |
Collapse
|