GAUDIE: Development, validation, and exploration of a naturalistic German AUDItory Emotional database

Since thoroughly validated naturalistic affective German speech stimulus databases are rare, we present here a novel validated database of speech sequences assembled with the purpose of emotion induction. The database comprises 37 audio speech sequences with a total duration of 92 minutes for the induction of positive, neutral, and negative emotion: comedian shows intending to elicit humorous and amusing feelings, weather forecasts, and arguments between couples and relatives from movies or television series. Multiple continuous and discrete ratings are used to validate the database to capture the time course and variabilities of valence and arousal. We analyse and quantify how well the audio sequences fulfil quality criteria of differentiation, salience/strength, and generalizability across participants. Hence, we provide a validated speech database of naturalistic scenarios suitable to investigate emotion processing and its time course with German-speaking participants. Information on using the stimulus database for research purposes can be found at the OSF project repository GAUDIE: https://osf.io/xyr6j/ .

AI-assisted ethics? considerations of AI simulation for the ethical assessment and design of assistive technologies

Current ethical debates on the use of artificial intelligence (AI) in healthcare treat AI as a product of technology in three ways. First, by assessing risks and potential benefits of currently developed AI-enabled products with ethical checklists; second, by proposing ex ante lists of ethical values seen as relevant for the design and development of assistive technology, and third, by promoting AI technology to use moral reasoning as part of the automation process. The dominance of these three perspectives in the discourse is demonstrated by a brief summary of the literature. Subsequently, we propose a fourth approach to AI, namely, as a methodological tool to assist ethical reflection. We provide a concept of an AI-simulation informed by three separate elements: 1) stochastic human behavior models based on behavioral data for simulating realistic settings, 2) qualitative empirical data on value statements regarding internal policy, and 3) visualization components that aid in understanding the impact of changes in these variables. The potential of this approach is to inform an interdisciplinary field about anticipated ethical challenges or ethical trade-offs in concrete settings and, hence, to spark a re-evaluation of design and implementation plans. This may be particularly useful for applications that deal with extremely complex values and behavior or with limitations on the communication resources of affected persons (e.g., persons with dementia care or for care of persons with cognitive impairment). Simulation does not replace ethical reflection but does allow for detailed, context-sensitive analysis during the design process and prior to implementation. Finally, we discuss the inherently quantitative methods of analysis afforded by stochastic simulations as well as the potential for ethical discussions and how simulations with AI can improve traditional forms of thought experiments and future-oriented technology assessment.

Grasp-specific high-frequency broadband mirror neuron activity during reach-and-grasp movements in humans

Broadly congruent mirror neurons, responding to any grasp movement, and strictly congruent mirror neurons, responding only to specific grasp movements, have been reported in single-cell studies with primates. Delineating grasp properties in humans is essential to understand the human mirror neuron system with implications for behavior and social cognition. We analyzed electrocorticography data from a natural reach-and-grasp movement observation and delayed imitation task with 3 different natural grasp types of everyday objects. We focused on the classification of grasp types from high-frequency broadband mirror activation patterns found in classic mirror system areas, including sensorimotor, supplementary motor, inferior frontal, and parietal cortices. Classification of grasp types was successful during movement observation and execution intervals but not during movement retention. Our grasp type classification from combined and single mirror electrodes provides evidence for grasp-congruent activity in the human mirror neuron system potentially arising from strictly congruent mirror neurons.

Improved Motion Artifact Correction in fNIRS Data by Combining Wavelet and Correlation-Based Signal Improvement

Functional near-infrared spectroscopy (fNIRS) is an optical non-invasive neuroimaging technique that allows participants to move relatively freely. However, head movements frequently cause optode movements relative to the head, leading to motion artifacts (MA) in the measured signal. Here, we propose an improved algorithmic approach for MA correction that combines wavelet and correlation-based signal improvement (WCBSI). We compare its MA correction accuracy to multiple established correction approaches (spline interpolation, spline-Savitzky-Golay filter, principal component analysis, targeted principal component analysis, robust locally weighted regression smoothing filter, wavelet filter, and correlation-based signal improvement) on real data. Therefore, we measured brain activity in 20 participants performing a hand-tapping task and simultaneously moving their head to produce MAs at different levels of severity. In order to obtain a "ground truth" brain activation, we added a condition in which only the tapping task was performed. We compared the MA correction performance among the algorithms on four predefined metrics (R, RMSE, MAPE, and ΔAUC) and ranked the performances. The suggested WCBSI algorithm was the only one exceeding average performance (p < 0.001), and it had the highest probability to be the best ranked algorithm (78.8% probability). Together, our results indicate that among all algorithms tested, our suggested WCBSI approach performed consistently favorably across all measures.

Inter-individual single-trial classification of MEG data using M-CCA

Neuroscientific studies often involve some form of group analysis over multiple participants. This requires alignment of recordings across participants. A naive solution is to assume that participants' recordings can be aligned anatomically in sensor space. However, this assumption is likely violated due to anatomical and functional differences between individual brains. In magnetoencephalography (MEG) recordings the problem of inter-subject alignment is exacerbated by the susceptibility of MEG to individual cortical folding patterns as well as the inter-subject variability of sensor locations over the brain due to the use of a fixed helmet. Hence, an approach to combine MEG data over individual brains should relax the assumptions that a) brain anatomy and function are tightly linked and b) that the same sensors capture functionally comparable brain activation across individuals. Here we use multiset canonical correlation analysis (M-CCA) to find a common representation of MEG activations recorded from 15 participants performing a grasping task. The M-CCA algorithm was applied to transform the data of a set of multiple participants into a common space with maximum correlation between participants. Importantly, we derive a method to transform data from a new, previously unseen participant into this common representation. This makes it useful for applications that require transfer of models derived from a group of individuals to new individuals. We demonstrate the usefulness and superiority of the approach with respect to previously used approaches. Finally, we show that our approach requires only a small number of labeled data from the new participant. The proposed method demonstrates that functionally motivated common spaces have potential applications in reducing training time of online brain-computer interfaces, where models can be pre-trained on previous participants/sessions. Moreover, inter-subject alignment via M-CCA has the potential for combining data of different participants and could become helpful in future endeavors on large open datasets.

FAIR human neuroscientific data sharing to advance AI driven research and applications: Legal frameworks and missing metadata standards

Modern AI supported research holds many promises for basic and applied science. However, the application of AI methods is often limited because most labs cannot, on their own, acquire large and diverse datasets, which are best for training these methods. Data sharing and open science initiatives promise some relief to the problem, but only if the data are provided in a usable way. The FAIR principles state very general requirements for useful data sharing: they should be findable, accessible, interoperable, and reusable. This article will focus on two challenges to implement the FAIR framework for human neuroscience data. On the one hand, human data can fall under special legal protection. The legal frameworks regulating how and what data can be openly shared differ greatly across countries which can complicate data sharing or even discourage researchers from doing so. Moreover, openly accessible data require standardization of data and metadata organization and annotation in order to become interpretable and useful. This article briefly introduces open neuroscience initiatives that support the implementation of the FAIR principles. It then reviews legal frameworks, their consequences for accessibility of human neuroscientific data and some ethical implications. We hope this comparison of legal jurisdictions helps to elucidate that some alleged obstacles for data sharing only require an adaptation of procedures but help to protect the privacy of our most generous donors to research … our study participants. Finally, it elaborates on the problem of missing standards for metadata annotation and introduces initiatives that aim at developing tools to make neuroscientific data acquisition and analysis pipelines FAIR by design. While the paper focuses on making human neuroscience data useful for data-intensive AI the general considerations hold for other fields where large amounts of openly available human data would be helpful.

Open and reproducible neuroimaging: From study inception to publication

Empirical observations of how labs conduct research indicate that the adoption rate of open practices for transparent, reproducible, and collaborative science remains in its infancy. This is at odds with the overwhelming evidence for the necessity of these practices and their benefits for individual researchers, scientific progress, and society in general. To date, information required for implementing open science practices throughout the different steps of a research project is scattered among many different sources. Even experienced researchers in the topic find it hard to navigate the ecosystem of tools and to make sustainable choices. Here, we provide an integrated overview of community-developed resources that can support collaborative, open, reproducible, replicable, robust and generalizable neuroimaging throughout the entire research cycle from inception to publication and across different neuroimaging modalities. We review tools and practices supporting study inception and planning, data acquisition, research data management, data processing and analysis, and research dissemination. An online version of this resource can be found at https://oreoni.github.io. We believe it will prove helpful for researchers and institutions to make a successful and sustainable move towards open and reproducible science and to eventually take an active role in its future development.

Driver's turning intent recognition model based on brain activation and contextual information

Traffic situations like turning at intersections are destined for safety-critical situations and accidents. Human errors are one of the main reasons for accidents in these situations. A model that recognizes the driver's turning intent could help to reduce accidents by warning the driver or stopping the vehicle before a dangerous turning maneuver. Most models that aim at predicting the probability of a driver's turning intent use only contextual information, such as gap size or waiting time. The objective of this study is to investigate whether the combination of context information and brain activation measurements enhances the recognition of turning intent. We conducted a driving simulator study while simultaneously measuring brain activation using high-density fNIRS. A neural network model for turning intent recognition was trained on the fNIRS and contextual data. The input variables were analyzed using SHAP (SHapley Additive exPlanations) feature importance analysis to show the positive effect of the inclusion of brain activation data. Both the model's evaluation and the feature importance analysis suggest that the combination of context information and brain activation leads to an improved turning intent recognition. The fNIRS results showed increased brain activation differences during the "turn" decision-making phase before turning execution in parts of the left motor cortices, such as the primary motor cortex (PMC; putative BA 4), premotor area (PMA; putative BA 6), and supplementary motor area (SMA; putative BA 8). Furthermore, we also observed increased activation differences in the left prefrontal areas, potentially in the left middle frontal gyrus (putative BA 9), which has been associated with the control of executive functions, such as decision-making and action planning. We hypothesize that brain activation measurements could be a more direct indicator with potentially high specificity for the turning behavior and thus help to increase the recognition model's performance.

Opportunities and Limitations of a Gaze-Contingent Display to Simulate Visual Field Loss in Driving Simulator Studies

Background

Research on task performance under visual field loss is often limited due to small and heterogenous samples. Simulations of visual impairments hold the potential to account for many of those challenges. Digitally altered pictures, glasses, and contact lenses with partial occlusions have been used in the past. One of the most promising methods is the use of a gaze-contingent display that occludes parts of the visual field according to the current gaze position. In this study, the gaze-contingent paradigm was implemented in a static driving simulator to simulate visual field loss and to evaluate parallels in the resulting driving and gaze behavior in comparison to patients.

Methods

The sample comprised 15 participants without visual impairment. All the subjects performed three drives: with full vision, simulated left-sided homonymous hemianopia, and simulated right-sided homonymous hemianopia, respectively. During each drive, the participants drove through an urban environment where they had to maneuver through intersections by crossing straight ahead, turning left, and turning right.

Results

The subjects reported reduced safety and increased workload levels during simulated visual field loss, which was reflected in reduced lane position stability and greater absence of large gaze movements. Initial compensatory strategies could be found concerning a dislocated gaze position and a distorted fixation ratio toward the blind side, which was more pronounced for right-sided visual field loss. During left-sided visual field loss, the participants showed a smaller horizontal range of gaze positions, longer fixation durations, and smaller saccadic amplitudes compared to right-sided homonymous hemianopia and, more distinctively, compared to normal vision.

Conclusion

The results largely mirror reports from driving and visual search tasks under simulated and pathological homonymous hemianopia concerning driving and scanning challenges, initially adopted compensatory strategies, and driving safety. This supports the notion that gaze-contingent displays can be a useful addendum to driving simulator research with visual impairments if the results are interpreted considering methodological limitations and inherent differences to the pathological impairment.

Investigating Differences in Behavior and Brain in Human-Human and Human-Autonomous Vehicle Interactions in Time-Critical Situations

Some studies provide evidence that humans could actively exploit the alleged technological advantages of autonomous vehicles (AVs). This implies that humans may tend to interact differently with AVs as compared to human driven vehicles (HVs) with the knowledge that AVs are programmed to be risk-averse. Hence, it is important to investigate how humans interact with AVs in complex traffic situations. Here, we investigated whether participants would value interactions with AVs differently compared to HVs, and if these differences can be characterized on the behavioral and brain-level. We presented participants with a cover story while recording whole-head brain activity using fNIRS that they were driving under time pressure through urban traffic in the presence of other HVs and AVs. Moreover, the AVs were programmed defensively to avoid collisions and had faster braking reaction times than HVs. Participants would receive a monetary reward if they managed to finish the driving block within a given time-limit without risky driving maneuvers. During the drive, participants were repeatedly confronted with left-lane turning situations at unsignalized intersections. They had to stop and find a gap to turn in front of an oncoming stream of vehicles consisting of HVs and AVs. While the behavioral results did not show any significant difference between the safety margin used during the turning maneuvers with respect to AVs or HVs, participants tended to be more certain in their decision-making process while turning in front of AVs as reflected by the smaller variance in the gap size acceptance as compared to HVs. Importantly, using a multivariate logistic regression approach, we were able to predict whether the participants decided to turn in front of HVs or AVs from whole-head fNIRS in the decision-making phase for every participant (mean accuracy = 67.2%, SD = 5%). Channel-wise univariate fNIRS analysis revealed increased brain activation differences for turning in front of AVs compared to HVs in brain areas that represent the valuation of actions taken during decision-making. The insights provided here may be useful for the development of control systems to assess interactions in future mixed traffic environments involving AVs and HVs.

High-gamma mirror activity patterns in the human brain during reach-to-grasp movement observation, retention, and execution-An MEG study

While the existence of a human mirror neuron system is evident, the involved brain areas and their exact functional roles remain under scientific debate. A number of functionally different mirror neuron types, neurons that selectively respond to specific grasp phases and types for example, have been reported with single cell recordings in monkeys. In humans, spatially limited, intracranially recorded electrophysiological signals in the high-gamma (HG) range have been used to investigate the human mirror system, as they are associated with spiking activity in single neurons. Our goal here is to complement previous intracranial HG studies by using magnetoencephalography to record HG activity simultaneously from the whole head. Participants performed a natural reach-to-grasp movement observation and delayed imitation task with different everyday objects and grasp types. This allowed us to characterize the spatial organization of cortical areas that show HG-activation modulation during movement observation (mirroring), retention (mnemonic mirroring), and execution (motor control). Our results show mirroring related HG modulation patterns over bilateral occipito-parietal as well as sensorimotor areas. In addition, we found mnemonic mirroring related HG modulation over contra-lateral fronto-temporal areas. These results provide a foundation for further human mirror system research as well as possible target areas for brain-computer interface and neurorehabilitation approaches.

Recommendations of Choice of Head Coil and Prescan Normalize Filter Depend on Region of Interest and Task

The performance of MRI head coils together with the influence of the prescan normalize filter in different brain regions was evaluated. Functional and structural data were recorded from 26 participants performing motor, auditory, and visual tasks in different conditions: with the 20- and 64-channel Siemens head/neck coil and the prescan normalize filter turned ON or OFF. Data were analyzed with the MRIQC tool to evaluate data quality differences. The functional data were statistically evaluated by comparison of the β estimates and the time-course signal-to-noise ratio (tSNR) in four regions of interest, i.e., the auditory, visual, and motor cortices and the thalamus. The MRIQC tool indicated a better data quality for both functional and structural data with the prescan normalize filter, with an advantage for the 20-channel head coil in functional data and an advantage for the 64-channel head coil in structural measurements. Nevertheless, recommendations for the functional data regarding choice of head coils and prescan normalize filter depend on the brain regions of interest. Higher β estimates and tSNR values occurred in the auditory cortex and thalamus with the prescan normalize filter, whereas the contrary was true for the visual and motor cortices. Due to higher β estimates in the visual cortex in the 64-channel head coil, this head coil is recommended for studies investigating the visual cortex. For most of the research questions, the 20-channel head coil is better suited for functional experiments, with the prescan normalize filter, especially when investigating deep brain areas. For anatomical studies, the 64-channel head coil seemed to be the better choice.

Generalizable dimensions of human cortical auditory processing of speech in natural soundscapes: A data-driven ultra high field fMRI approach

Speech comprehension in natural soundscapes rests on the ability of the auditory system to extract speech information from a complex acoustic signal with overlapping contributions from many sound sources. Here we reveal the canonical processing of speech in natural soundscapes on multiple scales by using data-driven modeling approaches to characterize sounds to analyze ultra high field fMRI recorded while participants listened to the audio soundtrack of a movie. We show that at the functional level the neuronal processing of speech in natural soundscapes can be surprisingly low dimensional in the human cortex, highlighting the functional efficiency of the auditory system for a seemingly complex task. Particularly, we find that a model comprising three functional dimensions of auditory processing in the temporal lobes is shared across participants' fMRI activity. We further demonstrate that the three functional dimensions are implemented in anatomically overlapping networks that process different aspects of speech in natural soundscapes. One is most sensitive to complex auditory features present in speech, another to complex auditory features and fast temporal modulations, that are not specific to speech, and one codes mainly sound level. These results were derived with few a-priori assumptions and provide a detailed and computationally reproducible account of the cortical activity in the temporal lobe elicited by the processing of speech in natural soundscapes.

Brain Oscillation Entrainment by Perceptible and Non-perceptible Rhythmic Light Stimulation

Objective and Background: Decades of research in the field of steady-state visual evoked potentials (SSVEPs) have revealed great potential of rhythmic light stimulation for brain-computer interfaces. Additionally, rhythmic light stimulation provides a non-invasive method for entrainment of oscillatory activity in the brain. Especially effective protocols enabling non-perceptible rhythmic stimulation and, thereby, reducing eye fatigue and user discomfort are favorable. Here, we investigate effects of (1) perceptible and (2) non-perceptible rhythmic light stimulation as well as attention-based effects of the stimulation by asking participants to focus (a) on the stimulation source directly in an overt attention condition or (b) on a cross-hair below the stimulation source in a covert attention condition. Method: SSVEPs at 10 Hz were evoked with a light-emitting diode (LED) driven by frequency-modulated signals and amplitudes of the current intensity either below or above a previously estimated individual threshold. Furthermore, we explored the effect of attention by asking participants to fixate on the LED directly in the overt attention condition and indirectly attend it in the covert attention condition. By measuring electroencephalography, we analyzed differences between conditions regarding the detection of reliable SSVEPs via the signal-to-noise ratio (SNR) and functional connectivity in occipito-frontal(-central) regions. Results: We could observe SSVEPs at 10 Hz for the perceptible and non-perceptible rhythmic light stimulation not only in the overt but also in the covert attention condition. The SNR and SSVEP amplitudes did not differ between the conditions and SNR values were in all except one participant above significance thresholds suggested by previous literature indicating reliable SSVEP responses. No difference between the conditions could be observed in the functional connectivity in occipito-frontal(-central) regions. Conclusion: The finding of robust SSVEPs even for non-intrusive rhythmic stimulation protocols below an individual perceptibility threshold and without direct fixation on the stimulation source reveals strong potential as a safe stimulation method for oscillatory entrainment in naturalistic applications.

Effects of age-related hearing loss and hearing aid experience on sentence processing

Age-related hearing loss typically affects the hearing of high frequencies in older adults. Such hearing loss influences the processing of spoken language, including higher-level processing such as that of complex sentences. Hearing aids may alleviate some of the speech processing disadvantages associated with hearing loss. However, little is known about the relation between hearing loss, hearing aid use, and their effects on higher-level language processes. This neuroimaging (fMRI) study examined these factors by measuring the comprehension and neural processing of simple and complex spoken sentences in hard-of-hearing older adults (n = 39). Neither hearing loss severity nor hearing aid experience influenced sentence comprehension at the behavioral level. In contrast, hearing loss severity was associated with increased activity in left superior frontal areas and the left anterior insula, but only when processing specific complex sentences (i.e. object-before-subject) compared to simple sentences. Longer hearing aid experience in a sub-set of participants (n = 19) was associated with recruitment of several areas outside of the core speech processing network in the right hemisphere, including the cerebellum, the precentral gyrus, and the cingulate cortex, but only when processing complex sentences. Overall, these results indicate that brain activation for language processing is affected by hearing loss as well as subsequent hearing aid use. Crucially, they show that these effects become apparent through investigation of complex but not simple sentences.

When Hearing Does Not Mean Understanding: On the Neural Processing of Syntactically Complex Sentences by Listeners With Hearing Loss

Purpose Adults with mild-to-moderate age-related hearing loss typically exhibit issues with speech understanding, but their processing of syntactically complex sentences is not well understood. We test the hypothesis that listeners with hearing loss' difficulties with comprehension and processing of syntactically complex sentences are due to the processing of degraded input interfering with the successful processing of complex sentences. Method We performed a neuroimaging study with a sentence comprehension task, varying sentence complexity (through subject-object order and verb-arguments order) and cognitive demands (presence or absence of a secondary task) within subjects. Groups of older subjects with hearing loss (n = 20) and age-matched normal-hearing controls (n = 20) were tested. Results The comprehension data show effects of syntactic complexity and hearing ability, with normal-hearing controls outperforming listeners with hearing loss, seemingly more so on syntactically complex sentences. The secondary task did not influence off-line comprehension. The imaging data show effects of group, sentence complexity, and task, with listeners with hearing loss showing decreased activation in typical speech processing areas, such as the inferior frontal gyrus and superior temporal gyrus. No interactions between group, sentence complexity, and task were found in the neuroimaging data. Conclusions The results suggest that listeners with hearing loss process speech differently from their normal-hearing peers, possibly due to the increased demands of processing degraded auditory input. Increased cognitive demands by means of a secondary visual shape processing task influence neural sentence processing, but no evidence was found that it does so in a different way for listeners with hearing loss and normal-hearing listeners.

Neural Mechanisms Underlying the Processing of Complex Sentences: An fMRI Study

Previous research has shown effects of syntactic complexity on sentence processing. In linguistics, syntactic complexity (caused by different word orders) is traditionally explained by distinct linguistic operations. This study investigates whether different complex word orders indeed result in distinct patterns of neural activity, as would be expected when distinct linguistic operations are applied. Twenty-two older adults performed an auditory sentence processing paradigm in German with and without increased cognitive load. The results show that without increased cognitive load, complex sentences show distinct activation patterns compared with less complex, canonical sentences: complex object-initial sentences show increased activity in the left inferior frontal and temporal regions, whereas complex adjunct-initial sentences show increased activity in occipital and right superior frontal regions. Increased cognitive load seems to affect the processing of different sentence structures differently, increasing neural activity for canonical sentences, but leaving complex sentences relatively unaffected. We discuss these results in the context of the idea that linguistic operations required for processing sentence structures with higher levels of complexity involve distinct brain operations.

Sentence processing is modulated by the current linguistic environment and a priori information: An fMRI study

INTRODUCTION

Words are not processed in isolation but in rich contexts that are used to modulate and facilitate language comprehension. Here, we investigate distinct neural networks underlying two types of contexts, the current linguistic environment and verb-based syntactic preferences.

METHODS

We had two main manipulations. The first was the current linguistic environment, where the relative frequencies of two syntactic structures (prepositional object [PO] and double-object [DO]) would either follow everyday linguistic experience or not. The second concerned the preference toward one or the other structure depending on the verb; learned in everyday language use and stored in memory. German participants were reading PO and DO sentences in German while brain activity was measured with functional magnetic resonance imaging.

RESULTS

First, the anterior cingulate cortex (ACC) showed a pattern of activation that integrated the current linguistic environment with everyday linguistic experience. When the input did not match everyday experience, the unexpected frequent structure showed higher activation in the ACC than the other conditions and more connectivity from the ACC to posterior parts of the language network. Second, verb-based surprisal of seeing a structure given a verb (PO verb preference but DO structure presentation) resulted, within the language network (left inferior frontal and left middle/superior temporal gyrus) and the precuneus, in increased activation compared to a predictable verb-structure pairing.

CONCLUSION

In conclusion, (1) beyond the canonical language network, brain areas engaged in prediction and error signaling, such as the ACC, might use the statistics of syntactic structures to modulate language processing, (2) the language network is directly engaged in processing verb preferences. These two networks show distinct influences on sentence processing.

Demonstrating Brain-Level Interactions Between Visuospatial Attentional Demands and Working Memory Load While Driving Using Functional Near-Infrared Spectroscopy

Driving is a complex task concurrently drawing on multiple cognitive resources. Yet, there is a lack of studies investigating interactions at the brain-level among different driving subtasks in dual-tasking. This study investigates how visuospatial attentional demands related to increased driving difficulty interacts with different working memory load (WML) levels at the brain level. Using multichannel whole-head high density functional near-infrared spectroscopy (fNIRS) brain activation measurements, we aimed to predict driving difficulty level, both separate for each WML level and with a combined model. Participants drove for approximately 60 min on a highway with concurrent traffic in a virtual reality driving simulator. In half of the time, the course led through a construction site with reduced lane width, increasing visuospatial attentional demands. Concurrently, participants performed a modified version of the n-back task with five different WML levels (from 0-back up to 4-back), forcing them to continuously update, memorize, and recall the sequence of the previous 'n' speed signs and adjust their speed accordingly. Using multivariate logistic ridge regression, we were able to correctly predict driving difficulty in 75.0% of the signal samples (1.955 Hz sampling rate) across 15 participants in an out-of-sample cross-validation of classifiers trained on fNIRS data separately for each WML level. There was a significant effect of the WML level on the driving difficulty prediction accuracies [range 62.2-87.1%; χ2(4) = 19.9, p < 0.001, Kruskal-Wallis H test] with highest prediction rates at intermediate WML levels. On the contrary, training one classifier on fNIRS data across all WML levels severely degraded prediction performance (mean accuracy of 46.8%). Activation changes in the bilateral dorsal frontal (putative BA46), bilateral inferior parietal (putative BA39), and left superior parietal (putative BA7) areas were most predictive to increased driving difficulty. These discriminative patterns diminished at higher WML levels indicating that visuospatial attentional demands and WML involve interacting underlying brain processes. The changing pattern of driving difficulty related brain areas across WML levels could indicate potential changes in the multitasking strategy with level of WML demand, in line with the multiple resource theory.

Cortical and subcortical areas involved in the regulation of reach movement speed in the human brain: An fMRI study

Reach movements are characterized by multiple kinematic variables that can change with age or due to medical conditions such as movement disorders. While the neural control of reach direction is well investigated, the elements of the neural network regulating speed (the nondirectional component of velocity) remain uncertain. Here, we used a custom made magnetic resonance (MR)-compatible arm movement tracking system to capture the real kinematics of the arm movements while measuring brain activation with functional magnetic resonance imaging to reveal areas in the human brain in which BOLD-activation covaries with the speed of arm movements. We found significant activation in multiple cortical and subcortical brain regions positively correlated with endpoint (wrist) speed (speed-related activation), including contralateral premotor cortex (PMC), supplementary motor area (SMA), thalamus (putative VL/VA nuclei), and bilateral putamen. The hand and arm regions of primary sensorimotor cortex (SMC) and a posterior region of thalamus were significantly activated by reach movements but showed a more binary response characteristics (movement present or absent) than with continuously varying speed. Moreover, a subregion of contralateral SMA also showed binary movement activation but no speed-related BOLD-activation. Effect size analysis revealed bilateral putamen as the most speed-specific region among the speed-related clusters whereas primary SMC showed the strongest specificity for movement versus non-movement discrimination, independent of speed variations. The results reveal a network of multiple cortical and subcortical brain regions that are involved in speed regulation among which putamen, anterior thalamus, and PMC show highest specificity to speed, suggesting a basal-ganglia-thalamo-cortical loop for speed regulation.

Recognizing Frustration of Drivers From Face Video Recordings and Brain Activation Measurements With Functional Near-Infrared Spectroscopy

Experiencing frustration while driving can harm cognitive processing, result in aggressive behavior and hence negatively influence driving performance and traffic safety. Being able to automatically detect frustration would allow adaptive driver assistance and automation systems to adequately react to a driver's frustration and mitigate potential negative consequences. To identify reliable and valid indicators of driver's frustration, we conducted two driving simulator experiments. In the first experiment, we aimed to reveal facial expressions that indicate frustration in continuous video recordings of the driver's face taken while driving highly realistic simulator scenarios in which frustrated or non-frustrated emotional states were experienced. An automated analysis of facial expressions combined with multivariate logistic regression classification revealed that frustrated time intervals can be discriminated from non-frustrated ones with accuracy of 62.0% (mean over 30 participants). A further analysis of the facial expressions revealed that frustrated drivers tend to activate muscles in the mouth region (chin raiser, lip pucker, lip pressor). In the second experiment, we measured cortical activation with almost whole-head functional near-infrared spectroscopy (fNIRS) while participants experienced frustrating and non-frustrating driving simulator scenarios. Multivariate logistic regression applied to the fNIRS measurements allowed us to discriminate between frustrated and non-frustrated driving intervals with higher accuracy of 78.1% (mean over 12 participants). Frustrated driving intervals were indicated by increased activation in the inferior frontal, putative premotor and occipito-temporal cortices. Our results show that facial and cortical markers of frustration can be informative for time resolved driver state identification in complex realistic driving situations. The markers derived here can potentially be used as an input for future adaptive driver assistance and automation systems that detect driver frustration and adaptively react to mitigate it.

Electrocorticography reveals continuous auditory and visual speech tracking in temporal and occipital cortex

During natural speech perception, humans must parse temporally continuous auditory and visual speech signals into sequences of words. However, most studies of speech perception present only single words or syllables. We used electrocorticography (subdural electrodes implanted on the brains of epileptic patients) to investigate the neural mechanisms for processing continuous audiovisual speech signals consisting of individual sentences. Using partial correlation analysis, we found that posterior superior temporal gyrus (pSTG) and medial occipital cortex tracked both the auditory and the visual speech envelopes. These same regions, as well as inferior temporal cortex, responded more strongly to a dynamic video of a talking face compared to auditory speech paired with a static face. Occipital cortex and pSTG carry temporal information about both auditory and visual speech dynamics. Visual speech tracking in pSTG may be a mechanism for enhancing perception of degraded auditory speech.

Encoding and Decoding Models in Cognitive Electrophysiology

Cognitive neuroscience has seen rapid growth in the size and complexity of data recorded from the human brain as well as in the computational tools available to analyze this data. This data explosion has resulted in an increased use of multivariate, model-based methods for asking neuroscience questions, allowing scientists to investigate multiple hypotheses with a single dataset, to use complex, time-varying stimuli, and to study the human brain under more naturalistic conditions. These tools come in the form of "Encoding" models, in which stimulus features are used to model brain activity, and "Decoding" models, in which neural features are used to generated a stimulus output. Here we review the current state of encoding and decoding models in cognitive electrophysiology and provide a practical guide toward conducting experiments and analyses in this emerging field. Our examples focus on using linear models in the study of human language and audition. We show how to calculate auditory receptive fields from natural sounds as well as how to decode neural recordings to predict speech. The paper aims to be a useful tutorial to these approaches, and a practical introduction to using machine learning and applied statistics to build models of neural activity. The data analytic approaches we discuss may also be applied to other sensory modalities, motor systems, and cognitive systems, and we cover some examples in these areas. In addition, a collection of Jupyter notebooks is publicly available as a complement to the material covered in this paper, providing code examples and tutorials for predictive modeling in python. The aim is to provide a practical understanding of predictive modeling of human brain data and to propose best-practices in conducting these analyses.

Tradeoff between User Experience and BCI Classification Accuracy with Frequency Modulated Steady-State Visual Evoked Potentials

Steady-state visual evoked potentials (SSVEPs) have been widely employed for the control of brain-computer interfaces (BCIs) because they are very robust, lead to high performance, and allow for a high number of commands. However, such flickering stimuli often also cause user discomfort and fatigue, especially when several light sources are used simultaneously. Different variations of SSVEP driving signals have been proposed to increase user comfort. Here, we investigate the suitability of frequency modulation of a high frequency carrier for SSVEP-BCIs. We compared BCI performance and user experience between frequency modulated (FM) and traditional sinusoidal (SIN) SSVEPs in an offline classification paradigm with four independently flickering light-emitting diodes which were overtly attended (fixated). While classification performance was slightly reduced with the FM stimuli, the user comfort was significantly increased. Comparing the SSVEPs for covert attention to the stimuli (without fixation) was not possible, as no reliable SSVEPs were evoked. Our results reveal that several, simultaneously flickering, light emitting diodes can be used to generate FM-SSVEPs with different frequencies and the resulting occipital electroencephalography (EEG) signals can be classified with high accuracy. While the performance we report could be further improved with adjusted stimuli and algorithms, we argue that the increased comfort is an important result and suggest the use of FM stimuli for future SSVEP-BCI applications.

Assessing the Driver's Current Level of Working Memory Load with High Density Functional Near-infrared Spectroscopy: A Realistic Driving Simulator Study

Cognitive overload or underload results in a decrease in human performance which may result in fatal incidents while driving. We envision that driver assistive systems which adapt their functionality to the driver's cognitive state could be a promising approach to reduce road accidents due to human errors. This research attempts to predict variations of cognitive working memory load levels in a natural driving scenario with multiple parallel tasks and to reveal predictive brain areas. We used a modified version of the n-back task to induce five different working memory load levels (from 0-back up to 4-back) forcing the participants to continuously update, memorize, and recall the previous 'n' speed sequences and adjust their speed accordingly while they drove for approximately 60 min on a highway with concurrent traffic in a virtual reality driving simulator. We measured brain activation using multichannel whole head, high density functional near-infrared spectroscopy (fNIRS) and predicted working memory load level from the fNIRS data by combining multivariate lasso regression and cross-validation. This allowed us to predict variations in working memory load in a continuous time-resolved manner with mean Pearson correlations between induced and predicted working memory load over 15 participants of 0.61 [standard error (SE) 0.04] and a maximum of 0.8. Restricting the analysis to prefrontal sensors placed over the forehead reduced the mean correlation to 0.38 (SE 0.04), indicating additional information gained through whole head coverage. Moreover, working memory load predictions derived from peripheral heart rate parameters achieved much lower correlations (mean 0.21, SE 0.1). Importantly, whole head fNIRS sampling revealed increasing brain activation in bilateral inferior frontal and bilateral temporo-occipital brain areas with increasing working memory load levels suggesting that these areas are specifically involved in workload-related processing.

An MR-compatible gyroscope-based arm movement tracking system

BACKGROUND

Functional magnetic resonance imaging is well suited to link neural population activation with movement parameters of complex natural arm movements. However, currently existing MR-compatible arm tracking devices are not constructed to measure arm joint movement parameters of unrestricted movements. Therefore, to date most research focuses on simple arm movements or includes very little knowledge about the actual movement kinematics.

NEW METHOD

We developed a low cost gyroscope-based arm movement tracking system (GAMTS) that features MR-compatibility. The system consists of dual-axis analogue gyroscopes that measure rotations of upper and lower arm joints. After MR artifact reduction, the rotation angles of the individual arm joints are calculated and used to animate a realistic arm model that is implemented in the OpenSim platform. The OpenSim platform can then provide the kinematics of any point on the arm model.

RESULTS

In order to demonstrate the capabilities of the system, we first assessed the quality of reconstructed wrist movements in a low-noise environment where typical MR-related problems are absent and finally, we validated the reconstruction in the MR environment.

COMPARISON WITH EXISTING METHODS

The system provides the kinematics of the whole arm when natural unrestricted arm movements are performed inside the MR-scanner.

CONCLUSION

The GAMTS is reliably capable of reconstructing the kinematics of trajectories and the reconstruction error is small in comparison with the movement induced variation of speed, displacement, and rotation. Moreover, the system can be used to probe brain areas for their correlation with movement kinematics.

Human centromedian-parafascicular complex signals sensory cues for goal-oriented behavior selection

Experimental research has shown that the centromedian-parafascicular complex (CM-Pf) of the intralaminar thalamus is activated in attentional orienting and processing of behaviorally relevant stimuli. These observations resulted in the hypothesis that the CM-Pf plays a pivotal role in goal-oriented behavior selection. We here set out to test this hypothesis with electrophysiological recordings from patients with electrodes implanted in CM-Pf for deep brain stimulation (DBS) treatment of chronic neuropathic pain. Six patients participated in (1) an auditory three-class oddball experiment, which required a button press to target tones, but not to standard and deviant tones and in (2) a multi-speaker experiment with a target word that required attention selection and a target image that required response selection. Subjects showed transient neural responses (8-15Hz) to the target tone and the target word. Two subjects additionally showed transient neural responses (15-25Hz) to the target image. All sensory target stimuli were related to an internal goal and required a behavior selection (attention selection, response selection). In group analyses, neural responses were greater to target tones than deviant and standard tones and to target words than other task-relevant words that did not require attention selection. The transient neural responses occurred after the target stimuli but prior to the overt behavioral response. Our results demonstrate that in human subjects the CM-Pf is involved in signaling sensory inputs related to goal-oriented selection of behavior.

Frontal and motor cortex contributions to response inhibition: evidence from electrocorticography

Changes in the environment require rapid modification or inhibition of ongoing behavior. We used the stop-signal paradigm and intracranial recordings to investigate response preparation, inhibition, and monitoring of task-relevant information. Electrocorticographic data were recorded in eight patients with electrodes covering frontal, temporal, and parietal cortex, and time-frequency analysis was used to examine power differences in the beta (13-30 Hz) and high-gamma bands (60-180 Hz). Over motor cortex, beta power decreased, and high-gamma power increased during motor preparation for both go trials (Go) and unsuccessful stops (US). For successful stops (SS), beta increased, and high-gamma was reduced, indexing the cancellation of the prepared response. In the middle frontal gyrus (MFG), stop signals elicited a transient high-gamma increase. The MFG response occurred before the estimated stop-signal reaction time but did not distinguish between SS and US trials, likely signaling attention to the salient stop stimulus. A postresponse high-gamma increase in MFG was stronger for US compared with SS and absent in Go, supporting a role in behavior monitoring. These results provide evidence for differential contributions of frontal subregions to response inhibition, including motor preparation and inhibitory control in motor cortex and cognitive control and action evaluation in lateral prefrontal cortex.

Qualitative assessment of patients' attitudes and expectations toward BCIs and implications for future technology development

Brain-computer-interfaces (BCIs) are important for the next generation of neuro-prosthesis innovations. Only few pilot projects have tested patients' abilities to control BCIs as well as their satisfaction with the offered technologies. On the one hand, little is known about patients' moral attitudes toward the benefit-risk-ratio of BCIs as well as their needs, priorities, and expectations. On the other hand, ethics experts intensively discuss the general risks of BCIs as well as the limits of neuro-enhancement. To our knowledge, we present here the first qualitative interview study with ten chronic patients matching the potential user categories for motor and communication BCIs to assess their practical and moral attitudes toward this technology. The interviews reveal practical and moral attitudes toward motor BCIs that can impact future technology development. We discuss our empirical findings on patients' perspectives and compare them to neuroscientists' and ethicists' perspectives. Our analysis indicates only partial overlap between the potential users' and the experts' assessments of BCI-technology. It points out the importance of considering the needs and desires of the targeted patient group. Based on our findings, we suggest a multi-fold approach to the development of clinical BCIs, rooted in the participatory technology-development. We conclude that clinical BCI development needs to be explored in a disease-related and culturally sensitive way.

Online tracking of the contents of conscious perception using real-time fMRI

Perception is an active process that interprets and structures the stimulus input based on assumptions about its possible causes. We use real-time functional magnetic resonance imaging (rtfMRI) to investigate a particularly powerful demonstration of dynamic object integration in which the same physical stimulus intermittently elicits categorically different conscious object percepts. In this study, we simulated an outline object that is moving behind a narrow slit. With such displays, the physically identical stimulus can elicit categorically different percepts that either correspond closely to the physical stimulus (vertically moving line segments) or represent a hypothesis about the underlying cause of the physical stimulus (a horizontally moving object that is partly occluded). In the latter case, the brain must construct an object from the input sequence. Combining rtfMRI with machine learning techniques we show that it is possible to determine online the momentary state of a subject's conscious percept from time resolved BOLD-activity. In addition, we found that feedback about the currently decoded percept increased the decoding rates compared to prior fMRI recordings of the same stimulus without feedback presentation. The analysis of the trained classifier revealed a brain network that discriminates contents of conscious perception with antagonistic interactions between early sensory areas that represent physical stimulus properties and higher-tier brain areas. During integrated object percepts, brain activity decreases in early sensory areas and increases in higher-tier areas. We conclude that it is possible to use BOLD responses to reliably track the contents of conscious visual perception with a relatively high temporal resolution. We suggest that our approach can also be used to investigate the neural basis of auditory object formation and discuss the results in the context of predictive coding theory.

Hidden Markov model and support vector machine based decoding of finger movements using electrocorticography

OBJECTIVE

Support vector machines (SVM) have developed into a gold standard for accurate classification in brain-computer interfaces (BCI). The choice of the most appropriate classifier for a particular application depends on several characteristics in addition to decoding accuracy. Here we investigate the implementation of hidden Markov models (HMM) for online BCIs and discuss strategies to improve their performance.

APPROACH

We compare the SVM, serving as a reference, and HMMs for classifying discrete finger movements obtained from electrocorticograms of four subjects performing a finger tapping experiment. The classifier decisions are based on a subset of low-frequency time domain and high gamma oscillation features.

MAIN RESULTS

We show that decoding optimization between the two approaches is due to the way features are extracted and selected and less dependent on the classifier. An additional gain in HMM performance of up to 6% was obtained by introducing model constraints. Comparable accuracies of up to 90% were achieved with both SVM and HMM with the high gamma cortical response providing the most important decoding information for both techniques.

SIGNIFICANCE

We discuss technical HMM characteristics and adaptations in the context of the presented data as well as for general BCI applications. Our findings suggest that HMMs and their characteristics are promising for efficient online BCIs.

BOLD responses in human V1 to local structure in natural scenes: Implications for theories of visual coding

In this study we tested predictions of two important theories of visual coding, contrast energy and sparse coding theory, on the dependence of population activity level and metabolic demands on spatial structure of the visual input. With carefully calibrated displays we find that in humans neither the V1 blood oxygenation level dependent (BOLD) response nor the initial visually evoked fields in magnetoencephalography (MEG) are sensitive to phase perturbations in photographs of natural scenes. As a control, we quantitatively show that the applied phase perturbations decrease sparseness (kurtosis) of our stimuli but preserve their root mean square (RMS) contrast. Importantly, we show that the lack of sensitivity of the V1 population response level to phase perturbations is not due to a lack of sensitivity of our methods because V1 responses were highly sensitive to variations of image RMS contrast. Our results suggest that the transition from a sparse to a distributed neural code in the early visual system induced by reducing image sparseness has negligible consequences for population metabolic cost. This result imposes a novel and important empirical constraint on quantitative models of sparse coding: Population metabolic rate and population activation level is sensitive to second order statistics (RMS contrast) of the input but not to its spatial phase and fourth order statistics (kurtosis).

Single trial discrimination of individual finger movements on one hand: a combined MEG and EEG study

It is crucial to understand what brain signals can be decoded from single trials with different recording techniques for the development of Brain-Machine Interfaces. A specific challenge for non-invasive recording methods are activations confined to small spatial areas on the cortex such as the finger representation of one hand. Here we study the information content of single trial brain activity in non-invasive MEG and EEG recordings elicited by finger movements of one hand. We investigate the feasibility of decoding which of four fingers of one hand performed a slight button press. With MEG we demonstrate reliable discrimination of single button presses performed with the thumb, the index, the middle or the little finger (average over all subjects and fingers 57%, best subject 70%, empirical guessing level: 25.1%). EEG decoding performance was less robust (average over all subjects and fingers 43%, best subject 54%, empirical guessing level 25.1%). Spatiotemporal patterns of amplitude variations in the time series provided best information for discriminating finger movements. Non-phase-locked changes of mu and beta oscillations were less predictive. Movement related high gamma oscillations were observed in average induced oscillation amplitudes in the MEG but did not provide sufficient information about the finger's identity in single trials. Importantly, pre-movement neuronal activity provided information about the preparation of the movement of a specific finger. Our study demonstrates the potential of non-invasive MEG to provide informative features for individual finger control in a Brain-Machine Interface neuroprosthesis.

Predicting decisions in human social interactions using real-time fMRI and pattern classification

Negotiation and trade typically require a mutual interaction while simultaneously resting in uncertainty which decision the partner ultimately will make at the end of the process. Assessing already during the negotiation in which direction one's counterpart tends would provide a tremendous advantage. Recently, neuroimaging techniques combined with multivariate pattern classification of the acquired data have made it possible to discriminate subjective states of mind on the basis of their neuronal activation signature. However, to enable an online-assessment of the participant's mind state both approaches need to be extended to a real-time technique. By combining real-time functional magnetic resonance imaging (fMRI) and online pattern classification techniques, we show that it is possible to predict human behavior during social interaction before the interacting partner communicates a specific decision. Average accuracy reached approximately 70% when we predicted online the decisions of volunteers playing the ultimatum game, a well-known paradigm in economic game theory. Our results demonstrate the successful online analysis of complex emotional and cognitive states using real-time fMRI, which will enable a major breakthrough for social fMRI by providing information about mental states of partners already during the mutual interaction. Interestingly, an additional whole brain classification across subjects confirmed the online results: anterior insula, ventral striatum, and lateral orbitofrontal cortex, known to act in emotional self-regulation and reward processing for adjustment of behavior, appeared to be strong determinants of later overt behavior in the ultimatum game. Using whole brain classification we were also able to discriminate between brain processes related to subjective emotional and motivational states and brain processes related to the evaluation of objective financial incentives.

Categorical speech representation in human superior temporal gyrus

Speech perception requires the rapid and effortless extraction of meaningful phonetic information from a highly variable acoustic signal. A powerful example of this phenomenon is categorical speech perception, in which a continuum of acoustically varying sounds is transformed into perceptually distinct phoneme categories. We found that the neural representation of speech sounds is categorically organized in the human posterior superior temporal gyrus. Using intracranial high-density cortical surface arrays, we found that listening to synthesized speech stimuli varying in small and acoustically equal steps evoked distinct and invariant cortical population response patterns that were organized by their sensitivities to critical acoustic features. Phonetic category boundaries were similar between neurometric and psychometric functions. Although speech-sound responses were distributed, spatially discrete cortical loci were found to underlie specific phonetic discrimination. Our results provide direct evidence for acoustic-to-higher order phonetic level encoding of speech sounds in human language receptive cortex.

PyMVPA: A Unifying Approach to the Analysis of Neuroscientific Data

The Python programming language is steadily increasing in popularity as the language of choice for scientific computing. The ability of this scripting environment to access a huge code base in various languages, combined with its syntactical simplicity, make it the ideal tool for implementing and sharing ideas among scientists from numerous fields and with heterogeneous methodological backgrounds. The recent rise of reciprocal interest between the machine learning (ML) and neuroscience communities is an example of the desire for an inter-disciplinary transfer of computational methods that can benefit from a Python-based framework. For many years, a large fraction of both research communities have addressed, almost independently, very high-dimensional problems with almost completely non-overlapping methods. However, a number of recently published studies that applied ML methods to neuroscience research questions attracted a lot of attention from researchers from both fields, as well as the general public, and showed that this approach can provide novel and fruitful insights into the functioning of the brain. In this article we show how PyMVPA, a specialized Python framework for machine learning based data analysis, can help to facilitate this inter-disciplinary technology transfer by providing a single interface to a wide array of machine learning libraries and neural data-processing methods. We demonstrate the general applicability and power of PyMVPA via analyses of a number of neural data modalities, including fMRI, EEG, MEG, and extracellular recordings.

Cortical functional anatomy of voluntary saccades in Parkinson disease

In Parkinson Disease (PD) several aspects of saccades are affected. The saccade-generating brainstem neurons are spared, however, the signals they receive may be flawed. In particular voluntary saccades suffer, but the functional anatomy of the impairment of saccade-related cortical control is unknown. We measured blood-oxygenation-level-dependent (BOLD) activation with functional Magnetic Resonance Imaging (fMRI) while healthy participants and patients with PD performed horizontal voluntary saccades between peripheral visual targets or fixated centrally. We compared saccade-related BOLD-activity vs. fixation in patients with PD and in healthy controls and correlated perisaccadic BOLD-activity in PD patients with saccade kinetics (multistep saccades). Saccade related BOLD-activation was found in both PD and healthy participants in the superior parietal cortex (PEF) and the occipital cortex. Our results suggest remarkable hypoactivity of the frontal and supplementary eye fields (FEF and SEF) in PD patients. On the other hand, PD patients showed a statistically more reliable BOLD modulation than healthy participants in the posterior cingulate gyrus, the parahippocampal gyrus, inferior parietal lobule, precuneus and in the middle temporal gyrus. Given abnormal frontal and normal PEF responses, our results suggest that in PD a frontal cortical circuitry, known to be associated with saccade planning, selection, and predicting a metric error of the saccade, is deficient.

Predicting the recognition of natural scenes from single trial MEG recordings of brain activity

In our daily life we look at many scenes. Some are rapidly forgotten, but others we recognize later. We accurately predicted recognition success with natural scene photographs using single trial magnetoencephalography (MEG) measures of brain activation. Specifically, we demonstrate that MEG responses in the initial 600 ms following the onset of scene photographs allow for prediction accuracy rates up to 84.1% using linear Support-Vector-Machine classification (lSVM). A permutation test confirmed that all lSVM based prediction rates were significantly better than "guessing". More generally, we present four approaches to analyzing brain function using lSVMs. (1) We show that lSVMs can be used to extract spatio-temporal patterns of brain activation from MEG-data. (2) We show lSVM classification can demonstrate significant correlations between comparatively early and late processes predictive of scene recognition, indicating dependencies between these processes over time. (3) We use lSVM classification to compare the information content of oscillatory and event-related MEG-activations and show they contain a similar amount of and largely overlapping information. (4) A more detailed analysis of single-trial predictiveness of different frequency bands revealed that theta band activity around 5 Hz allowed for highest prediction rates, and these rates are indistinguishable from those obtained with a full dataset. In sum our results clearly demonstrate that lSVMs can reliably predict natural scene recognition from single trial MEG-activation measures and can be a useful tool for analyzing predictive brain function.

Analysis of a choice-reaction task yields a new interpretation of Libet's experiments

Benjamin Libet has demonstrated that the readiness potential precedes the time at which participants consciously decide to perform an intentional motor act, and suggested that free will is an illusion. We performed an experiment where participants observed a stimulus on a computer monitor and were instructed to press one of two buttons, depending on the presented stimulus. We found neural activity preceding the motor response, similar to Libet's experiments. However, this activity was already present prior to stimulus presentation, and thus before participants could decide which button to press. Therefore, we argue that this activity does not specifically determine behaviour. Instead, it may reflect a general expectation. This interpretation would not interfere with the notion of free will.

Stimulus intensity affects early sensory processing: Visual contrast modulates evoked gamma-band activity in human EEG

We studied the effect of different contrast levels on the visual evoked gamma-band response (GBR) in order to investigate whether the GBR is modulated in a similar manner as previously reported for visual evoked potentials. Previous studies showed that the GBR can be modulated by individual characteristics (age) and experimental conditions (task difficulty, attention). However, stimulus properties, such as size and spatial frequency, also have a large impact on the GBR, which necessitates identification and control of relevant stimulus properties for optimal experimental setups. Twenty-one healthy participants were investigated during a forced-choice discrimination task. Sinusoidal gratings were presented at three contrast levels with a constant spatial frequency of 5 cycles per degree visual arc (cpd). The present data replicate the results reported for visual evoked potentials and exhibit a contrast dependent modulation of the GBR. Gamma activity is increased for higher contrast levels. These results demonstrate the importance of stimulus contrast for evoked gamma activity. Thus, it appears meaningful to control the contrast of stimuli in experiments investigating the role of gamma activity in perception and information processing.

The appearance of figures seen through a narrow aperture under free viewing conditions: effects of spontaneous eye motions

When moving figures are occluded and revealed piecemeal as they move across a narrow slit, observers may perceive them as integrated but distorted. They may also perceive much more of the figure as simultaneously visible than is actually presented at any moment. We obtained quantitative measures of both the perceived distortion and perceived simultaneity under free viewing conditions and related these phenomena to spontaneous pursuit eye movements, the retinal painting produced by this pursuit, and the occurrence of saccades. We found both shape compressions and expansions, depending on figure velocity. We also obtained quantitative evidence that observers perceived slices of the moving figures far wider than the slit through which they were presented. Eye-motion records and retinal stabilization revealed that spontaneous pursuit and the spatially extended images that could have been painted out by this pursuit played no role in the perceived global shape distortions and made only a small contribution to the perceived simultaneity. Therefore, under free viewing conditions, both the distortions and simultaneity of these "anorthoscopic" figure percepts must be the consequence of a postretinal process that integrates the figures over space and time independent of eye motions.

Endoscopic eye tracking system for fMRI

Here we introduce a new video-based real-time eye tracking system suitable for functional magnetic resonance imaging (fMRI) application. The described system monitors the subject's eye, which is illuminated with infrared light, directly at the headcoil using an endoscopic fibre optical system. This endoscopic technique assures reliable, easy-to-use and fast adjustment. It requires only a minimal amount of equipment at the headcoil and inside the examination room. Moreover, the short distance between the image acquisition optics and the eye provides high spatial tracking resolution. Interference from physiological head movement is effectively reduced by simultaneous tracking of both eye and head movements.

The effect of retinal stabilization on anorthoscopic percepts under free-viewing conditions

Outline shapes were moved back and forth at .5 Hz behind a simulated narrow (10' or 15') slit without visible borders. Under free-viewing conditions observers reported spontaneous transitions between periods in which they saw a horizontally moving shape and periods in which they saw only vertically moving contour segments. Eye movements were monitored with a dual-Purkinje image eye-tracker. On half the trials we used retinal stabilization to eliminate any pursuit dependent retinal painting. With no stabilization, we observed low amplitude spontaneous pursuit of the horizontal figure motion when a shape was seen, and occasionally lower amplitude pursuit even when no shape was seen. Pursuit amplitudes increased when the display was stabilized. However, stabilization did not alter either the duration or frequency of the episodes of shape perception. Our findings indicate that under the free-viewing conditions we employed, spontaneous smooth pursuit and pursuit dependent painting did not contribute to the formation or maintenance of anorthoscopic percepts.

The dynamics of visual pattern masking in natural scene processing: a magnetoencephalography study

We investigated the dynamics of natural scene processing and mechanisms of pattern masking in a scene-recognition task. Psychophysical recognition performance and the magnetoencephalogram (MEG) were recorded simultaneously. Photographs of natural scenes were briefly displayed and in the masked condition immediately followed by a pattern mask. Viewing the scenes without masking elicited a transient occipital activation that started approximately 70 ms after the pattern onset, peaked at 110 ms, and ended after 170 ms. When a mask followed the target an additional transient could be reliably identified in the MEG traces. We assessed psychophysical performance levels at different latencies of this transient. Recognition rates were reduced only when the additional activation produced by the pattern mask overlapped with the initial 170 ms of occipital activation from the target. Our results are commensurate with an early cortical locus of pattern masking and indicate that 90 ms of undistorted cortical processing is necessary to reliably recognize a scene. Our data also indicate that as little as 20 ms of undistorted processing is sufficient for above-chance discrimination of a scene from a distracter.

Functional measurements of human ventral occipital cortex: retinotopy and colour

Human colour vision originates in the cone photoreceptors, whose spatial density peaks in the fovea and declines rapidly into the periphery. For this reason, one expects to find a large representation of the cone-rich fovea in those cortical locations that support colour perception. Human occipital cortex contains several distinct foveal representations including at least two that extend onto the ventral surface: a region thought to be critical for colour vision. To learn more about these ventral signals, we used functional magnetic resonance imaging to identify visual field maps and colour responsivity on the ventral surface. We found a visual map of the complete contralateral hemifield in a 4 cm(2) region adjacent to ventral V3; the foveal representation of this map is confluent with that of areas V1/2/3. Additionally, a distinct foveal representation is present on the ventral surface situated 3-5 cm anterior from the confluent V1/2/3 foveal representations. This organization is not consistent with the definition of area V8, which assumes the presence of a quarter field representation adjacent to V3v. Comparisons of responses to luminance-matched coloured and achromatic patterns show increased activity to the coloured stimuli beginning in area V1 and extending through the new hemifield representation and further anterior in the ventral occipital lobe.