Holistic but with reduced right-hemisphere involvement: The case of dyslexia in Chinese character recognition

Understanding Human Cognition Through Computational Modeling

One important goal of cognitive science is to understand the mind in terms of its representational and computational capacities, where computational modeling plays an essential role in providing theoretical explanations and predictions of human behavior and mental phenomena. In my research, I have been using computational modeling, together with behavioral experiments and cognitive neuroscience methods, to investigate the information processing mechanisms underlying learning and visual cognition in terms of perceptual representation and attention strategy. In perceptual representation, I have used neural network models to understand how the split architecture in the human visual system influences visual cognition, and to examine perceptual representation development as the results of expertise. In attention strategy, I have developed the Eye Movement analysis with Hidden Markov Models method for quantifying eye movement pattern and consistency using both spatial and temporal information, which has led to novel findings across disciplines not discoverable using traditional methods. By integrating it with deep neural networks (DNN), I have developed DNN+HMM to account for eye movement strategy learning in human visual cognition. The understanding of the human mind through computational modeling also facilitates research on artificial intelligence's (AI) comparability with human cognition, which can in turn help explainable AI systems infer humans' belief on AI's operations and provide human-centered explanations to enhance human-AI interaction and mutual understanding. Together, these demonstrate the essential role of computational modeling methods in providing theoretical accounts of the human mind as well as its interaction with its environment and AI systems.

Atypical reliance on monocular visual pathway for face and word recognition in developmental dyslexia

Studies with individuals with developmental dyslexia (DD) have documented impaired perception of words and faces, both of which are domains of visual expertise for human adults. In this study, we examined a possible mechanism that might be associated with the impaired acquisition of visual expertise for words and faces in DD, namely, the atypical engagement of the monocular visual pathway. Participants with DD and typical readers (TR) judged whether a pair of sequentially presented unfamiliar faces or nonwords were the same or different, and the pair of stimuli were displayed in an eye-specific fashion using a stereoscope. Based on evidence of greater reliance on subcortical structures early in development, we predicted differences between the groups in the engagement of lower (monocular) versus higher (binocular) regions of the visual pathways. Whereas the TR group showed a monocular advantage for both stimulus types, the DD participants evinced a monocular advantage for faces and words that was much greater than that measured in the TRs. These findings indicate that the DD individuals have enhanced subcortical engagement and that this might arise from the failure to fine-tune cortical correlates mediating the discrimination of homogeneous exemplars in domains of expertise.

The Role of Visual Factors in Dyslexia

What are the causes of dyslexia? Decades of research reflect a determined search for a single cause where a common assumption is that dyslexia is a consequence of problems with converting phonological information into lexical codes. But reading is a highly complex activity requiring many well-functioning mechanisms, and several different visual problems have been documented in dyslexic readers. We critically review evidence from various sources for the role of visual factors in dyslexia, from magnocellular dysfunction through accounts based on abnormal eye movements and attentional processing, to recent proposals that problems with high-level vision contribute to dyslexia. We believe that the role of visual problems in dyslexia has been underestimated in the literature, to the detriment of the understanding and treatment of the disorder. We propose that rather than focusing on a single core cause, the role of visual factors in dyslexia fits well with risk and resilience models that assume that several variables interact throughout prenatal and postnatal development to either promote or hinder efficient reading.

Word reading transfer in two distinct languages in reading interventions: How Chinese-English bilingual children with reading difficulties learn to read

PURPOSE

Skills developed from literacy training in L1 are shown to transfer to reading in L2 when both languages involve an alphabetic writing system. However, transfer of literacy skills between a logographic L1 and an alphabetic L2 is less studied. This study examined whether the gain in literacy skills after an 8-week training on 1) Chinese character recognition or 2) English phonics, may generalize across the two languages in Chinese elementary students with reading disabilities.

METHODS

Chinese-speaking students identified with reading difficulties were randomly assigned to the Chinese intervention (Chinese character orthography training), English intervention (English phonics training), and control groups. Their Chinese and English literacy skills were measured before and after the interventions.

RESULTS

Though training on the orthography of Chinese characters significantly improved performance in Chinese word reading and Chinese orthographic awareness, our results did not provide evidence for the generalization of word-decoding skills from L1 Chinese to word reading in L2 English. However, phonics training in L2 English benefitted not only English word reading, but also cross-language word reading in L1 Chinese.

CONCLUSION

We postulated that teaching children analytical skills in decoding words in an alphabetic writing system might likewise benefit their word decoding in a logographic script.

How does face mask in COVID-19 pandemic disrupt face learning and recognition in adults with autism spectrum disorder?

Use of face masks is one of the measures adopted by the general community to stop the transmission of disease during this ongoing COVID-19 pandemic. This wide use of face masks has indeed been shown to disrupt day-to-day face recognition. People with autism spectrum disorder (ASD) often have predisposed impairment in face recognition and are expected to be more vulnerable to this disruption in face recognition. Here, we recruited typically developing adult participants and those with ASD, and we measured their non-verbal intelligence, autism spectrum quotient, empathy quotient, and recognition performances of faces with and without a face mask covering the lower halves of the face. When faces were initially learned unobstructed, we showed that participants had a general reduced face recognition performance for masked faces. In contrast, when masked faces were first learned, typically developing adults benefit with an overall advantage in recognizing both masked and unmasked faces; while adults with ASD recognized unmasked faces with a significantly more reduced level of performance than masked faces—this face recognition discrepancy is predicted by a higher level of autistic traits. This paper also discusses how autistic traits influence processing of faces with and without face masks.

Electrophysiological measurements of holistic processing of Chinese characters

Holistic processing (HP) is a marker of perceptual expertise in facial recognition. In the present study, we examined neural responses to the HP of Chinese characters, adopting the composite paradigm. The behavioral results showed that the discrimination of congruent trials was significantly higher than that of incongruent trials, and participants responded faster. Moreover, the congruent trials elicited significantly larger N170 amplitude than the incongruent trials. The HP effect of the N170 component was observed for upright characters, as the configural information of inverted characters and misaligned characters were destroyed. Right-lateralization of processing Chinese characters was observed in the N170 amplitudes and delta-theta band oscillations. The results suggested that Chinese character recognition employed a strategy of HP, and the finding that neural indicators provide a better signal of the strength of HP in Chinese characters than behavioral indicators was also crucial.

Developmental trajectories of expert perception processing of Chinese characters in primary school children

Previous studies have demonstrated that inversion effect and left-side bias are stable expertise markers in Chinese character processing among adults. However, it is less clear how these markers develop early on (i.e., among primary school students). Therefore, this study aimed to investigate the development of the two markers by comparing primary school-aged students of three age groups (Grade 1, Grade 3, and Grade 5) and adults in tests of inversion effect (Experiment 1) and left-sided bias effect (Experiment 2). The results replicated that both effects during Chinese character processing were present among adults. However, more importantly, the effects were different among primary school-aged students in different grades: the inversion effect was found as early as in Grade 1, but the left-side bias effect did not emerge in Grade 1 and as approximated that of adults until Grade 3. The study suggested a potential dissociation in developing different aspects of expertise during Chinese character processing in early childhood.

Music reading experience modulates eye movement pattern in English reading but not in Chinese reading

Here we tested the hypothesis that in Chinese-English bilinguals, music reading experience may modulate eye movement planning in reading English but not Chinese sentences due to the similarity in perceptual demands on processing sequential symbol strings separated by spaces between music notation and English sentence reading. Chinese–English bilingual musicians and non-musicians read legal, semantically incorrect, and syntactically (and semantically) incorrect sentences in both English and Chinese. In English reading, musicians showed more dispersed eye movement patterns in reading syntactically incorrect sentences than legal sentences, whereas non-musicians did not. This effect was not observed in Chinese reading. Musicians also had shorter saccade lengths when viewing syntactically incorrect than correct musical notations and sentences in an unfamiliar alphabetic language (Tibetan), whereas non-musicians did not. Thus, musicians’ eye movement planning was disturbed by syntactic violations in both music and English reading but not in Chinese reading, and this effect was generalized to an unfamiliar alphabetic language. These results suggested that music reading experience may modulate perceptual processes in reading differentially in bilinguals’ two languages, depending on their processing similarities.

Non-monotonic developmental trend of holistic processing in visual expertise: the case of Chinese character recognition

Holistic processing has been identified as an expertise marker of face and object recognition. By contrast, reduced holistic processing is purportedly an expertise marker in recognising orthographic characters in Chinese. Does holistic processing increase or decrease in expertise development? Is orthographic recognition a domain-specific exception to all other kinds of recognition (e.g. face and objects)? In two studies, we examined the developmental trend of holistic processing in Chinese character recognition in Chinese and non-Chinese children, and its relationship with literacy abilities: Chinese first graders—with emergent Chinese literacy acquired in kindergarten—showed increased holistic processing perhaps as an inchoate expertise marker when compared with kindergartners and non-Chinese first graders; however, the holistic processing effect was reduced in higher-grade Chinese children. These results suggest a non-monotonic inverted U-shape trend of holistic processing in visual expertise development: An increase in holistic processing due to initial reading experience followed by a decrease in holistic processing due to literacy enhancement. This result marks the development of holistic and analytic processing skills, both of which can be essential for mastering visual recognition. This study is the first to investigate the developmental trend of holistic processing in Chinese character recognition using the composite paradigm.

Faces and words are both associated and dissociated as evidenced by visual problems in dyslexia

Faces and words are traditionally assumed to be independently processed. Dyslexia is also traditionally thought to be a non-visual deficit. Counter to both ideas, face perception deficits in dyslexia have been reported. Others report no such deficits. We sought to resolve this discrepancy. 60 adults participated in the study (24 dyslexic, 36 typical readers). Feature-based processing and configural or global form processing of faces was measured with a face matching task. Opposite laterality effects in these tasks, dependent on left-right orientation of faces, supported that they tapped into separable visual mechanisms. Dyslexic readers tended to be poorer than typical readers at feature-based face matching while no differences were found for global form face matching. We conclude that word and face perception are associated when the latter requires the processing of visual features of a face, while processing the global form of faces apparently shares minimal-if any-resources with visual word processing. The current results indicate that visual word and face processing are both associated and dissociated-but this depends on what visual mechanisms are task-relevant. We suggest that reading deficits could stem from multiple factors, and that one such factor is a problem with feature-based processing of visual objects.

Music-reading expertise associates with face but not Chinese character processing ability

A growing number of behavioural and neuroimaging studies have investigated the cognitive mechanisms and neural substrates underlying various forms of visual expertise, such as face and word processing. However, it remains poorly understood whether and to what extent the acquisition of one form of expertise would be associated with that of another. The current study examined the relationship between music-reading expertise and face and Chinese character processing abilities. In a series of experiments, music experts and novices performed discrimination and recognition tasks of musical notations, faces, and words. Results consistently showed that musical experts responded more accurately to musical notations and faces, but not to words, than did musical novices. More intriguingly, the music expert's age of training onset could well predict their face but not word processing performance: the earlier musical experts began musical notation reading, the better their face processing performance. Taken together, our findings provide preliminary and converging evidence that music-reading expertise links with face, but not word, processing, and lend support to the notion that the development of different types of visual expertise may not be independent, but rather interact with each other during their acquisition.

Eye movement analysis with hidden Markov models (EMHMM) with co-clustering

The eye movement analysis with hidden Markov models (EMHMM) method provides quantitative measures of individual differences in eye-movement pattern. However, it is limited to tasks where stimuli have the same feature layout (e.g., faces). Here we proposed to combine EMHMM with the data mining technique co-clustering to discover participant groups with consistent eye-movement patterns across stimuli for tasks involving stimuli with different feature layouts. Through applying this method to eye movements in scene perception, we discovered explorative (switching between the foreground and background information or different regions of interest) and focused (mainly looking at the foreground with less switching) eye-movement patterns among Asian participants. Higher similarity to the explorative pattern predicted better foreground object recognition performance, whereas higher similarity to the focused pattern was associated with better feature integration in the flanker task. These results have important implications for using eye tracking as a window into individual differences in cognitive abilities and styles. Thus, EMHMM with co-clustering provides quantitative assessments on eye-movement patterns across stimuli and tasks. It can be applied to many other real-life visual tasks, making a significant impact on the use of eye tracking to study cognitive behavior across disciplines.