Main concepts for two picture description tasks: An addition to

Large Language Models' Ability to Assess Main Concepts in Story Retelling: A Proof-of-Concept Comparison of Human Versus Machine Ratings

PURPOSE

Despite an abundance of manual, labor-intensive discourse analysis methods, there remains a dearth of clinically convenient, psychometrically robust instruments to measure change in real-world communication in aphasia. The Brief Assessment of Transactional Success (BATS) addresses this gap while developing automated methods for analyzing story retelling discourse. This study investigated automation of main concept (MC) analysis of stories by comparing scores from three large language models (LLMs) to those of human raters.

METHOD

After watching/listening to each of the eight short video/audio BATS stimuli and retelling each story, 96 persons with aphasia (PWA; n = 48 female) engaged in topic-constrained conversations over Zoom with 94 familiar and 107 unfamiliar conversation partners (CPs). CPs then retold each story as co-constructed during their conversations with PWA. Audio files from the resulting 1,760 story retells were transcribed using Python and AssemblyAI's speech-to-text application programming interface. Each MC was first scored by human raters for presence, accuracy, and completeness. Raters used a semiautomated application, MainConcept. For each transcript, an MC composite ratio score was obtained. We evaluated three state-of-the-art LLMs: two proprietary models, GPT-4 and GPT-4o, and one open-source model, Llama-3-70B. The interrater reliability between each LLM versus human MC scoring was assessed via the Pearson correlation coefficient and reliability coefficients based on the generalizability theory (G-theory).

RESULTS

The Pearson correlation coefficients indicate strong positive linear relationships between LLM and human MC scores. G-theory reliability coefficients also indicate reliable scoring between LLM and human scoring across the spectrum of participants and conditions.

CONCLUSIONS

This promising proof-of-concept study affirms the reliability of three LLMs in evaluating BATS story retell MCs and justifies ongoing investigation into their use. Providing clinicians and clinical researchers with automated tools for analyzing discourse without the need for prohibitively labor-intensive manual scoring could be a paradigm shift, potentially revolutionizing the aphasia intervention landscape.

Age-related changes in connected speech production: evidence from eye-tracking in the culturally adapted picture description task

Purpose

Age-related changes in connected speech production remain a subject of debate, yielding inconsistent findings across various tasks and measures. This study aimed to investigate the effects of aging on picture description tasks using two types of pictures: a standardized picture (the Beach picture) and a culturally and linguistically modified picture tailored for Korean speakers (the Han River picture).

Method

Twenty-four young adults and 22 older adults participated in two picture description tasks while their eye movements were recorded. Word-level linguistic variables were used to assess informativeness (Correct Information Units per minute) and productivity (noun and verb counts per utterance) of connected speech production. Eye-movement measures were employed to evaluate real-time cognitive processing associated with planning connected speech (pre-speech fixation counts and durations; eye fixations before the speech onset of each utterance).

Results and conclusions

The findings revealed age-related declines in linguistic measures, with older adults exhibiting decreased CIUs per minute and smaller counts of nouns and verbs per utterance. Age-related changes in eye movement measures were evident in that older adults displayed longer pre-speech fixation durations. Unlike younger adults, older adults exhibited higher pre-speech fixation counts on the Han River picture compared to the Beach picture, suggesting cognitive challenges in performing the task that requires producing more words and detailed descriptions. These results suggest that aging is associated with reduced informativeness and productivity of connected speech, as well as a decline in cognitive processing efficiency.

The Relationship Between Executive Functioning and Narrative Language Abilities in Aphasia

PURPOSE

Although individuals with aphasia commonly exhibit challenges in executive functioning (EF) and spoken discourse, there is limited research exploring connections between these abilities within this specific population. Therefore, this study investigated the relationship between verbal and nonverbal EF and narrative productions in aphasia using a multilevel linguistic approach.

METHOD

Participants included 22 persons with aphasia (PWA) and 24 age- and education-matched, neurologically healthy controls (NHC). All participants completed assessments for EF and a story retelling task. Narrative samples were analyzed for microlinguistic (productivity, lexical and syntactic features, semantic content, word and sentence errors) and macrolinguistic (coherence, informational content, organization, and language use) characteristics. Correlational analyses were employed to explore the relationships among narrative variables. EF factors, extracted from principal component analysis, were used as predictive variables in hierarchical stepwise regression analyses to evaluate their role in predicting narrative performances of PWA and NHC.

RESULTS

Relative to NHC, PWA exhibited impaired narrative performance affecting both microlinguistic and macrolinguistic levels. Breakdowns at the structural level (i.e., reduced productivity, syntax, lexical retrieval, and diversity) correlated with impaired story completeness, organization, and connectedness; this relationship was more prominent for PWA. Three EF factors representing (1) verbal EF, (2) verbal and nonverbal fluency, and (3) nonverbal EF were extracted. Factors 1 and 2 largely predicted narrative performance, whereas Factor 3 (i.e., nonverbal EF) contributed prominently to predicting macrolinguistic discourse performance in both groups although accounting for less variance in the data. Overall, lower EF scores, particularly verbal EF variables, predicted poor narrative performance in both groups.

CONCLUSIONS

Our results indicate that both linguistic and extralinguistic cognitive abilities play a role in story retelling performances among PWA. Notably, both verbal and nonverbal EF skills were found to be correlated with narrative abilities. However, the extent of their contributions varied depending on the discourse levels assessed. These findings provide a significant contribution to our understanding of the cognitive factors associated with breakdowns in discourse among PWA and highlight the importance of comprehensive assessment of EF and discourse within this population.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.26485627.

Reliability of the Brief Assessment of Transactional Success in Communication in Aphasia

Background

While many measures exist for assessing discourse in aphasia, manual transcription, editing, and scoring are prohibitively labor intensive, a major obstacle to their widespread use by clinicians (Bryant et al., 2017; Cruice et al., 2020). Many tools also lack rigorous psychometric evidence of reliability and validity (Azios et al., 2022; Carragher et al., 2023). Establishing test reliability is the first step in our long-term goal of automating the Brief Assessment of Transactional Success in aphasia (BATS; Kurland et al., 2021) and making it accessible to clinicians and clinical researchers.

Aims

We evaluated multiple aspects of test reliability of the BATS by examining correlations between human/machine and human/human interrater edited transcripts, raw vs. edited transcripts, interrater scoring of main concepts, and test-retest performance. We hypothesized that automated methods of transcription and discourse analysis would demonstrate sufficient reliability to move forward with test development.

Methods & Procedures

We examined 576 story retelling narratives from a sample of 24 persons with aphasia and familiar and unfamiliar conversation partners (CP). Participants with aphasia (PWA) retold stories immediately after watching/listening to short video/audio clips. CP retold stories after six-minute topic-constrained conversations with a PWA in which the dyad co-constructed the stories. We utilized two macrostructural measures to analyze the automated speech-to-text transcripts of story retells: 1) a modified version of a semi-automated tool for measuring main concepts (mainConcept: Cavanaugh et al., 2021); and 2) an automated natural language processing 'pipeline' to assess topic similarity.

Outcomes & Results

Correlations between raw and edited scores were excellent, interrater reliability on transcripts and main concept scoring were acceptable. Test-retest on repeated stimuli was acceptable. This was especially true of aphasic story retellings where there were actual within subject repeated stimuli.

Conclusions

Results suggest that automated speech-to-text was generally sufficient in most cases to avoid the time-consuming, labor intensive step of transcribing and editing discourse. Overall, our study results suggest that natural language processing automated methods such as text vectorization and cosine similarity are a fast, efficient way to obtain a measure of topic similarity between two discourse samples. Although test-retest reliability for the semi-automated mainConcept method was generally higher than for automated methods of measuring topic similarity, we found no evidence of a difference between machine automated and human-reliant scoring.

How do speech-language pathologists assess and treat spoken discourse after TBI? A survey of clinical practice

BACKGROUND

Spoken discourse is commonly affected after traumatic brain injury (TBI). Although guidelines recommend prioritizing discourse-level skills in cognitive communication management, previous literature has highlighted challenges in managing discourse clinically. Little is known about how speech-language pathologists (SLPs) assess and treat discourse after TBI.

AIMS

To investigate current SLP practice to determine the alignment of clinical practice with research evidence and recommendations.

METHODS & PROCEDURES

This online survey consisted of 30 questions on SLPs' practice with discourse assessment, analysis and treatment processes, including the materials and methods used and rationales for decision-making. Participants were recruited through national and international SLP professional bodies, TBI-specific or SLP special-interest groups and social media. Survey responses were analysed using descriptive statistics, with free text included to support individual responses.

OUTCOMES & RESULTS

There were 70 participants, from Australia, the United States, UK and New Zealand. Nearly half the participants had over 11 years of experience working with adults with TBI and a quarter had over 20 years of experience. Participants reported that they regularly evaluated the discourse ability of people with TBI, most commonly during spontaneous conversation or with a personal narrative task. Discourse intervention approaches mostly targeted client self-monitoring ability, social skills or conversational interactions. Practice varied dependent on setting, with more SLPs in community or outpatient services undertaking discourse assessment and treatment than in hospital settings.

CONCLUSIONS & IMPLICATIONS

Overall, survey respondents' management of spoken discourse aligned with recommendations in the research literature, incorporating an individualized, goal-based approach. Factors affecting the use of discourse in practice included client-specific factors and needs, availability of time for transcription and analysis, and SLPs' knowledge level and confidence with discourse. Increased knowledge of discourse methods and treatment approaches could help inform decision-making for SLPs working in TBI.

WHAT THIS PAPER ADDS

What is already known on this subject Spoken discourse is one of the most affected areas of communication for people with TBI. Although recent research has provided guidance on assessment and treatment options for TBI discourse, it is unknown how SLPs manage spoken discourse clinically. What this paper adds to existing knowledge This research adds to the limited research on how SLPs across clinical settings and regions manage discourse assessment and treatment. Respondents' practice was generally aligned with recommendations, featuring individualized, goal-based practice. Potential barriers to discourse implementation included client factors and SLP knowledge, training, and service delivery factors. What are the potential or actual clinical implications of this work? This survey provides evidence that clinical translation is starting to occur in this field, but there remains a need for increased knowledge of assessment and treatment methods, training, and TBI-specific resources to better support SLP practice, particularly relating to transcription. Core reporting outcomes in research on TBI discourse management would assist with translation of the findings to practice.

Adaptation and Reliability of the Cinderella Story Retell Task in Canadian French Persons Without Brain Injury

PURPOSE

Main concept (MC) analysis is a well-documented method of discourse analysis in adults with and without brain injury. This study aims to develop a MC checklist that is culturally and linguistically adapted for Canadian French speakers and examine its reliability. We also documented microstructural properties and provide a normative reference in persons not brain injured (PNBIs).

METHOD

Discourse samples from 43 PNBIs were collected. All participants completed the Cinderella story retell task twice. Manual transcription was performed for all samples. The 34 MCs for the Cinderella story retell task were adapted into Canadian French and used to score all transcripts. In addition, microstructural variables were extracted using Computerized Language Analysis (CLAN). Intraclass correlation coefficients were computed to assess interrater reliability for MC codes and microstructural variables. Test-retest reliability was assessed using intraclass correlations, Spearman's rho correlations, and the Wilcoxon signed-ranks test. Bland-Altman plots were used to examine the agreement of the discourse measures between the two sessions.

RESULTS

The MC checklist for the Cinderella story retell task adapted for Canadian French speakers is provided. Good-to-excellent interrater reliability was obtained for most MC codes; however, reliability ranged from poor to excellent for the "inaccurate and incomplete" code. Microstructural variables demonstrated excellent interrater reliability. Test-retest reliability ranged from poor to excellent for all variables, with the majority falling between moderate and excellent. Bland-Altman plots illustrated the limits of agreement between test and retest.

CONCLUSIONS

This study provides the MC checklist for clinicians and researchers working with Canadian French speakers when assessing discourse with the Cinderella story retell task. It also addresses the gap in available psychometric data regarding test-retest reliability in PNBIs.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.24171087.

Applying core lexicon analysis in patients with anomic aphasia: Based on Mandarin AphasiaBank

BACKGROUND

Patients with anomic aphasia experience difficulties in narrative processing. General discourse measures are time consuming and require necessary skills. Core lexicon analysis has been proposed as an effort-saving approach but has not been developed in Mandarin discourse.

AIMS

This exploratory study was aimed (1) to apply core lexicon analysis in Mandarin patients with anomic aphasia at the discourse level and (2) to verify the problems with core words among people with anomic aphasia.

METHODS & PROCEDURE

The core nouns and verbs were extracted from narrative language samples from 88 healthy participants. The production of core words for 12 anomic aphasia and 12 age- and education-matched controls were then calculated and compared. The correlation between the percentages and the Aphasia Quotients of the revised Western Aphasia Battery was analyzed as well.

OUTCOMES & RESULTS

The core nouns and verbs were successfully extracted. Patients with anomic aphasia produced fewer core words than healthy people, and the percentages differed significantly in different tasks as well as word classes. There was no correlation between the core lexicon use and the severity of aphasia in patients with anomic aphasia.

CONCLUSIONS & IMPLICATIONS

Core lexicon analysis may potentially serve as a clinician-friendly manner of quantifying core words produced at the discourse level in Mandarin patients with anomic aphasia.

WHAT THIS PAPER ADDS

What is already known on the subject Discourse analyses in aphasia assessment and treatment have increasingly garnered attention. Core lexicon analysis based on English AphasiaBank has been reported in recent years. It is correlated with microlinguistic and macrolinguistic measures in aphasia narratives. Nevertheless, the application based on Mandarin AphasiaBank is still under development in healthy individuals and patients with anomic aphasia. What this paper adds to existing knowledge A Mandarin core lexicon set was developed for different tasks. The feasibility of core lexicon analysis to evaluate the corpus of patients with anomic aphasia was preliminarily discussed and the speech performance of patients and healthy people was then compared to provide a reference for the evaluation and treatment of clinical aphasia corpus. What are the potential or actual clinical implications of this work? The purpose of this exploratory study was to consider the potential use of core lexicon analysis to evaluate core word production in narrative discourse. Moreover, normative and aphasia data were provided for comparison to develop clinical use for Mandarin patients with anomic aphasia.

Clinical Guidelines for Eliciting Discourse Using the RHDBank Protocol

PURPOSE

Examining discourse after right hemisphere brain damage (RHD) can provide invaluable clinical data and insight into functional communication capabilities. Yet, clinicians preparing to enter the field may have limited experience eliciting and analyzing discourse for therapeutic purposes. The purpose of this work is to present a practical guide for the clinical use of the RHDBank protocol.

METHOD

Following a brief overview of the RHDBank protocol, elicitation guidelines and protocol administration considerations are offered. Measures that should be considered when analyzing RHDBank-elicited discourse are summarized. The RHDBank protocol is positioned as an accessible resource for discourse elicitation in the clinical setting, and examples for the clinical use of the protocol are provided.

CONCLUSIONS

Discourse production can be a key source of information regarding communication ability following RHD. The RHDBank provides a structured and scripted approach to the clinical elicitation of language and consideration of discourse production. Its immediate clinical utility will assist developing clinicians with the acquisition of a clinically relevant elicitation approach and more comprehensive view of the RHD communication impairments. Application of the protocol can aid in collective knowledge advancement that will promote a better understanding of RHD language production as a clinical entity with an array of possible characteristics.

Adaptation and validation of the main concept analysis of spoken discourse by native Japanese adults

Many people with aphasia demonstrate problems of oral production at the discourse level. The Main Concept Analysis (MCA) for oral discourse production is a published evidence-based battery for quantifying the degree of presence, accuracy, completeness, and efficiency of targeted main concepts in oral discourse. In Japan, such a standardized tool specialized for assessing spoken discourse is currently lacking. The purpose of this study was to adapt the Japanese version of MCA for oral discourse production (the Japanese-MCA) and examine its validity and reliability. Stage 1 of the study involved the establishment of linguistically-specific main concepts (MCs) of the Japanese-MCA. Ten speech-language-hearing therapists and 60 healthy participants who were native monolingual Japanese speakers were recruited to determine MCs. Stage 2 examined the criterion validity and reliability of the Japanese-MCA. Language samples of 20 participants with aphasia, as verified by Standard Language Test of Aphasia (SLTA), and 20 healthy older participants were used. Results of Stage 1 of the study yielded normative data with a set of target MCs that were geographically and linguistically specific for use in Japan. The results also revealed the comparability of the Japanese-MCA and previously reported versions of other languages. Stage 2 findings indicated not only a high correlation of criterion validity, but also good reliability of the test. With established norms and specific scoring criteria of the Japanese-MCA, it is believed that this new tool will become a useful addition to clinical management and research of aphasia in Japan.

Assessment of language impairment and function

This chapter is written for the qualified neurologist or related professional working with persons who have had a stroke or other sudden brain injury. It is critical that the presence of aphasia is detected, no matter how mild the presentation, and to support that assertion, this chapter highlights the plight of persons with latent aphasia. At the individual level, the impact of aphasia is devastating, with overwhelming evidence that aphasia negatively impacts psychosocial outcomes. At the global level, sensitive detection and accurate diagnosis of aphasia are critical for accurate characterization and quantification of the global burden of aphasia. The word "LANGUAGE" is leveraged as an acronym to create a useful and memorable checklist to guide navigation of aphasia screening and assessment: it begins with the definition of language (L), followed by the definition and diagnostic criteria for aphasia (A). Then language abilities and characteristics to be considered in assessment are presented: naming (N); grammar and syntax (G); unintelligible words, jargon, and paraphasias (U); auditory comprehension and repetition (A); graphemic abilities-reading and writing (G); and everyday communication and discourse (E). Recommendations for improving procedural adherence are provided, and a list of potential brief assessment measures are introduced.

Spectral Resting-State EEG (rsEEG) in Chronic Aphasia Is Reliable, Sensitive, and Correlates With Functional Behavior

We investigated spectral resting-state EEG in persons with chronic stroke-induced aphasia to determine its reliability, sensitivity, and relationship to functional behaviors. Resting-state EEG has not yet been characterized in this population and was selected given the demonstrated potential of resting-state investigations using other neuroimaging techniques to guide clinical decision-making. Controls and persons with chronic stroke-induced aphasia completed two EEG recording sessions, separated by approximately 1 month, as well as behavioral assessments of language, sensorimotor, and cognitive domains. Power in the classic frequency bands (delta, theta, alpha, and beta) was examined via spectral analysis of resting-state EEG data. Results suggest that power in the theta, alpha, and beta bands is reliable for use as a repeated measure. Significantly greater theta and lower beta power was observed in persons with aphasia (PWAs) than controls. Finally, in PWAs theta power negatively correlated with performance on a discourse informativeness measure, while alpha and beta power positively correlated with performance on the same measure. This indicates that spectral rsEEG slowing observed in PWAs in the chronic stage is pathological and suggests a possible avenue for directly altering brain activation to improve behavioral function. Taken together, these results suggest that spectral resting-state EEG holds promise for sensitive measurement of functioning and change in persons with chronic aphasia. Future studies investigating the utility of these measures as biomarkers of frank or latent aphasic deficits and treatment response in chronic stroke-induced aphasia are warranted.

Main Concept, Sequencing, and Story Grammar Analyses of Cinderella Narratives in a Large Sample of Persons with Aphasia

Recently, a multilevel analytic approach called Main Concept, Sequencing, and Story Grammar (MSSG) was presented along with preliminary normative information. MSSG analyses leverage the strong psychometrics and rich procedural knowledge of both main concept analysis and story grammar component coding, complementing it with easy-to-obtain sequencing information for a rich understanding of discourse informativeness and macrostructure. This study is the next critical step for demonstrating the clinical usefulness of MSSG’s six variables (main concept composite, sequencing, main concept+sequencing, essential story grammar components, total episodic components, and episodic complexity) for persons with aphasia (PWAs). We present descriptive statistical information for MSSG variables for a large sample of PWAs and compare their performance to a large sample of persons not brain injured (PNBIs). We observed significant differences between PWAs and PNBIs for all MSSG variables. These differences occurred at the omnibus group level and for each aphasia subtype, even for PWAs with very mild impairment that is not detected with standardized aphasia assessment. Differences between PWAs and PNBIs were also practically significant, with medium to large effect sizes observed for nearly all aphasia subtypes and MSSG variables. This work deepens our understanding of discourse informativeness and macrostructure in PWAs and further develops an efficient tool for research and clinical use. Future research should investigate ways to expand MSSG analyses and to improve sensitivity and specificity.

Making Sense of Right Hemisphere Discourse Using RHDBank

PURPOSE

Right hemisphere brain damage (RHD) commonly causes pragmatic language disorders that are apparent in discourse production. Specific characteristics and approaches to assessment, diagnosis, and treatment of these disorders are not well-defined. RHDBank, a shared database of multimedia interactions for the study of communication using discourse, was created to address these gaps. The database, materials, and related analysis programs are free resources to clinicians, researchers, educators, and students.

METHOD

A standard discourse protocol was developed to elicit multiple types of discourse: free speech, conversation, picture description, storytelling, procedural discourse, and question-asking. Testing included measures of cognition, unilateral neglect, and communicative participation. Language samples were video-recorded and transcribed in CHAT format. Currently, the database includes 24 adults with RHD and 24 controls.

RESULTS

Illustrative analyses show how RHDBank can facilitate research using micro- and macrolinguistic discourse analysis techniques both within this population and across populations. Educational resources, such as the Grand Rounds tutorial, were developed using case studies from the database.

CONCLUSIONS

RHDBank is a shared database of resources that can facilitate educational and research efforts to address the gaps in knowledge about RHD communication and improve the clinical management of individuals with RHD.

Moving Toward Non-transcription based Discourse Analysis in Stable and Progressive Aphasia

Measurement of communication ability at the discourse level holds promise for predicting how well persons with stable (e.g., stroke-induced), or progressive aphasia navigate everyday communicative interactions. However, barriers to the clinical utilization of discourse measures have persisted. Recent advancements in the standardization of elicitation protocols and the existence of large databases for development of normative references have begun to address some of these barriers. Still, time remains a consistently reported barrier by clinicians. Non-transcription based discourse measurement would reduce the time required for discourse analysis, making clinical utilization a reality. The purpose of this article is to present evidence regarding discourse measures (main concept analysis, core lexicon, and derived efficiency scores) that are well suited to non-transcription based analysis. Combined with previous research, our results suggest that these measures are sensitive to changes following stroke or neurodegenerative disease. Given the evidence, further research specifically assessing the reliability of these measures in clinical implementation is warranted.

A Large-Scale Comparison of Main Concept Production Between Persons With Aphasia and Persons Without Brain Injury

Purpose The purposes of this study are to provide clinicians and researchers with introductory psychometric data for the main concept analysis (MCA), a measure of discourse informativeness, and specifically, to provide descriptive and comparative statistical information about the performance of a large sample of persons not brain injured (PNBIs) and persons with aphasia (PWAs) on AphasiaBank discourse tasks. Method Transcripts of 5 semi-spontaneous discourse tasks were retrieved from the AphasiaBank database and scored according to detailed checklists and scoring procedures. Transcripts from 145 PNBIs and 238 PWAs were scored; descriptive statistics, median tests, and effect sizes are reported. Results PWAs demonstrated overall lower informativeness scores and more frequent production of statements that were inaccurate and/or incomplete. Differences between PNBIs and PWAs were observed for all main concept measures and stories. Comparisons of PNBIs and aphasia subtypes revealed significant differences for all groups, although the pattern of differences and strength of effect sizes varied by group and discourse task. Conclusions These results may improve the investigative and clinical utility of the MCA by providing descriptive and comparative information for PNBIs and PWAs for standardized discourse tasks that can be reliably scored. The results indicate that the MCA is sensitive to differences in discourse as a result of aphasia. Supplemental Material https://doi.org/10.23641/asha.7485647.