Reliability of the Brief Assessment of Transactional Success in Communication in Aphasia

Large Language Models' Ability to Assess Main Concepts in Story Retelling: A Proof-of-Concept Comparison of Human Versus Machine Ratings

PURPOSE

Despite an abundance of manual, labor-intensive discourse analysis methods, there remains a dearth of clinically convenient, psychometrically robust instruments to measure change in real-world communication in aphasia. The Brief Assessment of Transactional Success (BATS) addresses this gap while developing automated methods for analyzing story retelling discourse. This study investigated automation of main concept (MC) analysis of stories by comparing scores from three large language models (LLMs) to those of human raters.

METHOD

After watching/listening to each of the eight short video/audio BATS stimuli and retelling each story, 96 persons with aphasia (PWA; n = 48 female) engaged in topic-constrained conversations over Zoom with 94 familiar and 107 unfamiliar conversation partners (CPs). CPs then retold each story as co-constructed during their conversations with PWA. Audio files from the resulting 1,760 story retells were transcribed using Python and AssemblyAI's speech-to-text application programming interface. Each MC was first scored by human raters for presence, accuracy, and completeness. Raters used a semiautomated application, MainConcept. For each transcript, an MC composite ratio score was obtained. We evaluated three state-of-the-art LLMs: two proprietary models, GPT-4 and GPT-4o, and one open-source model, Llama-3-70B. The interrater reliability between each LLM versus human MC scoring was assessed via the Pearson correlation coefficient and reliability coefficients based on the generalizability theory (G-theory).

RESULTS

The Pearson correlation coefficients indicate strong positive linear relationships between LLM and human MC scores. G-theory reliability coefficients also indicate reliable scoring between LLM and human scoring across the spectrum of participants and conditions.

CONCLUSIONS

This promising proof-of-concept study affirms the reliability of three LLMs in evaluating BATS story retell MCs and justifies ongoing investigation into their use. Providing clinicians and clinical researchers with automated tools for analyzing discourse without the need for prohibitively labor-intensive manual scoring could be a paradigm shift, potentially revolutionizing the aphasia intervention landscape.

Expanding the scope: multimodal dimensions in aphasia discourse analysis-preliminary findings

Background

Aphasia, resulting from acquired brain injury, disrupts language processing and usage, significantly impacting individuals' social communication and life participation. Given the limitations of traditional assessments in capturing the nuanced challenges faced by individuals with aphasia, this study seeks to explore the potential benefits of integrating multimodal communication elements into discourse analysis to better capture narrative proficiency in this population.

Objective

This study examined how incorporating multimodal communication elements (e.g., physical gestures, writing, drawing) into discourse analysis may affect the narrative outcomes of persons with aphasia compared to those observed using methods that exclude multimodal considerations.

Methods

Participants included individuals with chronic aphasia and age-and education-matched healthy controls who completed a storytelling task-the Bear and the Fly story. Macrolinguistic scores were obtained using verbal-only and multimodal scoring approaches. Additionally, the frequency and type of multimodal communication use during storytelling were examined in relation to aphasia characteristics. Statistical analyses included both within-group and between-group comparisons as well as correlational analyses.

Results

Individuals with aphasia scored significantly higher in terms of their macrolinguistic abilities when multimodal scoring was considered compared to verbal-only scoring. Within the aphasia group, there were prominent differences noted in macrolinguistic scores for both fluent and nonfluent aphasia. Specifically, both groups scored higher on Main Concepts when multimodal scoring was considered, with the nonfluent group demonstrating significantly higher Main Concept and total macrolinguistic rubric scores in multimodal scoring compared to verbal scoring on the storytelling task. Additionally, aphasia severity showed moderate positive correlations with total macrolinguistic scores, indicating that individuals with less severe aphasia tended to produce higher quality narratives. Lastly, although persons with aphasia used different types of nonverbal modalities (i.e., drawing, writing), the use of meaning-laden gestures was most predominant during storytelling, emphasizing the importance of multimodal elements in communication for individuals with aphasia.

Conclusion

Our preliminary study findings underscore the importance of considering multimodal communication in assessing discourse performance among individuals with aphasia. Tailoring assessment approaches based on aphasia subtypes can provide valuable insights into linguistic abilities and inform targeted intervention strategies for improving communication outcomes.