Automated speech analysis tools for children's speech production: A systematic literature review

Evaluating acoustic representations and normalization for rhoticity classification in children with speech sound disorders

The effects of different acoustic representations and normalizations were compared for classifiers predicting perception of children's rhotic versus derhotic /ɹ/. Formant and Mel frequency cepstral coefficient (MFCC) representations for 350 speakers were z-standardized, either relative to values in the same utterance or age-and-sex data for typical /ɹ/. Statistical modeling indicated age-and-sex normalization significantly increased classifier performances. Clinically interpretable formants performed similarly to MFCCs and were endorsed for deep neural network engineering, achieving mean test-participant-specific F1-score = 0.81 after personalization and replication (σx = 0.10, med = 0.83, n = 48). Shapley additive explanations analysis indicated the third formant most influenced fully rhotic predictions.

Improving Text-Independent Forced Alignment to Support Speech-Language Pathologists with Phonetic Transcription

Problem: Phonetic transcription is crucial in diagnosing speech sound disorders (SSDs) but is susceptible to transcriber experience and perceptual bias. Current forced alignment (FA) tools, which annotate audio files to determine spoken content and its placement, often require manual transcription, limiting their effectiveness. Method: We introduce a novel, text-independent forced alignment model that autonomously recognises individual phonemes and their boundaries, addressing these limitations. Our approach leverages an advanced, pre-trained wav2vec 2.0 model to segment speech into tokens and recognise them automatically. To accurately identify phoneme boundaries, we utilise an unsupervised segmentation tool, UnsupSeg. Labelling of segments employs nearest-neighbour classification with wav2vec 2.0 labels, before connectionist temporal classification (CTC) collapse, determining class labels based on maximum overlap. Additional post-processing, including overfitting cleaning and voice activity detection, is implemented to enhance segmentation. Results: We benchmarked our model against existing methods using the TIMIT dataset for normal speakers and, for the first time, evaluated its performance on the TORGO dataset containing SSD speakers. Our model demonstrated competitive performance, achieving a harmonic mean score of 76.88% on TIMIT and 70.31% on TORGO. Implications: This research presents a significant advancement in the assessment and diagnosis of SSDs, offering a more objective and less biased approach than traditional methods. Our model's effectiveness, particularly with SSD speakers, opens new avenues for research and clinical application in speech pathology.

Reproducible Speech Research With the Artificial Intelligence-Ready PERCEPT Corpora

BACKGROUND

Publicly available speech corpora facilitate reproducible research by providing open-access data for participants who have consented/assented to data sharing among different research teams. Such corpora can also support clinical education, including perceptual training and training in the use of speech analysis tools.

PURPOSE

In this research note, we introduce the PERCEPT (Perceptual Error Rating for the Clinical Evaluation of Phonetic Targets) corpora, PERCEPT-R (Rhotics) and PERCEPT-GFTA (Goldman-Fristoe Test of Articulation), which together contain over 36 hr of speech audio (> 125,000 syllable, word, and phrase utterances) from children, adolescents, and young adults aged 6-24 years with speech sound disorder (primarily residual speech sound disorders impacting /ɹ/) and age-matched peers. We highlight PhonBank as the repository for the corpora and demonstrate use of the associated speech analysis software, Phon, to query PERCEPT-R. A worked example of research with PERCEPT-R, suitable for clinical education and research training, is included as an appendix. Support for end users and information/descriptive statistics for future releases of the PERCEPT corpora can be found in a dedicated Slack channel. Finally, we discuss the potential for PERCEPT corpora to support the training of artificial intelligence clinical speech technology appropriate for use with children with speech sound disorders, the development of which has historically been constrained by the limited representation of either children or individuals with speech impairments in publicly available training corpora.

CONCLUSIONS

We demonstrate the use of PERCEPT corpora, PhonBank, and Phon for clinical training and research questions appropriate to child citation speech. Increased use of these tools has the potential to enhance reproducibility in the study of speech development and disorders.

Talkitt: toward a new instrument based on artificial intelligence for augmentative and alternative communication in children with down syndrome

Introduction

Individuals with Down syndrome (DS) often exhibit a severe speech impairment, with important consequences on language intelligibility. For these cases, the use of Augmentative Alternative Communication instruments, that increase an individual's communication abilities, becomes crucial. Talkitt is a mobile application created by Voiceitt Company, exploiting speech recognition technology and artificial intelligence models to translate in real-time unintelligible sounds into clear words, allowing individuals with language production impairment to verbally communicate in real-time.

Methods

The study evaluated the usability and satisfaction related to the Talkitt application use, as well as effects on adapted behavior and communication, of participants with DS. A final number of 23 individuals with DS, aged 5.54 to 28.9 years, participated in this study and completed 6 months of training. The application was trained to consistently recognize at least 20 different unintelligible words (e.g., nouns and/or short phrases)/person.

Results

Results revealed good usability and high levels of satisfaction related to the application use. Moreover, we registered improvement in linguistic abilities, particularly naming.

Discussion

These results paves the road for a potential role of Talkitt application as a supportive and rehabilitative tool for DS.

Speech assessment tool methods for speech impaired children: a systematic literature review on the state-of-the-art in Speech impairment analysis

Speech is a powerful, natural mode of communication that facilitates effective interactions in human societies. However, when fluency or flow of speech is affected or interrupted, it leads to speech impairment. There are several types of speech impairment depending on the speech pattern and range from mild to severe. Childhood apraxia of speech (CAS) is the most common speech disorder in children, with 1 out of 12 children diagnosed globally. Significant advancements in speech assessment tools have been reported to assist speech-language pathologists diagnosis speech impairment. In recent years, speech assessment tools have also gained popularity among pediatricians and teachers who work with preschoolers. Automatic speech tools can be more accurate for detecting speech sound disorders (SSD) than human-based speech assessment methods. This systematic literature review covers 88 studies, including more than 500 children, infants, toddlers, and a few adolescents, (both male and female) (age = 0-17) representing speech impairment from more than 10 countries. It discusses the state-of-the-art speech assessment methods, including tools, techniques, and protocols for speech-impaired children. Additionally, this review summarizes notable outcomes in detecting speech impairments using said assessment methods and discusses various limitations such as universality, reliability, and validity. Finally, we consider the challenges and future directions for speech impairment assessment tool research.

Application of Digital Games for Speech Therapy in Children: A Systematic Review of Features and Challenges

Introduction

Treatment of speech disorders during childhood is essential. Many technologies can help speech and language pathologists (SLPs) to practice speech skills, one of which is digital games. This study aimed to systematically investigate the games developed to treat speech disorders and their challenges in children.

Methods

A comprehensive search was conducted in four databases, including Medline (through PubMed), Scopus, Web of Science, and IEEE Xplore, to retrieve English articles published by July 14, 2021. The articles in which a digital game was developed to treat speech disorders in children were included in the study. Then, the features of the designed games and their challenges were extracted from the studies.

Results

After reviewing the full texts of 69 articles and assessing them in terms of inclusion and exclusion criteria, 27 articles were included in the systematic review. In these articles, 59.25% of the games had been developed in English language and children with hearing impairments had received much attention from researchers compared to other patients. Also, the Mel-Frequency Cepstral Coefficients (MFCC) algorithm and the PocketSphinx speech recognition engine had been used more than any other speech recognition algorithm and tool. In terms of the games, 48.15% had been designed in a way that children could practice with the help of their parents. The evaluation of games showed a positive effect on children's satisfaction, motivation, and attention during speech therapy exercises. The biggest barriers and challenges mentioned in the studies included sense of frustration, low self-esteem after several failures in playing games, environmental noise, contradiction between games levels and the target group's needs, and problems related to speech recognition.

Conclusion

The results of this study showed that the games positively affect children's motivation to continue speech therapy, and they can also be used as the SLPs' aids. Before designing these tools, the obstacles and challenges should be considered, and also, the solutions should be suggested.

Telepractice Treatment of Residual Rhotic Errors Using App-Based Biofeedback: A Pilot Study

PURPOSE

Although mobile apps are used extensively by speech-language pathologists, evidence for app-based treatments remains limited in quantity and quality. This study investigated the efficacy of app-based visual-acoustic biofeedback relative to nonbiofeedback treatment using a single-case randomization design. Because of COVID-19, all intervention was delivered via telepractice.

METHOD

Participants were four children aged 9-10 years with residual errors affecting American English /ɹ/. Using a randomization design, individual sessions were randomly assigned to feature practice with or without biofeedback, all delivered using the speech app Speech Therapist's App for /r/ Treatment. Progress was assessed using blinded listener ratings of word probes administered at baseline, posttreatment, and immediately before and after each treatment session.

RESULTS

All participants showed a clinically significant response to the overall treatment package, with effect sizes ranging from moderate to very large. One participant showed a significant advantage for biofeedback over nonbiofeedback treatment, although the order of treatment delivery poses a potential confound for interpretation in this case.

CONCLUSIONS

While larger scale studies are needed, these results suggest that app-based treatment for residual errors can be effective when delivered via telepractice. These results are compatible with previous findings in the motor learning literature regarding the importance of treatment dose and the timing of feedback conditions.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.18461576.

An Automated Lexical Stress Classification Tool for Assessing Dysprosody in Childhood Apraxia of Speech

Childhood apraxia of speech (CAS) commonly affects the production of lexical stress contrast in polysyllabic words. Automated classification tools have the potential to increase reliability and efficiency in measuring lexical stress. Here, factors affecting the accuracy of a custom-built deep neural network (DNN)-based classification tool are evaluated. Sixteen children with typical development (TD) and 26 with CAS produced 50 polysyllabic words. Words with strong-weak (SW, e.g., dinosaur) or WS (e.g., banana) stress were fed to the classification tool, and the accuracy measured (a) against expert judgment, (b) for speaker group, and (c) with/without prior knowledge of phonemic errors in the sample. The influence of segmental features and participant factors on tool accuracy was analysed. Linear mixed modelling showed significant interaction between group and stress type, surviving adjustment for age and CAS severity. For TD, agreement for SW and WS words was >80%, but CAS speech was higher for SW (>80%) than WS (~60%). Prior knowledge of segmental errors conferred no clear advantage. Automatic lexical stress classification shows promise for identifying errors in children's speech at diagnosis or with treatment-related change, but accuracy for WS words in apraxic speech needs improvement. Further training of algorithms using larger sets of labelled data containing impaired speech and WS words may increase accuracy.

A Longitudinal Evaluation of Tablet-Based Child Speech Therapy with Apraxia World

Digital games can make speech therapy exercises more enjoyable for children and increase their motivation during therapy. However, many such games developed to date have not been designed for long-term use. To address this issue, we developed Apraxia World, a speech therapy game specifically intended to be played over extended periods. In this study, we examined pronunciation improvements, child engagement over time, and caregiver and automated pronunciation evaluation accuracy while using our game over a multi-month period. Ten children played Apraxia World at home during two counterbalanced 4-week treatment blocks separated by a 2-week break. In one treatment phase, children received pronunciation feedback from caregivers and in the other treatment phase, utterances were evaluated with an automated framework built into the game. We found that children made therapeutically significant speech improvements while using Apraxia World, and that the game successfully increased engagement during speech therapy practice. Additionally, in offline mispronunciation detection tests, our automated pronunciation evaluation framework outperformed a traditional method based on goodness of pronunciation scoring. Our results suggest that this type of speech therapy game is a valid complement to traditional home practice.

The use of technology in the management of children with phonological delay and adults with acquired dysarthria: A UK survey of current speech-language pathology practice

PURPOSE

Technology is increasingly important for the speech-language pathology profession, but little is currently known about its use by clinicians. This study aimed to determine (i) the types of technology that speech-language pathologists (SLPs) in the UK have access to and use in practice and (ii) the barriers they encounter when assessing and treating adults with acquired dysarthria and children with phonological delay.

METHOD

UK SLPs were invited to complete two online surveys covering device availability, the use of technology for the assessment and treatment of acquired dysarthria and phonological delay, and barriers to using technology. Results were analysed using descriptive statistics.

RESULT

126 SLPs completed the surveys. Most respondents had a range of devices available in clinic, including computer and touchscreen devices. Technology was primarily used for treatment to engage clients, provide direct feedback in sessions and encourage home practice. Reported key barriers include lack of knowledge and training, and technical support issues.

CONCLUSION

The use of technology in UK clinical practice varies widely, and technology adoption is hampered by various barriers. Findings indicate a need for more collaborative work between SLPs, technologists and policy-makers to develop the evidence-base for technology use in the management of acquired dysarthria and phonological delay.

Supporting Children With Speech Sound Disorders During COVID-19 Restrictions: Technological Solutions

Purpose

“Children are the hidden victims of the COVID-19 pandemic” (United Nations Children's Fund, 2020). Timely and effective speech intervention is important to reduce the impact on children's school achievement, ability to make friends, mental health, future life opportunities, and government resources. Prior to the coronavirus disease (COVID-19) pandemic, many Australian children did not receive sufficient speech-language pathology (SLP) services due to long waiting lists in the public health system. COVID-19 restrictions exacerbated this issue, as even children who were at the top of lengthy SLP waiting lists often received limited services, particularly in rural areas. To facilitate children receiving speech intervention remotely during the COVID-19 pandemic, evidence from randomized controlled trials regarding three technological solutions are examined: (a) Phoneme Factory Sound Sorter (Sound Start Study), (b) Waiting for Speech Pathology website, and (c) Apraxia World.

Conclusions

For the first two technological solutions, there were similar gains in speech production between the intervention and control groups, whereas, for the third solution, the average magnitude of treatment effect was comparable to face-to-face SLP therapy. Automated therapy management systems may be able to accelerate speech development and support communication resilience to counteract the effects of the COVID-19 restrictions on children with speech sound disorders. Technology-based strategies may also provide a potential solution to the chronic shortage of SLP services in rural areas into the future.

Consonant and vowel articulation accuracy in younger and middle-aged Spanish healthy adults

Children acquire vowels earlier than consonants, and the former are less vulnerable to speech disorders than the latter. This study explores the hypothesis that a similar contrast exists later in life and that consonants are more vulnerable to ageing than vowels. Data was obtained with two experiments comparing the speech of Younger Adults (YAs) and Middle–aged Adults (MAs). In the first experiment an Automatic Speech Recognition (ASR) system was trained with a balanced corpus of 29 YAs and 27 MAs. The productions of each speaker were obtained in a Spanish language word (W) and non–word (NW) repetition task. The performance of the system was evaluated with the same corpus used for training using a cross validation approach. The ASR system recognized to a similar extent the Ws of both groups of speakers, but it was more successful with the NWs of the YAs than with those of the MAs. Detailed error analysis revealed that the MA speakers scored below the YA speakers for consonants and also for the place and manner of articulation features; the results were almost identical in both groups of speakers for vowels and for the voicing feature. In the second experiment a group of healthy native listeners was asked to recognize isolated syllables presented with background noise. The target speakers were one YA and one MA that had taken part in the first experiment. The results were consistent with those of the ASR experiment: the manner and place of articulation were better recognized, and vowels and voicing were worse recognized, in the YA speaker than in the MA speaker. We conclude that consonant articulation is more vulnerable to ageing than vowel articulation. Future studies should explore whether or not these early and selective changes in articulation accuracy might be caused by changes in speech perception skills (e.g., in auditory temporal processing).

Feasibility of Automatic Speech Recognition for Providing Feedback During Tablet-Based Treatment for Apraxia of Speech Plus Aphasia

Purpose Individuals with neurogenic speech disorders require ongoing therapeutic support to achieve functional communication goals. Alternative methods for service delivery, such as tablet-based speech therapy applications, may help bridge the gap and bring therapeutic interventions to the patient in an engaging way. The purpose of this study was to evaluate an iPad-based speech therapy app that uses automatic speech recognition (ASR) software to provide feedback on speech accuracy to determine the ASR's accuracy against human judgment and whether participants' speech improved with this ASR-based feedback. Method Five participants with apraxia of speech plus aphasia secondary to stroke completed an intensive 4-week at-home therapy program using a novel word training app with built-in ASR. Multiple baselines across participants and behaviors designs were employed, with weekly probes and follow-up at 1 month posttreatment. Four sessions a week of 100 practice trials each were prescribed, with 1 being clinician-run and the remainder done independently. Dependent variables of interest were ASR-human agreement on accuracy during practice trials and human-judged word production accuracy over time in probes. Also, user experience surveys were completed immediately posttreatment. Results ASR-human agreement on accuracy averaged ~80%, which is a common threshold applied for interrater agreement. All participants demonstrated improved word production accuracy over time with the ASR-based feedback and maintenance of gains after 1 month. All participants reported enjoying using the app with support of a speech pathologist. Conclusion For these participants with apraxia of speech plus aphasia due to stroke, satisfactory gains were made in word production accuracy with an app-based therapy program providing ASR-based feedback on accuracy. Findings support further testing of this ASR-based approach as a supplement to clinician-run sessions to assist clients with similar profiles in achieving higher amount and intensity of practice as well as empowering them to manage their own therapy program. Supplemental Material https://doi.org/10.23641/asha.8206628.