Statistical considerations for crowdsourced perceptual ratings of human speech productions

The Multilevel Word Accuracy Composite Scale: A Novel Measure of Speech Production in Childhood Apraxia of Speech

PURPOSE

The Multilevel word Accuracy Composite Scale (MACS) is a novel whole-word measure of speech production accuracy designed to evaluate behaviors commonly targeted in motor-based intervention for childhood apraxia of speech (CAS). The MACS yields a composite score generated through ratings of segmental accuracy, word structure maintenance, prosody, and movement transition. This study examined the validity of the MACS through comparison to established measures of speech accuracy. Reliability was also examined within and between practicing speech-language pathologists (SLPs).

METHOD

The MACS was used to rate 117 tokens produced by children with severe CAS. Ratings were performed in the laboratory setting by two expert raters and by practicing SLPs (N = 19). Concurrent validity was estimated through comparison of expert MACS ratings (i.e., MACS score and each component rating) to measures of speech accuracy (percent phoneme correct and the 3-point scale) using correlational analyses. Reliability was examined between expert raters and across SLP raters using the intraclass correlation coefficient to examine interrater reliability of expert ratings, in addition to inter- and intrarater reliability of SLP ratings.

RESULTS

Correlation analyses between MACS ratings (i.e., MACS score and component ratings) and existing measures of speech accuracy revealed small to large positive correlations between measures. Reliability analyses revealed moderate to excellent reliability for MACS ratings performed by expert raters and between (interrater) and within (intrarater) SLP raters.

CONCLUSIONS

Analyses of concurrent validity indicate that the MACS aligns with established measures, yet contributes novel elements for rating speech accuracy. Results further support the MACS as a reliable measure for rating speech accuracy in children with severe speech impairment for ratings performed by expert raters and practicing clinicians.

Crowdsourcing as a tool in the clinical assessment of intelligibility in dysarthria: How to deal with excessive variation

PURPOSE

Independent laypersons are essential in the assessment of intelligibility in persons with dysarthria (PWD), as they reflect intelligibility limitations in the most ecologically valid way, without being influenced by familiarity with the speaker. The present work investigated online crowdsourcing as a convenient method to involve lay people as listeners, with the objective of exploring how to constrain the expected variability of crowd-based judgements to make them applicable in clinical diagnostics.

METHOD

Intelligibility was assessed using a word transcription task administered via crowdsourcing. In study 1, speech samples of 23 PWD were transcribed by 18 crowdworkers each. Four methods of aggregating the intelligibility scores of randomly sampled panels of 4 to 14 listeners were compared for accuracy, i.e. the stability of the resulting intelligibility estimates across different panels, and their validity, i.e. the degree to which they matched data obtained under controlled laboratory conditions ("gold standard"). In addition, we determined an economically acceptable number of crowdworkers per speaker which is needed to obtain accurate and valid intelligibility estimates. Study 2 examined the robustness of the chosen aggregation method against downward outliers due to spamming in a larger sample of 100 PWD.

RESULTS

In study 1, an interworker aggregation method based on negative exponential weightings of the scores as a function of their distance from the "best" listener's score (exponentially weighted mean) outperformed three other methods (median value, arithmetic mean, maximum). Under cost-benefit considerations, an optimum panel size of 9 crowd listeners per examination was determined. Study 2 demonstrated the robustness of this aggregation method against spamming crowd listeners.

CONCLUSION

Though intelligibility data collected through online crowdsourcing are noisy, accurate and valid intelligibility estimates can be obtained by appropriate aggregation of the raw data. This makes crowdsourcing a suitable method for incorporating real-world perspectives into clinical dysarthria assessment.

Vocal development in a large-scale crosslinguistic corpus

This study evaluates whether early vocalizations develop in similar ways in children across diverse cultural contexts. We analyze data from daylong audio recordings of 49 children (1-36 months) from five different language/cultural backgrounds. Citizen scientists annotated these recordings to determine if child vocalizations contained canonical transitions or not (e.g., "ba" vs. "ee"). Results revealed that the proportion of clips reported to contain canonical transitions increased with age. Furthermore, this proportion exceeded 0.15 by around 7 months, replicating and extending previous findings on canonical vocalization development but using data from the natural environments of a culturally and linguistically diverse sample. This work explores how crowdsourcing can be used to annotate corpora, helping establish developmental milestones relevant to multiple languages and cultures. Lower inter-annotator reliability on the crowdsourcing platform, relative to more traditional in-lab expert annotators, means that a larger number of unique annotators and/or annotations are required, and that crowdsourcing may not be a suitable method for more fine-grained annotation decisions. Audio clips used for this project are compiled into a large-scale infant vocalization corpus that is available for other researchers to use in future work.