Time-interval measurement of stuttering: systematic replication of Ingham, Cordes, and Gow (1993)

Comparison of performance of automatic recognizers for stutters in speech trained with event or interval markers

Introduction

Automatic recognition of stutters (ARS) from speech recordings can facilitate objective assessment and intervention for people who stutter. However, the performance of ARS systems may depend on how the speech data are segmented and labelled for training and testing. This study compared two segmentation methods: event-based, which delimits speech segments by their fluency status, and interval-based, which uses fixed-length segments regardless of fluency.

Methods

Machine learning models were trained and evaluated on interval-based and event-based stuttered speech corpora. The models used acoustic and linguistic features extracted from the speech signal and the transcriptions generated by a state-of-the-art automatic speech recognition system.

Results

The results showed that event-based segmentation led to better ARS performance than interval-based segmentation, as measured by the area under the curve (AUC) of the receiver operating characteristic. The results suggest differences in the quality and quantity of the data because of segmentation method. The inclusion of linguistic features improved the detection of whole-word repetitions, but not other types of stutters.

Discussion

The findings suggest that event-based segmentation is more suitable for ARS than interval-based segmentation, as it preserves the exact boundaries and types of stutters. The linguistic features provide useful information for separating supra-lexical disfluencies from fluent speech but may not capture the acoustic characteristics of stutters. Future work should explore more robust and diverse features, as well as larger and more representative datasets, for developing effective ARS systems.

Feasibility of automated speech sample collection with stuttering children using interactive voice response (IVR) technology

PURPOSE

To investigate the feasibility of adopting automated interactive voice response (IVR) technology for remotely capturing standardized speech samples from stuttering children.

METHOD

Participants were 10 6-year-old stuttering children. Their parents called a toll-free number from their homes and were prompted to elicit speech from their children using a standard protocol involving conversation, picture description and games. The automated IVR system was implemented using an off-the-shelf telephony software program and delivered by a standard desktop computer. The software infrastructure utilizes voice over internet protocol. Speech samples were automatically recorded during the calls. Video recordings were simultaneously acquired in the home at the time of the call to evaluate the fidelity of the telephone collected samples. Key outcome measures included syllables spoken, percentage of syllables stuttered and an overall rating of stuttering severity using a 10-point scale.

RESULT

Data revealed a high level of relative reliability in terms of intra-class correlation between the video and telephone acquired samples on all outcome measures during the conversation task. Findings were less consistent for speech samples during picture description and games.

CONCLUSION

Results suggest that IVR technology can be used successfully to automate remote capture of child speech samples.

Event- and interval-based measurement of stuttering: a review

BACKGROUND

Event- and interval-based measurements are two different ways of computing frequency of stuttering. Interval-based methodology emerged as an alternative measure to overcome problems associated with reproducibility in the event-based methodology. No review has been made to study the effect of methodological factors in interval-based absolute reliability data or to compute the agreement between the two methodologies in terms of inter-judge, intra-judge and accuracy (i.e., correspondence between raters' scores and an established criterion).

AIMS

To provide a review related to reproducibility of event-based and time-interval measurement, and to verify the effect of methodological factors (training, experience, interval duration, sample presentation order and judgment conditions) on agreement of time-interval measurement; in addition, to determine if it is possible to quantify the agreement between the two methodologies

METHODS & PROCEDURES

The first two authors searched for articles on ERIC, MEDLINE, PubMed, B-on, CENTRAL and Dissertation Abstracts during January-February 2013 and retrieved 495 articles. Forty-eight articles were selected for review. Content tables were constructed with the main findings.

MAIN CONTRIBUTION

Articles related to event-based measurements revealed values of inter- and intra-judge greater than 0.70 and agreement percentages beyond 80%. The articles related to time-interval measures revealed that, in general, judges with more experience with stuttering presented significantly higher levels of intra- and inter-judge agreement. Inter- and intra-judge values were beyond the references for high reproducibility values for both methodologies. Accuracy (regarding the closeness of raters' judgements with an established criterion), intra- and inter-judge agreement were higher for trained groups when compared with non-trained groups. Sample presentation order and audio/video conditions did not result in differences in inter- or intra-judge results. A duration of 5 s for an interval appears to be an acceptable agreement. Explanation for high reproducibility values as well as parameter choice to report those data are discussed.

CONCLUSIONS & IMPLICATIONS

Both interval- and event-based methodologies used trained or experienced judges for inter- and intra-judge determination and data were beyond the references for good reproducibility values. Inter- and intra-judge values were reported in different metric scales among event- and interval-based methods studies, making it unfeasible to quantify the agreement between the two methods.

Webcam Delivery of the Camperdown Program for Adolescents Who Stutter: A Phase I Trial

Purpose

This Phase I clinical trial explored the viability of webcam Internet delivery of the Camperdown Program for adolescents who stutter.

Method and Procedure

Participants were 3 adolescents ages 13, 15, and 16 years, with moderate-severe stuttering. Each was treated with the Camperdown Program delivered by webcam with no clinic attendance. Primary outcome measures were percentage of syllables stuttered and number of treatment sessions to maintenance. Secondary outcome measures were speech naturalness, situation avoidance, self-reported stuttering severity, and parent and adolescent satisfaction. Data were collected pre treatment and at 1 day, 6 months, and 12 months post entry to maintenance.

Results

Participants entered maintenance after means of 18 sessions and 11 clinician hours. Group mean reduction of stuttering from pre treatment to entry to maintenance was 83%, from pre treatment to 6 months post entry to maintenance was 93%, and from pre treatment to 12 months post entry to maintenance was 74%. Self-reported stuttering severity ratings confirmed these results. Post entry to maintenance speech naturalness for participants fell within the range of that of 3 matched controls. However, avoidance of speech situations showed no corresponding improvements for 2 of the participants.

Conclusion

The service delivery model was efficacious and efficient. All of the participants and their parents also found it appealing. Results justify a Phase II trial of the delivery model.

Construct validity of modified time-interval analysis in measuring stuttering and trained speaking patterns

PURPOSE

The aim of the present study was to examine the construct validity of modified time-interval analysis. This measure allows judgments on stuttered and fluent speech as well as trained speaking patterns and might be valuable for outcome studies.

METHOD

Construct validity was investigated in an intervention study with two treatment groups (24 clients received stuttering modification treatment, 30 clients received fluency modification treatment) and a control group (38 stuttering adults). All participants were interviewed during surprise phone calls before and after treatment; the speech samples were analyzed by means of modified time-interval analysis and stuttering frequency counts.

RESULTS

The outcomes confirmed prior hypotheses for the most part. First, the amount of trained speaking patterns after therapy was significantly higher in both treatment groups than in the control group. Secondly, longitudinal changes in the treatment groups met prior expectations based on differing treatment goals and exceeded the changes in the control group. Modified time-interval analysis was sufficiently sensitive to detect changes of speech fluency, but underestimated spontaneous fluent speech when trained speaking patterns were applied.

CONCLUSION

The present study supports construct validity of modified time-interval analysis in measuring stuttering and trained speaking patterns, but also reveals a lack of accuracy.

EDUCATIONAL OBJECTIVES

At the end of this activity the reader will be able to (a) explain different forms of validity in relation to the use of modified time-interval analysis, (b) evaluate whether construct validity of modified time-interval analysis has been supported by the outcomes of an intervention study and (c) describe the usefulness and limitations of modified time-interval analysis for future research.

Measurement of trained speech patterns in stuttering: interjudge and intrajudge agreement of experts by means of modified time-interval analysis

Improved fluency after stuttering therapy is usually measured by the percentage of stuttered syllables. However, outcome studies rarely evaluate the use of trained speech patterns that speakers use to manage stuttering. This study investigated whether the modified time interval analysis can distinguish between trained speech patterns, fluent speech, and stuttered speech. Seventeen German experts on stuttering judged a speech sample on two occasions. Speakers of the sample were stuttering adults, who were not undergoing therapy, as well as participants in a fluency shaping and a stuttering modification therapy. Results showed satisfactory inter-judge and intra-judge agreement above 80%. Intervals with trained speech patterns were identified as consistently as stuttered and fluent intervals. We discuss limitations of the study, as well as implications of our findings for the development of training for identification of trained speech patterns and future outcome studies.

EDUCATIONAL OBJECTIVES

The reader will be able to (a) explain different methods to measure the use of trained speech patterns, (b) evaluate whether German experts are able to discriminate intervals with trained speech patterns reliably from fluent and stuttered intervals and (c) describe how the measurement of trained speech patterns can contribute to outcome studies.

Does language influence the accuracy of judgments of stuttering in children?

PURPOSE

To determine whether stuttering judgment accuracy is influenced by familiarity with the stuttering speaker's language.

METHOD

Audiovisual 7-min speech samples from nine 3- to 5-year-olds were used. Icelandic children who stutter (CWS), preselected for different levels of stuttering, were subdivided into 5-s intervals. Ten experienced Icelandic speech-language pathologists (ICE-SLPs) and 10 experienced U.S. speech-language pathologists (US-SLPs), the latter being unfamiliar with the Icelandic language, independently judged each 5-s interval (n = 756) as stuttered or nonstuttered on 2 separate occasions.

RESULTS

As in previous studies, intervals judged to contain stuttering showed wide variability within the ICE-SLP and US-SLP groups. However, both SLP groups (a) displayed satisfactory mean intrajudge agreement, (b) met an independent stuttering judgment accuracy criterion test using English-speaking CWS samples, and (c) met an agreement criterion on approximately 90% of their stuttering and nonstuttering judgments on the Icelandic-speaking CWS samples.

CONCLUSION

Experienced SLPs were shown to be highly accurate in recognizing stuttering and nonstuttering exemplars from young CWS speaking in an unfamiliar language. The findings suggest that judgments of occurrences of stuttering in CWS are not generally language dependent, although some exceptions were noted.

Characteristics of stuttering-like disfluencies in Dutch-speaking children

The purpose of this study was to compare the characteristics of stuttering-like disfluencies in a group of native Dutch-speaking children who stutter (n=693), with a group of normally fluent children (n=79). Methods involved the observation of stuttering-like disfluencies in participants' conversational speech samples (total 77,200 words), particularly the frequency, duration and physical tension of instances of stuttering. Findings indicate that stuttering-like disfluencies exhibited by children who stutter are significantly more frequent, longer in duration and involve more physical tension when compared to those of normally fluent children. Furthermore, applying a criterion of 3% stuttering-like disfluencies to distinguish stuttering from normally fluent children resulted in a high degree of sensitivity (0.9452) and specificity (0.9747). Results were taken to suggest that characteristics of stuttering-like disfluencies of Dutch-speaking children are similar to those of English-speaking children and that talker group membership criteria for childhood stuttering can reasonably be extrapolated from the Dutch to the English language and vice versa.

EDUCATIONAL OBJECTIVES

The reader will be able to: (1) describe characteristics of stuttering-like disfluencies, (2) define properties such as frequency, duration and physical tension for stuttering children and normally fluent children, and (3) make use of data on sensitivity and specificity of the criterion of 3% stuttering-like disfluencies to distinguish stuttering and normally fluent children.

Comparing judgments of stuttering made by students, clinicians, and highly experienced judges

PURPOSE

The purpose of this study was to compare judgments of stuttering made by students and clinicians with previously available judgments made by highly experienced judges in stuttering.

METHOD

On two occasions, 41 university students and 31 speech-language pathologists judged the presence or absence of stuttering in each of 216 audiovisually recorded 5-s intervals of the speech of adults who stutter. Intrajudge and interjudge agreement were calculated, and comparisons were made to judgments previously made about the same recordings by 10 highly experienced judges of stuttering.

RESULTS

Students and clinicians showed similar and relatively high levels of intrajudge and interjudge agreement, but both students and clinicians identified less than half as much stuttering as the highly experienced judges had identified.

CONCLUSIONS

These results replicate previous findings of high agreement coexisting with low accuracy in students' judgments of stuttering, extending those findings to show that similar problems are evident in judgments made by practicing clinicians. Implications include the need for explicit stuttering judgment training programs for both students and practicing clinicians.

EDUCATIONAL OBJECTIVES

After reading this article, the reader will be able to: (1) describe different methods for identifying stuttering and possible problems associated with each method; (2) describe two different methods for reporting interjudge reliability; (3) describe how the identification of stuttering differs for student, clinician, and highly experienced judges.

Interobserver agreement of perceived intelligibility of magnitude of r in children

This study was designed to test whether independent listeners could correctly classify 162 stimuli (words) that started with a multiple trill, magnitude of r, followed by a vowel magnitude of a. The sounds were from 27 Spanish children between the ages of 3.0 and 6.6 years, pronouncing Spanish words current in their vocabulary. Twelve listeners were presented with the recordings of the children's magnitude of r production and were instructed to rate the intelligibility of the pronounced sound as high, medium, or low. Inter-rater agreement ranged from 85 to 96%. Analyses of variance for each of the three production categories showed that there were significant differences, so it seems possible to classify the words starting with a trilled magnitude of r based on auditory-perceptual features into three broad categories according to intelligibility.

Functional-lesion investigation of developmental stuttering with positron emission tomography

Positron emission tomographic (PET) H2(15)O measurements of resting-state regional cerebral blood flow (CBF) were obtained in 29 right-handed men, 10 of whom stuttered and 19 of whom did not. PET images were analyzed by sampling 74 regions of interest (ROIs), 37 per hemisphere. ROI placement was guided both physiologically and anatomically. Physiological ROI placement was based on speech motor activations. Anatomical ROIs were positioned by reference to a stereotactic, neurosurgical atlas with positions confirmed and finely adjusted by co-registered magnetic-resonance images (MRIs). For all subjects, PET and MR images were normal to visual inspection. Highly significant (p < 0.0001) between-region and between-hemisphere effects were found for both groups, as have been previously reported for normal subjects, but no significant between-group differences were found for any regional CBF values. Analysis by a laterality index found a weakly significant between-groups effect (p = 0.04) that was isolated to five regions, four of which are implicated in speech or hearing. However, these regional laterality effects showed no consistent directionality, nor did these regions have absolute differences in regional blood flow between groups. Present findings do not support recent suggestions that developmental stuttering is associated with abnormalities of brain blood flow at rest. Rather, our findings indicate an essentially normal functional brain terrain with a small number of minor differences in hemispheric symmetry.

Treatment efficacy: stuttering

The purpose of this article is to review the state of the art regarding treatment efficacy for stuttering in children, teenagers, and adults. Available evidence makes it apparent that individuals who stutter benefit from the services of speech-language pathologists, but it is also apparent that determining the outcome of stuttering treatment is neither easy nor simple. Whereas considerable research has documented the positive influence of treatment on stuttering frequency and behavior, far less attention has been paid to the effects of treatment on the daily life activities of people who stutter and their families. Although it seems reasonable to assume that ameliorating the disability of stuttering lessens the handicap of stuttering, considerably more evidence is needed to confirm this assumption. Despite such concerns, it also seems reasonable to suggest that the outcomes of treatment for many people who stutter are positive and should become increasingly so with advances in applied as well as basic research.

A PET study of the neural systems of stuttering

The cause of stuttering is unknown. Failure to develop left-hemispheric dominance for speech is a long-standing theory although others implicated the motor system more broadly, often postulating hyperactivity of the right (language nondominant) cerebral hemisphere. As knowledge of motor circuitry has advanced, theories of stuttering have become more anatomically specific, postulating hyperactivity of premotor cortex, either directly or through connectivity with the thalamus and basal ganglia. Alternative theories target the auditory and speech production systems. By contrasting stuttering with fluent speech using positron emission tomography combined with chorus reading to induce fluency, we found support for each of these hypotheses. Stuttering induced widespread overactivations of the motor system in both cerebrum and cerebellum, with right cerebral dominance. Stuttered reading lacked left-lateralized activations of the auditory system, which are thought to support the self-monitoring of speech, and selectively deactivated a frontal-temporal system implicated in speech production. Induced fluency decreased or eliminated the overactivity in most motor areas, and largely reversed the auditory-system underactivations and the deactivation of the speech production system. Thus stuttering is a disorder affecting the multiple neural systems used for speaking.

Time-interval measurement of stuttering: establishing and modifying judgment accuracy

The purpose of this study was to determine whether accuracy training for interval judgments of stuttering might generalize to increased accuracy and/or interjudge agreement for intervals other than those used during training. Ten upper-division speech-language pathology students judged 5-s audiovisually recorded speech intervals as stuttered or nonstuttered in a series of group and single-subject experiments. Judgment accuracy was determined with respect to judgments provided previously by 10 recognized authorities on stuttering and its treatment. Training occurred within single-subject experiments that used multiple baselines across speakers and repeated generalization probes to assess training effects. Results showed that judgment accuracy tended to increase after training for speakers used during the training process as well as for unfamiliar speakers. Results also replicated previous findings of slight increases in interjudge and intrajudge agreement after interval-judgment training. The implications of these results for developing a valid and reliable stuttering measurement system are discussed.

Identifying the onset and offset of stuttering events

This study was designed to investigate the apparent contradiction between recent reports of physiological and interpersonal research on stuttering that claim or imply high agreement levels, and studies of stuttering judgment agreement itself that report much lower agreement levels. Four experienced stuttering researchers in one university department used laser videodisks of spontaneous speech, from persons whose stuttering could be described as mild to severe, to locate the precise onset and offset of individual stuttering events. Results showed a series of interjudge disagreements that raise serious questions about the reliability and validity of stuttering event onset and offset judgments. These results highlight the potentially poor reliability of a measurement procedure that is currently widespread in stuttering research. At the same time, they have isolated some few highly agreed stuttering events that might serve as the basis for the further development of either event-based or interval-based judgment procedures.

Judgments of stuttered and nonstuttered intervals by recognized authorities in stuttering research

The study reported in this paper gathered judgments of stuttering on brief (5.0-sec) audiovisual speech samples taken from six adults who stuttered. Judgments were made by 10 highly experienced authorities on stuttering treatment and research, located in seven different universities or clinical research centers. Results showed considerable agreement between pairs of judges working in the same center, but large and potentially fundamental differences were identified in the amount of stuttering recorded in different centers. Approximately 40% of the 5.0-sec speech intervals used in this study were assigned the same judgment, either Stuttered or Nonstuttered, by all judges on two judgment occasions. The possibility that these intervals may serve as a core for establishing an across-center standard for behavioral judgments of stuttering is discussed.

Time-interval measurement of stuttering: effects of interval duration

The study reported in this article used a binary forced-choice judgment procedure to investigate the effects of sample duration on observers' judgements of stuttering. Two groups of judges, differing in their previous experience with stuttering, categorized 270 speech intervals as stuttered or nonstuttered; the intervals were drawn from 30 persons who stuttered and ranged from 1 sec to 15 sec in duration. Results showed that judgments were consistently related to interval duration, with shorter intervals significantly more likely than longer intervals to be labeled nonstuttered. Interjudge agreement levels, however, were largely unaffected by the different interval durations for most speakers and for both judge groups, with the exception of the longest and shortest intervals drawn from speakers evidencing the mildest and most severe stuttering. An interval duration in the 3- to 5-sec region appeared to attract the most satisfactory level of agreement. The implications of these findings for interval-based clinical and experimental measurements of stuttering are discussed.

The reliability of observational data: I. Theories and methods for speech-language pathology

Much research and clinical work in speech-language pathology depends on the validity and reliability of data gathered through the direct observation of human behavior. This paper reviews several definitions of reliability, concluding that behavior observation data are reliable if they, and the experimental conclusions drawn from them, are not affected by differences among observers or by other variations in the recording context. The theoretical bases of several methods commonly used to estimate reliability for observational data are reviewed, with examples of the use of these methods drawn from a recent volume of the Journal of Speech and Hearing Research (35, 1992). Although most recent research publications in speech-language pathology have addressed the issue of reliability for their observational data to some extent, most reliability estimates do not clearly establish that the data or the experimental conclusions were replicable or unaffected by differences among observers. Suggestions are provided for improving the usefulness of the reliability estimates published in speech-language pathology research.

The reliability of observational data: II. Issues in the identification and measurement of stuttering events

Much attention has been directed recently toward the problem of measuring occurrences of stuttering with satisfactory levels of interjudge agreement. This paper reviews the prominent concepts of the stuttering event, arguing that they may be one cause of the stuttering measurement problem. The evidence that has led to concerns about the reliability of stuttering event measurements is also reviewed. Reliability and measurement issues that were discussed in the first paper of this series (Cordes, 1994) emerge as basic to the interpretation of much stuttering research, and it is argued that the stuttering measurement problem is not confined to research on stuttering judgments but actually permeates other important stuttering research areas. Some recent attempts to resolve the stuttering measurement problem are reviewed, and the implications of developing an improved measurement system for this disorder are discussed.