Time-interval measurement of stuttering: modifying interjudge agreement

The reliability of simultaneous versus individual data collection during stuttering assessment

BACKGROUND

Speech-language pathologists often multitask in order to be efficient with their commonly large caseloads. In stuttering assessment, multitasking often involves collecting multiple measures simultaneously.

AIMS

The present study sought to determine reliability when collecting multiple measures simultaneously versus individually.

METHODS & PROCEDURES

Over two time periods, 50 graduate students viewed videos of four persons who stutter (PWS) and counted the number of stuttered syllables and total number of syllables uttered, and rated speech naturalness. Students were randomly assigned to one of two groups: the simultaneous group, in which all measures were gathered during one viewing; and the individual group, in which one measure was gathered per viewing. Relative and absolute intra- and inter-rater reliability values were calculated for each measure.

OUTCOMES & RESULTS

The following results were notable: better intra-rater relative reliability for the number of stuttered syllables for the individual group (intraclass correlation coefficient (ICC) = 0.839) compared with the simultaneous group (ICC = 0.350), smaller intra-rater standard error of measurement (SEM) (i.e., better absolute reliability) for the number of stuttered syllables for the individual group (7.40) versus the simultaneous group (15.67), and better inter-rater absolute reliability for the total number of syllables for the individual group (88.29) compared with the simultaneous group (125.05). Absolute reliability was unacceptable for all measures across both groups.

CONCLUSIONS & IMPLICATIONS

These findings show that judges are likely to be more reliable when identifying stuttered syllables in isolation than when simultaneously collecting them with total syllables spoken and naturalness data. Results are discussed in terms of narrowing the reliability gap between data collection methods for stuttered syllables, improving overall reliability of stuttering measurements, and a procedural change when implementing widely used stuttering assessment protocols.

WHAT THIS PAPER ADDS

What is already known on the subject The reliability of stuttering judgments has been found to be unacceptable across a number of studies, including those examining the reliability of the most popular stuttering assessment tool, the Stuttering Severity Instrument (4th edition). The SSI-4, and other assessment applications, involve collecting multiple measures simultaneously. It has been suggested, but not examined, that collecting measures simultaneously, which occurs in the most popular stuttering assessment protocols, may result in substantially inferior reliability when compared to collecting measures individually. What this paper adds to existing knowledge The present study has multiple novel findings. First, relative and absolute intra-rater reliability were substantially better when stuttered syllables data were collected individually compared to when the same data were collected simultaneously with total number of syllables and speech naturalness data. Second, inter-rater absolute reliability for total number of syllables was also substantially better when collected individually. Third, intra-rater and inter-rater reliability were similar when speech naturalness ratings were given individually compared to when they were given while simultaneously counting stuttered and fluent syllables. What are the potential or actual clinical implications of this work? Clinicians can be more reliable when identifying stuttered syllables individually compared to when they judge stuttering along with other clinical measures of stuttering. In addition, when clinicians and researchers use current popular protocols for assessing stuttering that recommend simultaneous data collection, including the SSI-4, they should instead consider collecting stuttering event counts individually. This procedural change will lead to more reliable data and stronger clinical decision making.

Reliability of judgments of stuttering-related variables: The effect of language familiarity

Previous studies demonstrate mixed results and some methodological limitations regarding judges' ability to reliably assess stuttering-related variables in an unfamiliar language. The present study examined intra- and inter-rater reliability for percent syllables stuttered (%SS), stuttering severity (SEV), syllables per minute (SPM), and speech naturalness (NAT) when English-speaking judges viewed speech samples in English and in a language with which they had no or minimal familiarity (Spanish). Over two time periods, 21 judges viewed eight videos of four bilingual persons who stutter. Data were analyzed for relative and absolute intra- and inter-rater reliability as well as for an effect of language on time period differences. Intra- and inter-rater relative reliability were good or excellent for all measures in both languages, with the exception of inter-rater relative reliability for NAT in both languages and %SS in Spanish. Intra-rater absolute reliability was acceptable in both languages for NAT and SEV and unacceptable in both for SPM and %SS. Inter-rater absolute reliability in both languages was unacceptable for all measures, even with judges with the same training. There was a clinically significant effect of language on %SS scores, but, despite a statistically significant effect of language for SPM and SEV, the differences were not clinically significant. Results indicate that reliability across and within languages varies by measure and is impacted by intra- vs. inter-rater reliability, relative vs. absolute reliability, and language familiarity. Modifications in training may be able to address some of the limitations found, particularly with regard to SPM and NAT.

Effects of measurement method and transcript availability on inexperienced raters' stuttering frequency scores

PURPOSE

To examine the effects of measurement method and transcript availability on the accuracy, reliability, and efficiency of inexperienced raters' stuttering frequency measurements.

METHOD

44 adults, all inexperienced at evaluating stuttered speech, underwent 20 min of preliminary training in stuttering measurement and then analyzed a series of sentences, with and without access to transcripts of sentence stimuli, using either a syllable-based analysis (SBA) or an utterance-based analysis (UBA). Participants' analyses were compared between groups and to a composite analysis from two experienced evaluators.

RESULTS

Stuttering frequency scores from the SBA and UBA groups differed significantly from the experienced evaluators' scores; however, UBA scores were significantly closer to the experienced evaluators' scores and were completed significantly faster than the SBA scores. Transcript availability facilitated scoring accuracy and efficiency in both groups. The internal reliability of stuttering frequency scores was acceptable for the SBA and UBA groups; however, the SBA group demonstrated only modest point-by-point agreement with ratings from the experienced evaluators.

CONCLUSIONS

Given its accuracy and efficiency advantages over syllable-based analysis, utterance-based fluency analysis appears to be an appropriate context for introducing stuttering frequency measurement to raters who have limited experience in stuttering measurement. To address accuracy gaps between experienced and inexperienced raters, however, use of either analysis must be supplemented with training activities that expose inexperienced raters to the decision-making processes used by experienced raters when identifying stuttered syllables.

Comparison of audio and audiovisual measures of adult stuttering: Implications for clinical trials

PURPOSE

This study investigated whether measures of percentage syllables stuttered (%SS) and stuttering severity ratings with a 9-point scale differ when made from audiovisual compared with audio-only recordings.

METHOD

Four experienced speech-language pathologists measured %SS and assigned stuttering severity ratings to 10-minute audiovisual and audio-only recordings of 36 adults.

RESULT

There was a mean 18% increase in %SS scores when samples were presented in audiovisual compared with audio-only mode. This result was consistent across both higher and lower %SS scores and was found to be directly attributable to counts of stuttered syllables rather than the total number of syllables. There was no significant difference between stuttering severity ratings made from the two modes.

CONCLUSION

In clinical trials research, when using %SS as the primary outcome measure, audiovisual samples would be preferred as long as clear, good quality, front-on images can be easily captured. Alternatively, stuttering severity ratings may be a more valid measure to use as they correlate well with %SS and values are not influenced by the presentation mode.

Event- and interval-based measurement of stuttering: a review

BACKGROUND

Event- and interval-based measurements are two different ways of computing frequency of stuttering. Interval-based methodology emerged as an alternative measure to overcome problems associated with reproducibility in the event-based methodology. No review has been made to study the effect of methodological factors in interval-based absolute reliability data or to compute the agreement between the two methodologies in terms of inter-judge, intra-judge and accuracy (i.e., correspondence between raters' scores and an established criterion).

AIMS

To provide a review related to reproducibility of event-based and time-interval measurement, and to verify the effect of methodological factors (training, experience, interval duration, sample presentation order and judgment conditions) on agreement of time-interval measurement; in addition, to determine if it is possible to quantify the agreement between the two methodologies

METHODS & PROCEDURES

The first two authors searched for articles on ERIC, MEDLINE, PubMed, B-on, CENTRAL and Dissertation Abstracts during January-February 2013 and retrieved 495 articles. Forty-eight articles were selected for review. Content tables were constructed with the main findings.

MAIN CONTRIBUTION

Articles related to event-based measurements revealed values of inter- and intra-judge greater than 0.70 and agreement percentages beyond 80%. The articles related to time-interval measures revealed that, in general, judges with more experience with stuttering presented significantly higher levels of intra- and inter-judge agreement. Inter- and intra-judge values were beyond the references for high reproducibility values for both methodologies. Accuracy (regarding the closeness of raters' judgements with an established criterion), intra- and inter-judge agreement were higher for trained groups when compared with non-trained groups. Sample presentation order and audio/video conditions did not result in differences in inter- or intra-judge results. A duration of 5 s for an interval appears to be an acceptable agreement. Explanation for high reproducibility values as well as parameter choice to report those data are discussed.

CONCLUSIONS & IMPLICATIONS

Both interval- and event-based methodologies used trained or experienced judges for inter- and intra-judge determination and data were beyond the references for good reproducibility values. Inter- and intra-judge values were reported in different metric scales among event- and interval-based methods studies, making it unfeasible to quantify the agreement between the two methods.

Construct validity of modified time-interval analysis in measuring stuttering and trained speaking patterns

PURPOSE

The aim of the present study was to examine the construct validity of modified time-interval analysis. This measure allows judgments on stuttered and fluent speech as well as trained speaking patterns and might be valuable for outcome studies.

METHOD

Construct validity was investigated in an intervention study with two treatment groups (24 clients received stuttering modification treatment, 30 clients received fluency modification treatment) and a control group (38 stuttering adults). All participants were interviewed during surprise phone calls before and after treatment; the speech samples were analyzed by means of modified time-interval analysis and stuttering frequency counts.

RESULTS

The outcomes confirmed prior hypotheses for the most part. First, the amount of trained speaking patterns after therapy was significantly higher in both treatment groups than in the control group. Secondly, longitudinal changes in the treatment groups met prior expectations based on differing treatment goals and exceeded the changes in the control group. Modified time-interval analysis was sufficiently sensitive to detect changes of speech fluency, but underestimated spontaneous fluent speech when trained speaking patterns were applied.

CONCLUSION

The present study supports construct validity of modified time-interval analysis in measuring stuttering and trained speaking patterns, but also reveals a lack of accuracy.

EDUCATIONAL OBJECTIVES

At the end of this activity the reader will be able to (a) explain different forms of validity in relation to the use of modified time-interval analysis, (b) evaluate whether construct validity of modified time-interval analysis has been supported by the outcomes of an intervention study and (c) describe the usefulness and limitations of modified time-interval analysis for future research.

The effect of listener experience and anchors on judgments of dysphonia

PURPOSE

To determine the effect of auditory anchors on judgments of overall severity (OS) and vocal effort (VE) in dysphonic speech when judgments are made by experienced and inexperienced listeners, and when self-rated by individuals with dysphonia (speaker-listeners).

METHOD

Twenty individuals with dysphonia and 4 normal controls provided speech recordings. Speaker-listeners judged their own speech samples for OS and VE without auditory anchors, and then in the presence of anchors, using 100-mm visual analog scales (VAS). Twenty inexperienced and 10 experienced listeners evaluated the same speech samples for OS and VE in similar rating conditions. Twenty inexperienced listeners also made judgments of the speech samples solely in anchored conditions.

RESULTS

All listeners judged the speech samples as significantly less severe and effortful in the anchored conditions. No significant effects of anchors or experience were found for intrarater agreement within 7.14 mm on the VAS. Both inexperienced and experienced listeners were significantly less variable and showed improved interrater agreement in the anchored conditions. Anchors significantly improved agreement across groups of listeners, particularly between inexperienced and speaker-listeners.

CONCLUSION

Listeners systematically shift judgments of voice quality in response to auditory anchors. Anchors reduce interrater variability and may improve agreement across some types of listeners.

Measurement of trained speech patterns in stuttering: interjudge and intrajudge agreement of experts by means of modified time-interval analysis

Improved fluency after stuttering therapy is usually measured by the percentage of stuttered syllables. However, outcome studies rarely evaluate the use of trained speech patterns that speakers use to manage stuttering. This study investigated whether the modified time interval analysis can distinguish between trained speech patterns, fluent speech, and stuttered speech. Seventeen German experts on stuttering judged a speech sample on two occasions. Speakers of the sample were stuttering adults, who were not undergoing therapy, as well as participants in a fluency shaping and a stuttering modification therapy. Results showed satisfactory inter-judge and intra-judge agreement above 80%. Intervals with trained speech patterns were identified as consistently as stuttered and fluent intervals. We discuss limitations of the study, as well as implications of our findings for the development of training for identification of trained speech patterns and future outcome studies.

EDUCATIONAL OBJECTIVES

The reader will be able to (a) explain different methods to measure the use of trained speech patterns, (b) evaluate whether German experts are able to discriminate intervals with trained speech patterns reliably from fluent and stuttered intervals and (c) describe how the measurement of trained speech patterns can contribute to outcome studies.

Does language influence the accuracy of judgments of stuttering in children?

PURPOSE

To determine whether stuttering judgment accuracy is influenced by familiarity with the stuttering speaker's language.

METHOD

Audiovisual 7-min speech samples from nine 3- to 5-year-olds were used. Icelandic children who stutter (CWS), preselected for different levels of stuttering, were subdivided into 5-s intervals. Ten experienced Icelandic speech-language pathologists (ICE-SLPs) and 10 experienced U.S. speech-language pathologists (US-SLPs), the latter being unfamiliar with the Icelandic language, independently judged each 5-s interval (n = 756) as stuttered or nonstuttered on 2 separate occasions.

RESULTS

As in previous studies, intervals judged to contain stuttering showed wide variability within the ICE-SLP and US-SLP groups. However, both SLP groups (a) displayed satisfactory mean intrajudge agreement, (b) met an independent stuttering judgment accuracy criterion test using English-speaking CWS samples, and (c) met an agreement criterion on approximately 90% of their stuttering and nonstuttering judgments on the Icelandic-speaking CWS samples.

CONCLUSION

Experienced SLPs were shown to be highly accurate in recognizing stuttering and nonstuttering exemplars from young CWS speaking in an unfamiliar language. The findings suggest that judgments of occurrences of stuttering in CWS are not generally language dependent, although some exceptions were noted.

The effect of stuttering measurement training on judging stuttering occurrence in preschool children who stutter

PURPOSE

To evaluate the efficacy of a standardized training program to improve preschool teachers' ability to identify occurrences of stuttering accurately and reliably in preschool children who stutter (CWS).

METHOD

An Icelandic version of the Stuttering Measurement Assessment and Training (SMAAT) program [Ingham, R. J., Cordes, A. K., Kilgo, M., & Moglia, R. (1998). Stuttering measurement assessment and training (SMAAT). Santa Barbara, CA: University of California, Santa Barbara] was developed using 2-min audio-visual recordings of nine Icelandic-speaking CWS (3-5 years). Twenty preschool teachers from preschools in Iceland volunteered to participate and were randomly allocated to an experimental and control group. The preschool teachers judged stuttering on nine pre-judged, interval-classified speech samples on two occasions 2-3 weeks apart; only the experimental group received judgment training between Occasion 1 and 2.

RESULTS

The experimental and control groups displayed, on average, above 80% accuracy in identifying stuttering in the sampled CWS, even prior to training. After training the experimental group showed significantly higher mean percent judgment accuracy (p<.01), while the control group's mean accuracy level showed no significant difference between Occasion 1 and 2.

CONCLUSIONS

The interval-based training program did improve the accuracy with which the judges identified stuttering in preschool CWS. However, the findings also showed that the preschool teachers possessed relatively satisfactory stuttering judgment skills, even without training.

EDUCATIONAL OBJECTIVES

The reader will be able to (a) explain the development of a procedure for establishing standardized training material for measuring and identifying stuttering in preschool children, (b) evaluate whether preschool teachers are generally accurate judges of stuttering in young children, and (c) describe how their stuttering judgment accuracy can be improved through training.

Characteristics of stuttering-like disfluencies in Dutch-speaking children

The purpose of this study was to compare the characteristics of stuttering-like disfluencies in a group of native Dutch-speaking children who stutter (n=693), with a group of normally fluent children (n=79). Methods involved the observation of stuttering-like disfluencies in participants' conversational speech samples (total 77,200 words), particularly the frequency, duration and physical tension of instances of stuttering. Findings indicate that stuttering-like disfluencies exhibited by children who stutter are significantly more frequent, longer in duration and involve more physical tension when compared to those of normally fluent children. Furthermore, applying a criterion of 3% stuttering-like disfluencies to distinguish stuttering from normally fluent children resulted in a high degree of sensitivity (0.9452) and specificity (0.9747). Results were taken to suggest that characteristics of stuttering-like disfluencies of Dutch-speaking children are similar to those of English-speaking children and that talker group membership criteria for childhood stuttering can reasonably be extrapolated from the Dutch to the English language and vice versa.

EDUCATIONAL OBJECTIVES

The reader will be able to: (1) describe characteristics of stuttering-like disfluencies, (2) define properties such as frequency, duration and physical tension for stuttering children and normally fluent children, and (3) make use of data on sensitivity and specificity of the criterion of 3% stuttering-like disfluencies to distinguish stuttering and normally fluent children.

Comparing judgments of stuttering made by students, clinicians, and highly experienced judges

PURPOSE

The purpose of this study was to compare judgments of stuttering made by students and clinicians with previously available judgments made by highly experienced judges in stuttering.

METHOD

On two occasions, 41 university students and 31 speech-language pathologists judged the presence or absence of stuttering in each of 216 audiovisually recorded 5-s intervals of the speech of adults who stutter. Intrajudge and interjudge agreement were calculated, and comparisons were made to judgments previously made about the same recordings by 10 highly experienced judges of stuttering.

RESULTS

Students and clinicians showed similar and relatively high levels of intrajudge and interjudge agreement, but both students and clinicians identified less than half as much stuttering as the highly experienced judges had identified.

CONCLUSIONS

These results replicate previous findings of high agreement coexisting with low accuracy in students' judgments of stuttering, extending those findings to show that similar problems are evident in judgments made by practicing clinicians. Implications include the need for explicit stuttering judgment training programs for both students and practicing clinicians.

EDUCATIONAL OBJECTIVES

After reading this article, the reader will be able to: (1) describe different methods for identifying stuttering and possible problems associated with each method; (2) describe two different methods for reporting interjudge reliability; (3) describe how the identification of stuttering differs for student, clinician, and highly experienced judges.

Brain correlates of stuttering and syllable production: gender comparison and replication

This article reports a gender replication study of the P. T. Fox et al. (2000) performance correlation analysis of neural systems that distinguish between normal and stuttered speech in adult males. Positron-emission tomographic (PET) images of cerebral blood flow (CBF) were correlated with speech behavior scores obtained during PET imaging for 10 dextral female stuttering speakers and 10 dextral, age- and sex-matched normally fluent controls. Gender comparisons were made between the total number of voxels per region significantly correlated with speech performance (as in P. T. Fox et al., 2000) plus total voxels per region that were significantly correlated with stutter rate and not with syllable rate. Stutter-rate regional correlates were generally right-sided in males, but bilateral in the females. For both sexes the positive regional correlates for stuttering were in right (R) anterior insula and the negative correlates were in R Brodmann area 21/22 and an area within left (L) inferior frontal gyrus. The female stuttering speakers displayed additional positive correlates in L anterior insula and in basal ganglia (L globus pallidus, R caudate), plus extensive right hemisphere negative correlates in the prefrontal area and the limbic and parietal lobes. The male stuttering speakers were distinguished by positive correlates in L medial occipital lobe and R medial cerebellum. Regions that positively correlated with syllable rate (essentially stutter-free speech) in stuttering speakers and controls were very similar for both sexes. The findings strengthen claims that chronic developmental stuttering is functionally related to abnormal speech-motor and auditory region interactions. The gender differences may be related to differences between the genders with respect to susceptibility (males predominate) and recovery from chronic stuttering (females show higher recovery rates during childhood).

Effects of time-interval judgement training on real-time measurement of stuttering

The purpose of this study was to investigate whether a previously developed interval-based training program could improve judges' stuttering event judgments. Two groups of judges made real-time stuttering event judgments (computer-mouse button presses) in 3 to 6 trials before the response-contingent judgment training program and in another 3 to 6 trials after training, for recordings of 9 adults who stuttered. Their judgments were analyzed in terms of number of stuttering events, duration of stuttering, and 5-s intervals of speech that could be categorized as judged (or not judged) to contain stuttering. Results showed (a) changes in the amount of stuttering identified by the judges; (b) improved correspondence between the judges' identifications of stuttering events and interval-based standards previously developed from judgments made by experienced, authoritative judges; (c) improved correspondence between interval-based analyses of the judges' stuttering judgments and the previously developed standards; (d) improved intrajudge agreement; (e) improved interjudge agreement; and (f) convergence between the 2 judge groups, for samples and speakers used during training tasks and also for other speakers. Some implications of these findings for developing standardized procedures for the real-time measurement of stuttering are discussed.

Methods of interval selection, presence of noise and their effects on detectability of repetitions and prolongations

Accurate methods for locating specific types of stuttering events are necessary for diagnosis, treatment, and prognosis. A factor that could add variability to assessment of stuttering is noise on recordings. The effects of noise were assessed by adding noise to intervals of speech containing all fluent material, fluent material with a repetition, or fluent material with a prolongation. These intervals allow a unique dysfluency response to be made. A statistical analysis of the occurrence of such intervals in spontaneous speech showed that only a limited number of intervals met these criteria. This demonstrated that selecting intervals at random from spontaneous speech (as in time interval analysis procedure) will infrequently lead to a unique and unambiguous dysfluency specification for the interval. Intervals were selected for testing from the intervals that met the stipulated criteria. These were presented for dysfluency judgment when the position of the stuttering within an interval was varied and with different amounts of added noise (no added noise, 3 dB, and 6 dB of noise relative to mean speech amplitude). Accuracy in detecting stuttering type depended on noise level and the stuttering's position in the interval, both of which also depended on the type of stuttering: Noise level affected detection of repetitions more than prolongations: Repetitions were more difficult to detect when they occurred at the end of an interval whereas prolongations were more difficult to detect when they were at the beginning of an interval. The findings underline the importance of adopting rigorous recording standards when speech is to be employed to make stuttering assessments.

Identifying the authoritative judgments of stuttering: comparisons of self-judgments and observer judgments

Reliable and accurate stuttering measurement depends on the existence of unambiguous descriptions or exemplars of stuttered and nonstuttered speech. The development of clinically meaningful and useful exemplars, in turn, requires determining whether persons who stutter judge the same speech to be stuttered that other observers judge to be stuttered. The purpose of these experiments, therefore, was to compare stuttering judgements from several sources: 15 adults who stutter, judging their own spontaneous speech; the same adults who stutter, judging each other's speech; and a panel of 10 authorities on stuttering research and treatment. Judgments were mode under several conditions, including self-judgments made while the speaker was talking and self- and other-judgements made from recordings in continuous and interval formats. Results showed substantial differences in stuttering judgments across speakers, judges, and judgment conditions, but across-task comparisons were complicated by low self-agreement for many judges. Some intervals were judged consistently by all judges to be Stuttered or Nonstuttered, across multiple conditions, but many other intervals were either not assigned replicable judgments or were consistently judged to be Nonstuttered by the speaker who had produced them but were not assigned consistent judgments by other judges. The implications of these findings for stuttering measurement are considered.

Long-term results of an intensive treatment program for adults and adolescents who stutter

In order to determine the long-term effects of an intensive treatment program, 17 adult and 25 adolescent stutterers were tested 2 or 3 times during a 12- to 24-month post-intensive treatment phase. The results of this study are intended to augment and supplement the growing body of evidence about the effects of intensive treatment programs on adult and adolescent stutterers. Follow-up measures included surprise phone calls to clients at home/work and a self-administered Speech Performance Questionnaire. Careful training of speech raters was undertaken to ensure high reliability of speech measures. Results from the phone call samples indicated that about 69% of the subjects maintained a satisfactory level of post-treatment fluency, with an additional 7% maintaining a level that was judged to be marginally satisfactory. On the self-administered Questionnaire, 80% of the subjects rated their speech fluency as good or fair 12 to 24 months after treatment.

Time-interval measurement of stuttering: systematic replication of Ingham, Cordes, and Gow (1993)

The study reported in this paper was designed to replicate and extend the results of an earlier study (Ingham, Cordes, & Gow, 1993) that investigated time-interval judgments of stuttering. Results confirmed earlier findings that interjudge agreement is higher for these interval-recording tasks than has been previously reported for event-based analyses of stuttering judgments or for time-interval analyses of event judgments. Results also confirmed an earlier finding that judges with intrajudge agreement levels of 90% or better show higher interjudge agreement than judges with lower intrajudge agreement scores. This study failed to find differences between audiovisual and audio-only judgment conditions; between relatively experienced and relatively inexperienced student judges; and, most importantly, between the judgments made, and the agreement levels achieved, by judges from two different clinical research settings. The implications of these findings for attempts to develop a reliable measurement method for stuttering are discussed.