Time-interval measurement of stuttering: establishing and modifying judgment accuracy

The reliability of simultaneous versus individual data collection during stuttering assessment

BACKGROUND

Speech-language pathologists often multitask in order to be efficient with their commonly large caseloads. In stuttering assessment, multitasking often involves collecting multiple measures simultaneously.

AIMS

The present study sought to determine reliability when collecting multiple measures simultaneously versus individually.

METHODS & PROCEDURES

Over two time periods, 50 graduate students viewed videos of four persons who stutter (PWS) and counted the number of stuttered syllables and total number of syllables uttered, and rated speech naturalness. Students were randomly assigned to one of two groups: the simultaneous group, in which all measures were gathered during one viewing; and the individual group, in which one measure was gathered per viewing. Relative and absolute intra- and inter-rater reliability values were calculated for each measure.

OUTCOMES & RESULTS

The following results were notable: better intra-rater relative reliability for the number of stuttered syllables for the individual group (intraclass correlation coefficient (ICC) = 0.839) compared with the simultaneous group (ICC = 0.350), smaller intra-rater standard error of measurement (SEM) (i.e., better absolute reliability) for the number of stuttered syllables for the individual group (7.40) versus the simultaneous group (15.67), and better inter-rater absolute reliability for the total number of syllables for the individual group (88.29) compared with the simultaneous group (125.05). Absolute reliability was unacceptable for all measures across both groups.

CONCLUSIONS & IMPLICATIONS

These findings show that judges are likely to be more reliable when identifying stuttered syllables in isolation than when simultaneously collecting them with total syllables spoken and naturalness data. Results are discussed in terms of narrowing the reliability gap between data collection methods for stuttered syllables, improving overall reliability of stuttering measurements, and a procedural change when implementing widely used stuttering assessment protocols.

WHAT THIS PAPER ADDS

What is already known on the subject The reliability of stuttering judgments has been found to be unacceptable across a number of studies, including those examining the reliability of the most popular stuttering assessment tool, the Stuttering Severity Instrument (4th edition). The SSI-4, and other assessment applications, involve collecting multiple measures simultaneously. It has been suggested, but not examined, that collecting measures simultaneously, which occurs in the most popular stuttering assessment protocols, may result in substantially inferior reliability when compared to collecting measures individually. What this paper adds to existing knowledge The present study has multiple novel findings. First, relative and absolute intra-rater reliability were substantially better when stuttered syllables data were collected individually compared to when the same data were collected simultaneously with total number of syllables and speech naturalness data. Second, inter-rater absolute reliability for total number of syllables was also substantially better when collected individually. Third, intra-rater and inter-rater reliability were similar when speech naturalness ratings were given individually compared to when they were given while simultaneously counting stuttered and fluent syllables. What are the potential or actual clinical implications of this work? Clinicians can be more reliable when identifying stuttered syllables individually compared to when they judge stuttering along with other clinical measures of stuttering. In addition, when clinicians and researchers use current popular protocols for assessing stuttering that recommend simultaneous data collection, including the SSI-4, they should instead consider collecting stuttering event counts individually. This procedural change will lead to more reliable data and stronger clinical decision making.

A Point of View About Fluency

PURPOSE

This article presents several potential concerns with the common usage of the term fluency in the study of stuttering and people who stutter (or, as many speakers now prefer, stutterers). Our goal is to bridge gaps between clinicians, researchers, and stutterers to foster a greater sense of collaboration and understanding regarding the words that are used and meanings that are intended.

METHOD

We begin by reviewing the history of the term fluency. We then explore its usage and current connotations to examine whether the term meaningfully describes constructs that are relevant to the study of the stuttering condition.

RESULTS

By highlighting current research and perspectives of stutterers, we conclude that the term fluency (a) is not fully inclusive, (b) encourages the use of misleading measurement procedures, (c) constrains the subjective experience of stuttering within a false binary categorization, and (d) perpetuates a cycle of stigma that is detrimental to stutterers and to the stuttering community as a whole.

CONCLUSIONS

We recommend that researchers and clinicians cease referring to stuttering as a fluency disorder and simply refer to it as stuttering. Furthermore, we recommend that researchers and clinicians distinguish between moments of stuttering (i.e., what stutterers experience when they lose control of their speech or feel stuck) and the overall lived experience of the stuttering condition.

Reliability of judgments of stuttering-related variables: The effect of language familiarity

Previous studies demonstrate mixed results and some methodological limitations regarding judges' ability to reliably assess stuttering-related variables in an unfamiliar language. The present study examined intra- and inter-rater reliability for percent syllables stuttered (%SS), stuttering severity (SEV), syllables per minute (SPM), and speech naturalness (NAT) when English-speaking judges viewed speech samples in English and in a language with which they had no or minimal familiarity (Spanish). Over two time periods, 21 judges viewed eight videos of four bilingual persons who stutter. Data were analyzed for relative and absolute intra- and inter-rater reliability as well as for an effect of language on time period differences. Intra- and inter-rater relative reliability were good or excellent for all measures in both languages, with the exception of inter-rater relative reliability for NAT in both languages and %SS in Spanish. Intra-rater absolute reliability was acceptable in both languages for NAT and SEV and unacceptable in both for SPM and %SS. Inter-rater absolute reliability in both languages was unacceptable for all measures, even with judges with the same training. There was a clinically significant effect of language on %SS scores, but, despite a statistically significant effect of language for SPM and SEV, the differences were not clinically significant. Results indicate that reliability across and within languages varies by measure and is impacted by intra- vs. inter-rater reliability, relative vs. absolute reliability, and language familiarity. Modifications in training may be able to address some of the limitations found, particularly with regard to SPM and NAT.

The efficacy of stuttering measurement training: evaluating two training programs

PURPOSE

Two stuttering measurement training programs currently used for training clinicians were evaluated for their efficacy in improving the accuracy of total stuttering event counting.

METHOD

Four groups, each with 12 randomly allocated participants, completed a pretest-posttest design training study. They were evaluated by their counts of stuttering events on eight 3-min audiovisual speech samples from adults and children who stutter. Stuttering judgment training involved use of either the Stuttering Measurement System (SMS), Stuttering Measurement Assessment and Training (SMAAT) programs, or no training. To test for the reliability of any training effect, SMS training was repeated with the 4th group.

RESULTS

Both SMS-trained groups produced approximately 34% improvement, significantly better than no training or the SMAAT program. The SMAAT program produced a mixed result.

CONCLUSIONS

The SMS program was shown to produce a "medium" effect size improvement in the accuracy of stuttering event counts, and this improvement was almost perfectly replicated in a 2nd group. Half of the SMAAT judges produced a 36% improvement in accuracy, but the other half showed no improvement. Additional studies are needed to demonstrate the durability of the reported improvements, but these positive effects justify the importance of stuttering measurement training.

Event- and interval-based measurement of stuttering: a review

BACKGROUND

Event- and interval-based measurements are two different ways of computing frequency of stuttering. Interval-based methodology emerged as an alternative measure to overcome problems associated with reproducibility in the event-based methodology. No review has been made to study the effect of methodological factors in interval-based absolute reliability data or to compute the agreement between the two methodologies in terms of inter-judge, intra-judge and accuracy (i.e., correspondence between raters' scores and an established criterion).

AIMS

To provide a review related to reproducibility of event-based and time-interval measurement, and to verify the effect of methodological factors (training, experience, interval duration, sample presentation order and judgment conditions) on agreement of time-interval measurement; in addition, to determine if it is possible to quantify the agreement between the two methodologies

METHODS & PROCEDURES

The first two authors searched for articles on ERIC, MEDLINE, PubMed, B-on, CENTRAL and Dissertation Abstracts during January-February 2013 and retrieved 495 articles. Forty-eight articles were selected for review. Content tables were constructed with the main findings.

MAIN CONTRIBUTION

Articles related to event-based measurements revealed values of inter- and intra-judge greater than 0.70 and agreement percentages beyond 80%. The articles related to time-interval measures revealed that, in general, judges with more experience with stuttering presented significantly higher levels of intra- and inter-judge agreement. Inter- and intra-judge values were beyond the references for high reproducibility values for both methodologies. Accuracy (regarding the closeness of raters' judgements with an established criterion), intra- and inter-judge agreement were higher for trained groups when compared with non-trained groups. Sample presentation order and audio/video conditions did not result in differences in inter- or intra-judge results. A duration of 5 s for an interval appears to be an acceptable agreement. Explanation for high reproducibility values as well as parameter choice to report those data are discussed.

CONCLUSIONS & IMPLICATIONS

Both interval- and event-based methodologies used trained or experienced judges for inter- and intra-judge determination and data were beyond the references for good reproducibility values. Inter- and intra-judge values were reported in different metric scales among event- and interval-based methods studies, making it unfeasible to quantify the agreement between the two methods.

The reliability of a severity rating scale to measure stuttering in an unfamiliar language

With increasing multiculturalism, speech-language pathologists (SLPs) are likely to work with stuttering clients from linguistic backgrounds that differ from their own. No research to date has estimated SLPs' reliability when measuring severity of stuttering in an unfamiliar language. Therefore, this study was undertaken to estimate the reliability of SLPs' use of a 9-point severity rating (SR) scale, to measure severity of stuttering in a language that was different from their own. Twenty-six Australian SLPs rated 20 speech samples (10 Australian English [AE] and 10 Mandarin) of adults who stutter using a 9-point SR scale on two separate occasions. Judges showed poor agreement when using the scale to measure stuttering in Mandarin samples. Results also indicated that 50% of individual judges were unable to reliably measure the severity of stuttering in AE. The results highlight the need for (a) SLPs to develop intra- and inter-judge agreement when using the 9-point SR scale to measure severity of stuttering in their native language (in this case AE) and in unfamiliar languages; and (b) research into the development and evaluation of practice and/or training packages to assist SLPs to do so.

The role of psychological processes in estimates of stuttering severity

PURPOSE

To examine the associations of trait anxiety (STAI), social anxiety (SIAS), depression (BDI-II), and personality features (ADP-IV) with three measures of stuttering severity: %SS, Stuttering Severity, Instrument, and the Overall Assessment of the Speaker's Experience of Stuttering.

METHOD

Fifty adults with a history of stuttering served as participants. Participant scores on trait, anxiety, social anxiety, depression, and personality features were entered into a regression analysis, with the criterion variables (DVs) being: %SS, SSI-3, OASES total score. In order to explore the OASES, further, each of the four OASES subscales were also examined. A separate regression was conducted for, each dependent variable.

RESULTS

The OASES total score model was significant (p<.0001) and revealed that social anxiety and, trait anxiety were the only significant predictors, with medium effect sizes noted for both variables. In contrast, percent syllables stuttered and the SSI were not significantly associated with psychological, variables, suggesting that anxiety may not always be related to overt indicators of stuttering. Depression and personality dysfunction were not significantly associated with any measure of, stuttering severity.

CONCLUSION

Anxiety in the form of social and trait anxiety are significantly associated with stuttering, severity as indicated by the OASES. Traditional procedures for assigning severity ratings to individuals, who stutter based on percent syllables stuttered and the Stuttering Severity Instrument are not, significantly related to psychological processes central to the stuttering experience. Depression and, personality characteristics do not meaningfully account for stuttering.

EDUCATIONAL OBJECTIVES

The reader will be able to: (a) differentiate forms of anxiety that are likely to be associated with stuttering (b) understand the importance of determining features of stuttering that go beyond the obvious, surface characteristics of stuttering frequency, and (c) discuss the important clinical and theoretical implications for understanding the degree of psychological dysfunction that is likely to be characteristic of those who stutter.

Construct validity of modified time-interval analysis in measuring stuttering and trained speaking patterns

PURPOSE

The aim of the present study was to examine the construct validity of modified time-interval analysis. This measure allows judgments on stuttered and fluent speech as well as trained speaking patterns and might be valuable for outcome studies.

METHOD

Construct validity was investigated in an intervention study with two treatment groups (24 clients received stuttering modification treatment, 30 clients received fluency modification treatment) and a control group (38 stuttering adults). All participants were interviewed during surprise phone calls before and after treatment; the speech samples were analyzed by means of modified time-interval analysis and stuttering frequency counts.

RESULTS

The outcomes confirmed prior hypotheses for the most part. First, the amount of trained speaking patterns after therapy was significantly higher in both treatment groups than in the control group. Secondly, longitudinal changes in the treatment groups met prior expectations based on differing treatment goals and exceeded the changes in the control group. Modified time-interval analysis was sufficiently sensitive to detect changes of speech fluency, but underestimated spontaneous fluent speech when trained speaking patterns were applied.

CONCLUSION

The present study supports construct validity of modified time-interval analysis in measuring stuttering and trained speaking patterns, but also reveals a lack of accuracy.

EDUCATIONAL OBJECTIVES

At the end of this activity the reader will be able to (a) explain different forms of validity in relation to the use of modified time-interval analysis, (b) evaluate whether construct validity of modified time-interval analysis has been supported by the outcomes of an intervention study and (c) describe the usefulness and limitations of modified time-interval analysis for future research.

Quantifying dysphonia severity using a spectral/cepstral-based acoustic index: Comparisons with auditory-perceptual judgements from the CAPE-V

This study investigated the relationship between acoustic spectral/cepstral measures and listener severity ratings in normal and disordered voice samples. CAPE-V sentence samples and the vowel /a/were elicited from eight normal speakers and 24 patients with varying degrees of dysphonia severity. Samples were analysed for measures of the cepstral peak prominence (CPP), the ratio of low-to-high spectral energy, and their respective standard deviations. Perceptual ratings of overall severity were also obtained for all samples. Results showed that all acoustic variables combined in a four-factor model which correlated with perceived severity with R = 0.81 (R(2) = 0.65). For the vowel /a/, a five-factor model incorporating all acoustic variables and gender correlated with perceived severity with R = 0.96 (R(2) = 0.91). Results indicate that a strong relationship between perceptual and acoustic estimates of dysphonia severity can be achieved in both continuous speech and vowel contexts using a model incorporating spectral/cepstral measures.

Estimating dysphonia severity in continuous speech: application of a multi-parameter spectral/cepstral model

The purpose of the study was to identify a sub-set of spectral/cepstral-based analysis methods that would most effectively predict dysphonia severity (as estimated via auditory-perceptual analysis) in samples of continuous speech. Acoustic estimates of dysphonia severity were used as an objective treatment outcomes measure in a set of pre- vs post-treatment speech samples. Pre- and post-treatment continuous speech samples from 104 females with primary muscle tension dysphonia (MTD) were rated by listeners using a 100 point visual analogue scale (VAS) and analysed acoustically with spectral/cepstral-based measures. Stepwise linear regression produced a three-factor model consisting of the cepstral peak prominence (CPP); the mean ratio of low-to-high frequency spectral energy; and the standard deviation of the ratio of low-to-high frequency spectral energy that was strongly correlated with perceived dysphonia severity ratings (R = .85; R2 = .73). Mean differences between predicted vs perceptual ratings for pre- and post-treatment speech samples were < 6 points on the 100 point VAS; mean absolute differences between predicted and perceived ratings were < 16 points on the 100 point VAS (equivalent to within one scale value on commonly used 7-point equal-appearing interval rating scales). A multi-parameter acoustic model consisting of spectral/cepstral-based measures shows considerable promise as an objective measure of dysphonia severity in continuous speech, even across the diverse voice types and severities observed in pre- and post-treatment MTD speech samples.

The effect of stuttering measurement training on judging stuttering occurrence in preschool children who stutter

PURPOSE

To evaluate the efficacy of a standardized training program to improve preschool teachers' ability to identify occurrences of stuttering accurately and reliably in preschool children who stutter (CWS).

METHOD

An Icelandic version of the Stuttering Measurement Assessment and Training (SMAAT) program [Ingham, R. J., Cordes, A. K., Kilgo, M., & Moglia, R. (1998). Stuttering measurement assessment and training (SMAAT). Santa Barbara, CA: University of California, Santa Barbara] was developed using 2-min audio-visual recordings of nine Icelandic-speaking CWS (3-5 years). Twenty preschool teachers from preschools in Iceland volunteered to participate and were randomly allocated to an experimental and control group. The preschool teachers judged stuttering on nine pre-judged, interval-classified speech samples on two occasions 2-3 weeks apart; only the experimental group received judgment training between Occasion 1 and 2.

RESULTS

The experimental and control groups displayed, on average, above 80% accuracy in identifying stuttering in the sampled CWS, even prior to training. After training the experimental group showed significantly higher mean percent judgment accuracy (p<.01), while the control group's mean accuracy level showed no significant difference between Occasion 1 and 2.

CONCLUSIONS

The interval-based training program did improve the accuracy with which the judges identified stuttering in preschool CWS. However, the findings also showed that the preschool teachers possessed relatively satisfactory stuttering judgment skills, even without training.

EDUCATIONAL OBJECTIVES

The reader will be able to (a) explain the development of a procedure for establishing standardized training material for measuring and identifying stuttering in preschool children, (b) evaluate whether preschool teachers are generally accurate judges of stuttering in young children, and (c) describe how their stuttering judgment accuracy can be improved through training.

Identification of children's stuttered and nonstuttered speech by highly experienced judges: binary judgments and comparisons with disfluency-types definitions

PURPOSE

The purposes of this study were (a) to determine whether highly experienced clinicians and researchers agreed with each other in judging the presence or absence of stuttering in the speech of children who stutter and (b) to determine how those binary stuttered/nonstuttered judgments related to categorizations of the same speech based on disfluency-types descriptions of stuttering.

METHOD

Eleven highly experienced judges made binary judgments of the presence or absence of stuttering for 600 audiovisually recorded 5-s speech samples from twenty 2- to 8-year-old children who stuttered. These judgments were compared with each other and with disfluency-types judgments in multiple interval-by-interval assessments and by using multiple definitions of agreement.

RESULTS

Interjudge agreement for the highly experienced judges in the binary stuttered/nonstuttered task varied from 39.0% to 89.1%, depending on methods and definitions used. Congruence between binary judgments and categorizations based on disfluency types also varied depending on methods and definitions, from 21.6% to 100%.

CONCLUSIONS

Agreement among highly experienced judges, and congruence between their binary judgments of stuttering and categorizations based on disfluency types, were relatively high using some definitions and very low using others. These results suggest the use of measurement methods other than those based on disfluency types for quantifying or describing children's stuttering. They also suggest both the need for, and potential methods for, training to increase judges' accuracy and agreement in identifying children's stuttering.

Reliability issues and solutions for coding social communication performance in classroom settings

PURPOSE

To explore the utility of time-interval analysis for documenting the reliability of coding social communication performance of children in classroom settings. Of particular interest was finding a method for determining whether independent observers could reliably judge both occurrence and duration of ongoing behavioral dimensions for describing social communication performance.

METHOD

Four coders participated in this study. They observed and independently coded 6 social communication behavioral dimensions using handheld computers. The dimensions were mutually exclusive and accounted for all verbal and nonverbal productions during a specified time frame. The technology allowed for coding frequency and duration for each entered code. Data were collected from 20 different 2-min video segments of children in kindergarten through 3rd-grade classrooms. Data were analyzed for interobserver and intraobserver agreements using time-interval sorting and Cohen's kappa. Further, interval size and total observation length were manipulated to determine their influence on reliability.

RESULTS

The data revealed interval sorting and kappa to be a suitable method for examining reliability of occurrence and duration of ongoing social communication behavioral dimensions. Nearly all comparisons yielded medium to large kappa values; interval size and length of observation minimally affected results. Implications The analysis procedure described in this research solves a challenge in reliability: comparing coding by independent observers of both occurrence and duration of behaviors. Results indicate the utility of a new coding taxonomy and technology for application in online observations of social communication in a classroom setting.

Comparing judgments of stuttering made by students, clinicians, and highly experienced judges

PURPOSE

The purpose of this study was to compare judgments of stuttering made by students and clinicians with previously available judgments made by highly experienced judges in stuttering.

METHOD

On two occasions, 41 university students and 31 speech-language pathologists judged the presence or absence of stuttering in each of 216 audiovisually recorded 5-s intervals of the speech of adults who stutter. Intrajudge and interjudge agreement were calculated, and comparisons were made to judgments previously made about the same recordings by 10 highly experienced judges of stuttering.

RESULTS

Students and clinicians showed similar and relatively high levels of intrajudge and interjudge agreement, but both students and clinicians identified less than half as much stuttering as the highly experienced judges had identified.

CONCLUSIONS

These results replicate previous findings of high agreement coexisting with low accuracy in students' judgments of stuttering, extending those findings to show that similar problems are evident in judgments made by practicing clinicians. Implications include the need for explicit stuttering judgment training programs for both students and practicing clinicians.

EDUCATIONAL OBJECTIVES

After reading this article, the reader will be able to: (1) describe different methods for identifying stuttering and possible problems associated with each method; (2) describe two different methods for reporting interjudge reliability; (3) describe how the identification of stuttering differs for student, clinician, and highly experienced judges.

Effects of time-interval judgement training on real-time measurement of stuttering

The purpose of this study was to investigate whether a previously developed interval-based training program could improve judges' stuttering event judgments. Two groups of judges made real-time stuttering event judgments (computer-mouse button presses) in 3 to 6 trials before the response-contingent judgment training program and in another 3 to 6 trials after training, for recordings of 9 adults who stuttered. Their judgments were analyzed in terms of number of stuttering events, duration of stuttering, and 5-s intervals of speech that could be categorized as judged (or not judged) to contain stuttering. Results showed (a) changes in the amount of stuttering identified by the judges; (b) improved correspondence between the judges' identifications of stuttering events and interval-based standards previously developed from judgments made by experienced, authoritative judges; (c) improved correspondence between interval-based analyses of the judges' stuttering judgments and the previously developed standards; (d) improved intrajudge agreement; (e) improved interjudge agreement; and (f) convergence between the 2 judge groups, for samples and speakers used during training tasks and also for other speakers. Some implications of these findings for developing standardized procedures for the real-time measurement of stuttering are discussed.

Identifying the authoritative judgments of stuttering: comparisons of self-judgments and observer judgments

Reliable and accurate stuttering measurement depends on the existence of unambiguous descriptions or exemplars of stuttered and nonstuttered speech. The development of clinically meaningful and useful exemplars, in turn, requires determining whether persons who stutter judge the same speech to be stuttered that other observers judge to be stuttered. The purpose of these experiments, therefore, was to compare stuttering judgements from several sources: 15 adults who stutter, judging their own spontaneous speech; the same adults who stutter, judging each other's speech; and a panel of 10 authorities on stuttering research and treatment. Judgments were mode under several conditions, including self-judgments made while the speaker was talking and self- and other-judgements made from recordings in continuous and interval formats. Results showed substantial differences in stuttering judgments across speakers, judges, and judgment conditions, but across-task comparisons were complicated by low self-agreement for many judges. Some intervals were judged consistently by all judges to be Stuttered or Nonstuttered, across multiple conditions, but many other intervals were either not assigned replicable judgments or were consistently judged to be Nonstuttered by the speaker who had produced them but were not assigned consistent judgments by other judges. The implications of these findings for stuttering measurement are considered.

Experimental investigation of the effects of frequency-altered auditory feedback on the speech of adults who stutter

A series of single-subject experiments evaluated the effects of frequency-altered auditory feedback (FAF) on the speech performance of four adult males who stutter. Using alterations of plus or minus one octave, FAF was compared with normal auditory feedback (NAF) in oral reading and spontaneous speech with measurements made of stuttered intervals, stutter-free speech rate, and speech naturalness. The effects of extended FAF conditions on spontaneous speech were also evaluated for two subjects who demonstrated a positive response to FAF. Results showed no consistencies across subjects in responses to FAF: One subject showed no response, another produced an initial temporary response, a third showed a deterioration in speech quality with minimal reductions in stuttering, and a fourth displayed substantial and sustained improvements in speech performance. Some implications of these findings for current research and theory about the relationship between stuttering and FAF are discussed.