Intervention effects and relative variation as dimensions in experts' use of visual inference

Single-case intervention research design standards: Additional proposed upgrades and future directions

Single-case intervention research design standards have evolved considerably over the past decade. These standards serve the dual role of assisting in single-case design (SCD) intervention research methodology and as guidelines for literature syntheses within a particular research domain. In a recent article (Kratochwill et al., 2021), we argued for a need to clarify key features of these standards. In this article we offer additional recommendations for SCD research and synthesis standards that have been either underdeveloped or missing in the conduct of research and in literature syntheses. Our recommendations are organized into three categories: expanding design standards, expanding evidence standards, and expanding the applications and consistency of SCDs. The recommendations we advance are for consideration for future standards, research design training, and they are especially important to guide the reporting of SCD intervention investigations as they enter the literature-synthesis phase of evidence-based practice initiatives.

The influence of data characteristics on interrater agreement among visual analysts

Visual analysis is the primary method of analyzing single-case research data, yet relatively little is known about the variables that influence raters' decisions and rater agreement. Previous research has suggested that trend, variability, and autocorrelation may negatively affect interrater agreement, but studies have been limited by small numbers of graphs and participants whose knowledge of single-case research was not described. The purpose of this study was to examine the main and interaction effects of two values of each of six data characteristics (e.g., level, trend, and number of data points) on agreement among visual analysts. Using data from Lanovaz and Hranchuk (2021), we examined odds ratios to identify data characteristics that influence interrater agreement. Results suggest that trend and effect size, and to a lesser extent variability, have the largest effects on interrater agreement. We discuss the implications of our results for future research on improving interrater agreement among visual analysts.

Comparison of visual analysis, non-overlap methods, and effect sizes in the evaluation of parent implemented functional assessment based interventions

We used an existing body of research (i.e., parent implemented functional-assessment based interventions) to examine visual analysis features and processes and evaluate the reliability of two frequently used non-overlap indices (NAP & Tau-U) and a novel effect size index-the between-case standardized mean difference (BC-SMD). Results indicated that visual analysis terms and procedures were inconsistently used across studies. Further, there was limited agreement between the non-overlap indices and independent visual analysis. Results regarding the BC-SMD were inconclusive given only 5 of the 15 studies were eligible for analyses for different dependent variables. Our results suggest that visual analysis standards are needed by which single case researchers analyze and report their results. Further, additional research is needed refining SCR effect sizes, which can be used to describe the magnitude of change within and across SCR studies with functional relations.

Systematic Protocols for the Visual Analysis of Single-Case Research Data

Researchers in applied behavior analysis and related fields such as special education and school psychology use single-case designs to evaluate causal relations between variables and to evaluate the effectiveness of interventions. Visual analysis is the primary method by which single-case research data are analyzed; however, research suggests that visual analysis may be unreliable. In the absence of specific guidelines to operationalize the process of visual analysis, it is likely to be influenced by idiosyncratic factors and individual variability. To address this gap, we developed systematic, responsive protocols for the visual analysis of A-B-A-B and multiple-baseline designs. The protocols guide the analyst through the process of visual analysis and synthesize responses into a numeric score. In this paper, we describe the content of the protocols, illustrate their application to 2 graphs, and describe a small-scale evaluation study. We also describe considerations and future directions for the development and evaluation of the protocols.

Context-Sensitive Heuristics in Statistical Reasoning

Previous work has shown that people use anchor-and-adjust heuristics to forecast future data points from previous ones in the same series. We report three experiments that show that they use different versions of this heuristic for different types of series. To forecast an untrended series, our subjects always took a weighted average of the long-term mean of the series and the last data point. In contrast, the way that they forecast a trended series depended on the serial dependences in it. When these were low, people forecast by adding a proportion of the last difference in the series to the last data point. When stronger serial dependences made this difference less similar to the next one, they used a version of the averaging heuristic that they employed for untrended series. This could take serial dependences into account and included a separate component for trend. These results suggest that people use a form of the heuristic that is well adapted to the nature of the series that they are forecasting. However, we also found that the size of their adjustments tended to be suboptimal. They overestimated the degree of serial dependence in the data but underestimated trends. This biased their forecasts.

Perception of Linear and Nonlinear Trends: Using Slope and Curvature Information to Make Trend Discriminations

This study investigated several factors influencing the perception of nonlinear relationships in time series graphs. To model real-world data, the graphed data represented different underlying trends and included different sample sizes and amounts of variability. Six trends (increasing and decreasing linear, exponential, asymptotic) were presented on four graph types (histogram, line graph, scatterplot, suspended bar graph). The experiment assessed how these factors affect trend discrimination, with the overall goal of judging what types of graphs lead to better discrimination. Six participants (two psychology professors, four psychology graduate students) viewed graphs on a computer screen and identified the underlying trend. All participants were familiar with the types of trends presented and were aware of the purpose of the experiment. Analysis indicated higher accuracy when variability was lower and sample size was higher. Choice accuracy was higher for nonlinear trends and was highest when line graphs were used.

Visual Analysis of Student Evaluation Data: A Qualitative Analysis of Teacher Decision Making

Visual analysis of graphically displayed data typically is used by teachers of persons with severe disabilities to evaluate the effectiveness of educational interventions. Although the use of this process is widely advocated as a desirable educational practice, recent research indicates that levels of interrater agreement are unacceptably low. To examine the variables that influence the decisions made during visual appraisal in classroom settings, qualitative interviews were conducted with special education teachers. The implications of these results for teachers who collect and analyze student performance data in a classroom setting are discussed.

An Empirical Investigation of Interrater Agreement for Single-Subject Data Using Graphs with and without Trend Lines

The interrater agreement of visual judgments made from single-subject data was examined. Seventy-nine raters were given 21 single-subject graphs. Thirty-nine of the raters examined graphs containing single-subject data arrayed in the traditional format. The remaining 40 subjects reviewed AB graphs that were supplemented by a trend line. As measured by intraclass correlation coefficients, interrater agreement was higher for the trend line group than for the group relying only on visual analysis. There was a statistically significant correlation between the change in slope across the phases of the AB design and a score reflecting disagreement among raters in the visual analysis group. This relationship between change in slope and rater disagreement was not present in the trend line group. The results suggest that the low interrater agreement often associated with visual analysis of single-subject data may be improved by simple supplements to visually inspected charts.

Characteristics of Time-Series Data Collected Through Curriculum-Based Reading Measurement

Research on the analysis of time-series data has shown that decisions reached through visual analysis of the data may be influenced by the statistical parameters of those data. This study investigated the statistical properties of curriculum-based time-series data for 67 resource room students in three Minnesota school districts. Data for slope, standard error of estimate, and mean level of performance are presented. Results suggest that reading growth over time may be described by a negatively accelerated curve and that the statistical characterisics of time-series data are not necessarily independent in naturally occurring data. Implications for both time-series and conventional pre-post designs are discussed.

The Effect of Serial Dependence on Visual Judgment of Single-Case Charts: An Addendum

This study reanalyzed data presented in a previous investigation (Ottenbacher, 1986a) that explored the relationship between graphic characteristics and visual judgments in single-case charts. Specifically, Ottenbacher computed autocorrelation coefficients for five graphs and examined the relationship between serial dependence and rater agreement or disagreement. The method used by Ottenbacher (1986a) was adopted from previous research and involved analyzing data points combined across baseline and treatment phases. The argument is made that this analysis is incorrect and that serial dependence should be computed using only the baseline data. Reanalysis of the data presented by the previous investigation reveals a strong association (.94) between degree of serial dependence and rater disagreement. The concept of serial dependence and its influence on the analysis of single-case graphs is discussed and the appropriate method of computing serial dependence is presented.

The Relationship Between Visual Analysis and Five Statistical Analyses in a Simple AB Single-Case Research Design

This study explored some practical issues for single-case researchers who rely on visual analysis of graphed data, but who also may consider supplemental use of promising statistical analysis techniques. The study sought to answer three major questions: (a) What is a typical range of effect sizes from these analytic techniques for data from “effective interventions”? (b) How closely do results from these same analytic techniques concur with visual-analysis-based judgments of effective interventions? and (c) What role does autocorrelation play in interpretation of these analytic results? To answer these questions, five analytic techniques were compared with the judgments of 45 doctoral students and faculty, who rated intervention effectiveness from visual analysis of 35 fabricated AB design graphs. Implications for researchers and practitioners using single-case designs are discussed.

The Effects of Celeration Lines on Visual Data Analysis

Previous visual analysis research reported that the overall agreement between visual analysis and statistical analysis was poor. In response, some researchers suggested the use of celeration lines to improve the accuracy and reliability of visual analysis. However, subsequent research reported little or no improvement in accuracy with such lines. The present study presented 5 board-certified behavior analysts with a series of behavioral graphs. The participants were asked to answer questions similar to those posed in previous studies but were also asked to talk aloud as they viewed each graph. Results indicate that the participants made accurate decisions for only 72% of the graphs and that celeration lines did not improve overall accuracy. The verbal protocol analysis suggests that participants were as likely to attend to trend when celeration lines were absent as they were when they were present, with the most differences attributable to varying participant competencies and not graph (i.e., celeration line) characteristics.

Interrater Agreement Between Visual Analysts of Single-Case Data

Visual analysis is the most widely applied method of data interpretation for single-case research as it encompasses multifaceted considerations relevant to evaluating behavior change. However, a previous research synthesis found low levels of interrater agreement between visually analyzed ratings of graphed data across all variables under analysis. The purpose of this meta-analysis was to evaluate the peer-reviewed literature to date for potential moderators affecting the proportion of interrater agreement between visual analysts. Nineteen articles with 32 effects were assembled. Potential moderators evaluated included (a) design families, (b) rater expertise, (c) the provision of contextual information for graphs, (d) the use of visual aids, (e) the provision of an operational definition of the construct being rated, and (f) rating scale ranges. Results yielded an overall weighted interrater agreement proportion of .76. Moderator variables identified produced low to adequate levels of interrater agreement. Practical recommendations for future research are discussed.

Factors Affecting Visual Inference in Single-Case Designs

Visual inspection remains the most frequently applied method for detecting treatment effects in single-case designs. The advantages and limitations of visual inference are here discussed in relation to other procedures for assessing intervention effectiveness. The first part of the paper reviews previous research on visual analysis, paying special attention to the validation of visual analysts' decisions, inter-judge agreement, and false alarm and omission rates. The most relevant factors affecting visual inspection (i.e., effect size, autocorrelation, data variability, and analysts' expertise) are highlighted and incorporated into an empirical simulation study with the aim of providing further evidence about the reliability of visual analysis. Our results concur with previous studies that have reported the relationship between serial dependence and increased Type I rates. Participants with greater experience appeared to be more conservative and used more consistent criteria when assessing graphed data. Nonetheless, the decisions made by both professionals and students did not match sufficiently the simulated data features, and we also found low intra-judge agreement, thus suggesting that visual inspection should be complemented by other methods when assessing treatment effectiveness.

Visual inspection of data revisited: Do the eyes still have it?

In behavior analysis, visual inspection of graphic information is the standard by which data are evaluated. Efforts to supplement visual inspection using inferential statistical procedures to assess intervention effects (e.g., analysis of variance or time-series analysis) have met with opposition. However, when serial dependence is present in the data, the use of visual inspection by itself may prove to be problematic. Previously published reports demonstrate that autocorrelated data influence trained observers' ability to identify level treatment effects and trends that occur in the intervention phase of experiments. In this report, four recent studies are presented in which autoregressive equations were used to produce point-to-point functions to simulate experimental data. In each study, various parameters were manipulated to assess trained observers' responses to changes in point-to-point functions from the baseline condition to intervention. Level shifts over baseline behavior (treatment effect), as well as no change from baseline (no treatment effect or trend), were most readily identified by observers, but trends were rarely recognized. Furthermore, other factors previously thought to augment and improve observers' responses had no impact. Results are discussed in terms of the use of visual inspection and the training of behavior analysts.

When the Truth Hits You Between the Eyes

Visual data analysis is an important first step when evaluating intervention effects. This also holds for analyzing data from single-case experiments. Because most software packages do not offer customized facilities for constructing single-case graphs and are not particularly suited to perform single-case visual data analyses, we created an R package to help researchers in making graphical representations of single-case data and to transform graphical displays back to raw data. In addition to a basic plotting function, we included some tools to facilitate the use of three interpretative principles for visually analyzing single-case data: plotting a measure of central location as a horizontal reference line; displaying variability with (trimmed) range bars, range lines, and trended ranges; and displaying trends with a vertical line graph, by fitting a robust linear trend, or by plotting running medians. Finally, we included a function to extract raw data values from published graphs.

Data-based decision making: the impact of data variability, training, and context

The current study examines agreement among individuals with varying expertise in behavior analysis about the length of baseline when data were presented point by point. Participants were asked to respond to baseline data and to indicate when to terminate the baseline phase. When only minimal information was provided about the data set, experts and Board Certified Behavior Analyst participants generated baselines of similar lengths, whereas novices did not. Agreement was similar across participants when variability was low but deteriorated as variability in the data set increased. Participants generated shorter baselines when provided with information regarding the independent or dependent variable. Implications for training and the use of visual inspection are discussed.

Effects of lines of progress and semilogarithmic charts on ratings of charted data

The extent to which interrater agreement and ratings of significance on both changes in level and trend are affected by lines of progress and semilogarithmic charts was investigated. Thirteen graduate students rated four sets of charts, each set containing 19 phase changes. Set I data were plotted on equal interval charts. In Set II a line of progress was drawn through each phase on each chart. In Set III data points were replotted on semilogarithmic charts. In Set IV a line of progress was drawn through each phase of each Set III chart. A significant main effect on interrater agreement was found for lines of progress as well as a significant 2-way interaction between lines of progress and change type. Three main effects (chart type, lines of progress, and type of change) and a significant 3-way interaction were found for ratings of significance. Implications of these data for visual analysis of charted data are discussed.

Visual analysis of single-case time series: Effects of variability, serial dependence, and magnitude of intervention effects

Visual analysis is the dominant method of analysis for single-case time series. The literature assumes that visual analysts will be conservative judges. We show that previous research into visual analysis has not adequately examined false alarm and miss rates or the effect of serial dependence. In order to measure false alarm and miss rates while varying serial dependence, amount of random variability, and effect size, 37 students undertaking a postgraduate course in single-case design and analysis were required to assess the presence of an intervention effect in each of 27 AB charts constructed using a first-order autoregressive model. Three levels of effect size and three levels of variability, representative of values found in published charts, were combined with autocorrelation coefficients of 0, 0.3 and 0.6 in a factorial design. False alarm rates were surprisingly high (16% to 84%). Positive autocorrelation and increased random variation both significantly increased the false alarm rates and interacted in a nonlinear fashion. Miss rates were relatively low (0% to 22%) and were not significantly affected by the design parameters. Thus, visual analysts were not conservative, and serial dependence did influence judgment.

A Survey Evaluation of the Reliability of Visual Inspection and Functional Analysis Graphs

Visual inspection is the primary method used to analyze graphed behavioral data produced by functional analyses of problem behavior. The purpose of this study was to examine rater reliability of functional analysis graphs using visual inspection. Forty-three participants responded to a one-time anonymous survey ( N = 454) mailed to graduate programs accredited in applied behavior analysis ( N = 11). Respondents were instructed to classify single-subject data arrays depicted in multielement graphs from published studies. Classification was based on function categories by Hagopian, Fisher, Thompson, and Owen-DeSchryver. Three indices based on overall inter-rater agreement, the consistency of individual raters across graphs, and the aggregate performance of raters per graph were calculated and compared. Results for all three indices of rater performance were moderate to low. Results confirm prior preliminary work indicating relatively low levels of reliability and are discussed in relation to research, training, and practice issues.

Sufficiency and stability of evidence for public health interventions using cumulative meta-analysis

We propose cumulative meta-analysis as the procedure of completing a new meta-analysis at each successive wave in a research database. Two facets of cumulative knowledge are considered: the first, sufficiency, refers to whether the meta-analytic database adequately demonstrates that a public health intervention works. The second, stability, refers to the shifts over time in the accruing evidence about whether a public health intervention works. We used a hypothetical data set to develop the indicators of sufficiency and stability, and then applied them to existing, published datasets. Our discussion centers on the implications of the use of this procedure in evaluating public health interventions.

Psychological treatment of reported sleep disorder in adults with intellectual disability using a multiple baseline design

BACKGROUND

The literature on sleep disturbance in adults with intellectual disability (ID) is sparse. Although prevalence rates for sleep disorders appear similar to those of non-disabled populations, previous treatment studies have largely been comprised of uncontrolled cases. Therefore, the present study adopted a single-case experimental methodology to evaluate behavioural sleep intervention.

METHODS

A screening questionnaire was posted to 384 adults with ID and the sleep pattern of respondents with possible sleep disorders was further assessed using a structured diagnostic schedule. From the sleep-disordered subgroup, 12 participants were selected for a 4-week behavioural sleep intervention that was evaluated using randomly allocated, multiple-baseline, across-subjects designs and within-subject interrupted time series analyses (ITSAs).

RESULTS

A total of 155 adults with ID (83 females and 72 males; mean age = 32 years, SD = 16.5 years), or their carers, completed the questionnaire (return rate = 40%). The application of sleep diagnostic criteria revealed that 17% had clinically significant difficulty getting to sleep and 11% had difficulty remaining asleep. Nine out of the 12 participants recruited for the intervention completed all the experimental phases, thus providing three sets of three multiple-baseline designs. Visual inspection of within- and between-subject effects suggested beneficial treatment-specific effects across a range of target variables. The ITSA confirmed significant effects (P < 0.05) or trends (P < 0.10) for six out of the nine participants.

CONCLUSIONS

Behavioural sleep management may improve sleep pattern or sleep-related functioning in the majority of adults with ID who have significant sleep problems. The single-case methodology is helpful in addressing the heterogeneity of individual presentation, although clinical trial methodology is required to confirm these findings on a larger scale.

Evaluating data from behavioral analysis: visual inspection or statistical models?

Traditional behavior analysis relies upon single-subject study designs and visual inspection of graphed data to evaluate the efficacy of experimental manipulations. Attempts to apply statistical inferential procedures to analyze data have been successfully opposed for many decades, despite problems with visual inspection and increasingly cogent arguments to utilize inferential statistics. In a series of experiments, we show that trained behavior analysts often identify level shifts in responding during intervention phases ('treatment effect') in modestly autocorrelated data, but trends are either misconstrued as level treatment effects or go completely unnoticed. Errors in trend detection illustrate the liabilities of using visual inspection as the sole means by which to analyze behavioral data. Meanwhile, because of greatly increased computer power and advanced mathematical techniques, previously undeveloped or underutilized statistical methods have become far more sophisticated and have been brought to bear on a variety of problems associated with repeated measures data. I present several nonparametric procedures and other statistical techniques to evaluate traditional behavioral data to augment, not replace, visual inspection procedures.

Effectiveness of video feedback and self-management on inappropriate social behavior of youth with mild mental retardation

The effectiveness of a video feedback and self-management package was assessed with various inappropriate behaviors exhibited by six youth with mild mental retardation. The procedure consisted of (a) videotaping participants' inappropriate behavior, (b) having them self-monitor and record their behavior, (c) prompting them to evaluate their behavior against a criterion, and (d) allowing to reinforce themselves for appropriate behaviors. Data were collected within a nonconcurrent multiple baseline design across participants. Results showed a statistically significant decrease of the percentage intervals of inappropriate behavior when the procedure was in effect. The total number of interactions remained stable across the different phases of the study. Video feedback and self-management contributed to generalization across settings.

Matching training procedures to outcomes. A behavioral and quantitative analysis

To analyze the effects of matching the prompting procedure used in training to the specific behavior chain to be taught, 3 students with mild to moderate retardation were taught four independent tasks: making a bagged lunch, playing a matching game with a peer, ordering food at a restaurant, and participating in a social conversation. Following baseline, all 3 students were exposed to one of two types of training procedures for each task: a least-to-most prompting procedure or a most-to-least prompting procedure. The type of training procedure was counter-balanced across students and tasks, whereas performance on the tasks was evaluated within a combination of a multiple-baseline design across participants and multiple-probe design across tasks. When the method of prompting was matched to the naturally occurring discriminative stimulus (SD) of the training stimulus, it greatly affected acquisition and maintenance of the skill in terms of differences in levels of and variability of performance. The most-to-least method of prompting, the matched method in these cases, was more efficient and effective for acquisition and generalization of the bagged-lunch and matching-game skills. The least-to-most method, the matched method in these cases, was more efficient and effective for social questions and ordering-food skills.

Neural network simulation of visual inspection of graphs of single-subject interventions

Judgments of the effectiveness of single-subject behavioral interventions are often based on visual examination of graphs of response data, but previous research indicates that such judgments are often unreliable and flawed. Here it is proposed that artificial neural networks could be developed to simulate the judgments of expert judges. A prototype of such a network was designed and trained in the present study, and its use in novel experiments matched the estimates of the expert whose judgments were simulated significantly better than did a prediction equation developed using a multiple regression approach.

A statistical model for the "N-of-1" study

The controlled clinical trial has largely replaced case-reports as the authoritative source of information concerning the efficacy of treatment. However, many situations arise in clinical practice where treatment decisions cannot be made on the basis of such studies. The definitive clinical trial may not have been performed, or the results from a particular study may not be applicable to a particular patient. Recently, "N-of-1" studies have been proposed for the experimental evaluation of therapy in a single patient. Multiple courses of active and placebo treatments are administered, and efficacy is determined by following the response measure over a period of time. The purpose of this paper is to present a statistical model appropriate for data arising from this design. The model provides for serial correlation among the response measures captured from the subject, and for heteroskedasticity across the treatment periods. ML estimation procedures are considered, and their properties are investigated. A scoring algorithm is described to iterate to the solution of the ML equations, and considerations for hypothesis testing are presented. The techniques are illustrated through an example.

Evaluating the impact of a service fee on patient compliance

This paper addresses an evaluation of an administrative decision to change the manner in which services were paid for at a Student Health Center (SHC). The impact of the change in payment was observed through monitoring the number of scheduled appointments at the SHC which the patient failed to attend, reschedule, or cancel. The impact was assessed through a comparison of the weekly no-show rates from the year prior to the change in payment practices through the year following the change. A time-series statistical package was used to analyze the no-show data. Collateral measures on the number of students attending the university, staff opinions, and usage of the SHC by different student groups were collected. Evaluations of the impact of administrative decisions on health related behavior were discussed, in addition to a discussion of the usefulness of time-series models for this type of evaluation.