Global Measures of Treatment Integrity May Mask Important Errors in Discrete-Trial Training

Training individuals to implement discrete-trial teaching procedures using behavioral skills training: A scoping review with implications for practice and research

Behavioral skills training (BST) is an evidence-based approach for training individuals to implement discrete-trial teaching procedures. Despite the effectiveness of this approach, implementing BST can be time and resource intensive, which may interfere with a clinical organization's adoption of this training format. We conducted a scoping review of studies using BST components for training discrete-trial teaching procedures in peer-reviewed articles between 1977 and 2021. We identified 51 studies in 46 publications involving 354 participants. We coded descriptive data on (a) participant characteristics, (b) study characteristics, (c) training conditions (including instructions, modeling, rehearsal, and feedback), and (d) training outcomes. The results indicated that studies have primarily attempted to improve the efficacy and efficiency of BST by modifying or omitting common training components. We provide best-practice considerations for using BST to teach discrete-trial teaching procedures and offer a research agenda to guide future investigation in this area.

Effects of omission and commission errors during tact instruction

The purpose of the current study was to compare the effects of omission and commission errors of reinforcement during tact instruction via telehealth with three children, 6 to 7 years of age, who were diagnosed with an autism spectrum disorder. We used an adapted alternating treatment design to evaluate skill acquisition of target stimuli across high-integrity, commission errors, and omission errors conditions. The high-integrity condition produced mastery criteria in fewer sessions compared with the integrity-error conditions in four of six comparisons, and the omission condition reached mastery criteria in fewer sessions than the commission condition in five of six comparisons.

A detailed examination of reporting procedural fidelity in the Journal of Applied Behavior Analysis

Few reviews on procedural fidelity-the degree to which procedures are implemented as designed-provide details to gauge the quality of fidelity reporting in behavior-analytic research. This review focused on experiments in the Journal of Applied Behavior Analysis (2006-2021) with "integrity" or "fidelity" in the abstract or body. When fidelity data were collected, the coders characterized measurement details (e.g., description of calculation, report of single or multiple values, frequency of fidelity checks, checklist use). The researchers found increasing trends in describing the calculation(s), reporting multiple values, and stating the frequency of measurement. Few studies described using a checklist. Most studies reported fidelity as a percentage, with high obtained values (M = 97%). When not collecting fidelity data was stated as a limitation, authors were unlikely to provide a rationale for the omission. We discuss recommendations for reporting procedural fidelity to increase the quality of and transparency in behavior-analytic research.

Peter CC, Feng J, Long BP. Trends in Reporting Procedural Integrity: A Comparison

Procedural integrity refers to the extent to which an independent variable is implemented as described. Measuring procedural integrity is one important factor when considering internal and external validity of experiments. Experimental articles in behavior-analytic journals have rarely reported procedural-integrity data. The purpose of this study was to update previous reviews of whether articles published in the Journal of Applied Behavior Analysis reported procedural integrity, spanning a period from 1980 to 2020, and compare reporting in JABA to recent reviews of studies published in Behavior Analysis in Practice (2008-2019) and the Journal of Organizational Behavior Management (2000-2020). Procedural integrity continues to be underreported across all three journals, but an increasing trend in reporting procedural integrity is evident in the Journal of Applied Behavior Analysis and Behavior Analysis in Practice. In addition to our recommendations and implications for research and practice, we provide examples and resources to assist researchers and practitioners with recording and reporting integrity data.

Comparing Multiple Methods to Measure Procedural Fidelity of Discrete-trial Instruction

Procedural fidelity is the extent to which an intervention is implemented as designed and is an important component of research and practice. There are multiple ways to measure procedural fidelity, and few studies have explored how procedural fidelity varies based on the method of measurement. The current study compared adherence to discrete-trial instruction protocols by behavior technicians with a child with autism when observers used different procedural-fidelity measures. We collected individual-component and individual-trial fidelity with an occurrence-nonoccurrence data sheet and compared these scores to global fidelity and all-or-nothing, 3-point Likert scale, and 5-point Likert scale measurement methods. The all-or-nothing method required all instances of a component or trial be implemented without error to be scored correct. The Likert scales used a rating system to score components and trials. At the component level, we found that the global, 3-point Likert, and 5-point Likert methods were likely to overestimate fidelity and mask component errors, and the all-or-nothing method was unlikely to mask errors. At the trial level, we found that the global and 5-point Likert methods approximated individual-trial fidelity, the 3-point Likert method overestimated fidelity, and the all-or-nothing method underestimated fidelity. The occurrence-nonoccurrence method required the most time to complete, and all-or-nothing by trial required the least. We discuss the implications of measuring procedural fidelity with different methods of measurement, including false positives and false negatives, and provide suggestions for practice and research.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43494-023-00094-w.

A parametric analysis of procedural fidelity errors following mastery of a task: A translational study

Procedural fidelity is defined as the extent to which the independent variable is implemented as prescribed. Research using computerized tasks has shown that fidelity errors involving consequences for behavior can hinder skill acquisition. However, studies examining the effects of these errors once skills have been mastered are lacking. Thus, this translational study investigated the effects of varying levels of fidelity following mastery of a computerized arbitrary matching-to-sample task. A group design (consisting of five groups) was used in which college students initially completed 250 trials during which no programmed errors (i.e., perfect fidelity) were arranged, followed by an additional 250 trials with consequences delivered across various levels of fidelity (i.e., 20, 40, 60, 80, and 100% of trials administered without errors). The results showed that participants assigned to higher fidelity conditions performed better (on average). These results extended the findings of previous studies by demonstrating how errors involving consequences affect behavior across various stages of learning.

Nominally acceptable integrity failures negatively affect interventions involving intermittent reinforcement

The finding that differential reinforcement of alternative behavior (DRA) is efficacious at 80% integrity when continuous reinforcement is programmed for alternative responding may have contributed to a perception that integrity at 80% or above is acceptable. However, research also suggests that other interventions (e.g., noncontingent reinforcement) may not remain effective at 80% integrity. The conditions under which 80% integrity is acceptable for common behavioral interventions remains unclear. Therefore, we conducted two human-operant studies to evaluate effects of 80% integrity for interventions with contingent or noncontingent intermittent reinforcement schedules. During Experiment 1, we compared noncontingent reinforcement (NCR) and DRA when implemented with 80% integrity. During Experiment 2, we compared 2 variations of DRA, which included either a ratio or interval schedule to reinforce alternative behavior. Results replicated previous research showing that DRA with a FR-1 schedule programmed for alternative responding resulted in consistent target response suppression, even when integrity was reduced to 80%. In contrast, neither NCR nor interval-based DRA were consistently effective when implemented at 80% integrity. These results demonstrate that 80% integrity is not a uniformly acceptable minimal level of integrity.

Strengthening the procedural fidelity research-to-practice loop in animal behavior

Procedural fidelity is the extent to which components of an intervention are implemented as designed. Procedural fidelity is measured as a dependent variable and manipulated as an independent variable. In research and practice, procedural-fidelity data should be collected, monitored, and reported. Procedural fidelity as an independent variable has been investigated in humans using parametric analyses, and the current article summarizes some of the research conducted on the effects of procedural-fidelity errors in behavior-reduction and skill-acquisition interventions. Connections were drawn to applied animal researchers and the work of animal behavior practitioners to produce implications for practice with human and animal clients and suggestions for future research. Further, there are multiple ways to measure procedural fidelity, and different conclusions can be drawn based on the measure and computation method. The current article describes procedural-fidelity measures that are most applicable to animal behavior researchers and professionals.

Effects of environmental distractions on teachers' procedural integrity with three function-based treatments

Past research has demonstrated the effectiveness of teacher-implemented, function-based treatments for problem behavior, but no studies have evaluated the impact of distractions on teachers' procedural integrity. In this proof-of-concept study, the experimenters employed a laboratory analog to examine the impact of distractions on levels of integrity when 5 teachers implemented 3 different treatments. Although integrity was similar across treatments when the setting was free of distractions, integrity declined for all teachers in the presence of student-driven distractions. In general, distractions had a greater impact on the integrity of differential negative reinforcement of alternative behavior (DNRA) compared to differential negative reinforcement of other behavior (DNRO) and noncontingent escape (NCE), particularly for the delivery of reinforcement. However, teachers tended to have lower levels of integrity when responding to problem behavior during DNRO. These findings support the potential viability of this approach for studying factors that impede procedural integrity in the classroom.

Comparing training methods to improve volunteer skills during therapeutic horseback riding: A randomized control trial

Although in-vivo behavioral skills training (BST) is often effective, it may be too resource-intensive for organizations that rely on volunteers. Alternatives to in-vivo BST include video models or interactive computer training (ICT), but the utility of these procedures for training volunteers remains largely unknown. We used a randomized control trial to teach multiple skills to new volunteers at a therapeutic riding center. A total of 60 volunteers were assigned to one of three groups using block randomization. Depending on group assignment, volunteers received instructions and modeling through in-vivo interactions, a video model, or ICT. All volunteers completed in-vivo role plays with feedback. Skills were measured by unblinded observers during role plays. There were no statistically significant differences in accuracy of role-play performance between volunteers in the in-vivo BST and ICT groups, but both outperformed the video-model group. The ICT and video model required statistically significantly less time from a live instructor than did in-vivo training. Thus, although in-vivo BST remains best practice, ICT may be a viable alternative when training resources are limited.

Interventionist Acquisition of Incidental Teaching Using Pyramidal Training via Telehealth

We investigated the effects of a telehealth pyramidal training package on participants' implementation of incidental teaching. A total of eight adults worked with eight children with autism. Coaches were first taught to implement incidental teaching and then taught subsequent interventionists. The training package consisted of an online module and delayed video-based feedback provided via videoconferencing. Following the telehealth training program, coaches and interventionists reached the preset performance criteria and implemented incidental teaching with high fidelity. All of the child participants increased mands above baseline levels. Results suggest that interventionists can be trained via telehealth in behavior analytic interventions.