Statistical inference for individual organism research: mixed blessing or curse?

Understanding Individual Subject Differences through Large Behavioral Datasets: Analytical and Statistical Considerations

A core feature of behavior analysis is the single-subject design, in which each subject serves as its own control. This approach is powerful for identifying manipulations that are causal to behavioral changes but often fails to account for individual differences, particularly when coupled with a small sample size. It is more common for other subfields of psychology to use larger-N approaches; however, these designs also often fail to account for the individual by focusing on aggregate-level data only. Moving forward, it is important to study individual differences to identify subgroups of the population that may respond differently to interventions and to improve the generalizability and reproducibility of behavioral science. We propose that large-N datasets should be used in behavior analysis to better understand individual subject variability. First, we describe how individual differences have been historically treated and then outline practical reasons to study individual subject variability. Then, we describe various methods for analyzing large-N datasets while accounting for the individual, including correlational analyses, machine learning, mixed-effects models, clustering, and simulation. We provide relevant examples of these techniques from published behavioral literature and from a publicly available dataset compiled from five different rat experiments, which illustrates both group-level effects and heterogeneity across individual subjects. We encourage other behavior analysts to make use of the substantial advancements in online data sharing to compile large-N datasets and use statistical approaches to explore individual differences.

Confluence of Science and History in the Experimental Analysis of Behavior Course

This is a review of content and method for incorporating the history of the experimental analysis of behavior (EAB) into the EAB course, although the material also could be adapted for any course related to the topics of learning and behavior change, or the history of psychology. Six elements associated with establishing a new discipline are considered as a framework for introducing the history of EAB: the intellectual leader/founding scientist(s), early proponents of the new area who advance and elaborate on the founder's ideas, the cultural context in which the discipline develops, a set of methods, a textbook, and means of communicating with other, similarly inclined scientists. The historical ebb and flow of research and some of the reasons for these shifts are discussed next, with examples of EAB research themes that have shifted over time. Illustrating the history of EAB with specific milestone experiments seems a useful way to both introduce substantive research and its history. To that end, milestone experiments in EAB are discussed. The review ends with considerations about locating historical material within the EAB course.

Cochran's Q Test of Stimulus Overselectivity within the Verbal Repertoire of Children with Autism

Stimulus overselectivity remains an ill-defined concept within behavior analysis, because it can be difficult to distinguish truly restrictive stimulus control from random variation. Quantitative models of bias are useful, though perhaps limited in application. Over the last 50 years, research on stimulus overselectivity has developed a pattern of assessment and intervention repeatedly marred by methodological flaws. Here we argue that a molecular view of overselectivity, under which restricted stimulus control has heretofore been examined, is fundamentally insufficient for analyzing this phenomenon. Instead, we propose the use of the term "overselectivity" to define temporally extended patterns of restrictive stimulus control that have resulted in disproportionate populations of responding that cannot be attributed to chance alone, and highlight examples of overselectivity within the verbal behavior of children with autism spectrum disorder. Viewed as such, stimulus overselectivity lends itself to direct observation and measurement through the statistical analysis of single-subject data. In particular, we demonstrate the use of the Cochran Q test as a means of precisely quantifying stimulus overselectivity. We provide a tutorial on calculation, a model for interpretation, and a discussion of the implications for the use of Cochran's Q by clinicians and researchers.

A systematic review of applied single-case research published between 2016 and 2018: Study designs, randomization, data aspects, and data analysis

Single-case experimental designs (SCEDs) have become a popular research methodology in educational science, psychology, and beyond. The growing popularity has been accompanied by the development of specific guidelines for the conduct and analysis of SCEDs. In this paper, we examine recent practices in the conduct and analysis of SCEDs by systematically reviewing applied SCEDs published over a period of three years (2016-2018). Specifically, we were interested in which designs are most frequently used and how common randomization in the study design is, which data aspects applied single-case researchers analyze, and which analytical methods are used. The systematic review of 423 studies suggests that the multiple baseline design continues to be the most widely used design and that the difference in central tendency level is by far most popular in SCED effect evaluation. Visual analysis paired with descriptive statistics is the most frequently used method of data analysis. However, inferential statistical methods and the inclusion of randomization in the study design are not uncommon. We discuss these results in light of the findings of earlier systematic reviews and suggest future directions for the development of SCED methodology.

Consistent Methodological Errors Observed in Single-Case Studies: Suggested Guidelines

This article reviews current and consistent methodological problems that occur in research articles using single-case experimental designs. Problems discussed include (1) the use of pre- and quasi-experimental single-case designs, (2) the presentation and analysis of data, (3) the lack or inadequate demonstration of experimental reliability, and (4) the improper or ineffective use of generalization. The authors suggest ways of solving these problems and provide guidelines for avoiding their recurrence.

The Effectiveness of Aerobic Exercise Instruction for Totally Blind Women

This article reports on a study of the effectiveness of a multifaceted method of teaching aerobic exercises to three congenitally blind women. The results indicate that visually impaired persons can benefit physically from aerobic exercises, despite their inability to monitor body movements visually. All three women demonstrated positive gains in their performance, physical fitness, and attitudes toward participating in future mainstream aerobic exercise classes.

Interpreting functional analysis outcomes using automated nonparametric statistical analysis

Current methods employed to interpret functional analysis data include visual analysis and post-hoc visual inspection (PHVI). However, these methods may be biased by dataset complexity, hand calculations, and rater experience. We examined whether an automated approach using nonparametric rank-based statistics could increase the accuracy and efficiency of functional analysis data interpretation. We applied Automated Nonparametric Statistical Analysis (ANSA) to a sample of 65 published functional analyses for which additional experimental evidence was available to verify behavior function. Results showed that exact behavior function agreement between ANSA and the publications authors was 83.1%, exact agreement between ANSA and PHVI was 75.4%, and exact agreement across all 3 methods was 64.6%. These preliminary findings suggest that ANSA has the potential to support the data interpretation process. A web application that incorporates the calculations and rules utilized by ANSA is accessible at https://ansa.shinyapps.io/ansa/.

An Overview of Scientific Reproducibility: Consideration of Relevant Issues for Behavior Science/Analysis

For over a decade, the failure to reproduce findings in several disciplines, including the biomedical, behavioral, and social sciences, have led some authors to claim that there is a so-called "replication (or reproducibility) crisis" in those disciplines. The current article examines: (a) various aspects of the reproducibility of scientific studies, including definitions of reproducibility; (b) published concerns about reproducibility in the scientific literature and public press; (c) variables involved in assessing the success of attempts to reproduce a study; (d) suggested factors responsible for reproducibility failures; (e) types of validity of experimental studies and threats to validity as they relate to reproducibility; and (f) evidence for threats to reproducibility in the behavior science/analysis literature. Suggestions for improving the reproducibility of studies in behavior science and analysis are described throughout.

Randomization tests as alternative analysis methods for behavior-analytic data

Randomization statistics offer alternatives to many of the statistical methods commonly used in behavior analysis and the psychological sciences, more generally. These methods are more flexible than conventional parametric and nonparametric statistical techniques in that they make no assumptions about the underlying distribution of outcome variables, are relatively robust when applied to small-n data sets, and are generally applicable to between-groups, within-subjects, mixed, and single-case research designs. In the present article, we first will provide a historical overview of randomization methods. Next, we will discuss the properties of randomization statistics that may make them particularly well suited for analysis of behavior-analytic data. We will introduce readers to the major assumptions that undergird randomization methods, as well as some practical and computational considerations for their application. Finally, we will demonstrate how randomization statistics may be calculated for mixed and single-case research designs. Throughout, we will direct readers toward resources that they may find useful in developing randomization tests for their own data.

Beyond inference by eye: Statistical and graphing practices in JEAB, 1992-2017

Debates about the utility of p values and correct ways to analyze data have inspired new guidelines on statistical inference by the American Psychological Association (APA) and changes in the way results are reported in other scientific journals, but their impact on the Journal of the Experimental Analysis of Behavior (JEAB) has not previously been evaluated. A content analysis of empirical articles published in JEAB between 1992 and 2017 investigated whether statistical and graphing practices changed during that time period. The likelihood that a JEAB article reported a null hypothesis significance test, included a confidence interval, or depicted at least one figure with error bars has increased over time. Features of graphs in JEAB, including the proportion depicting single-subject data, have not changed systematically during the same period. Statistics and graphing trends in JEAB largely paralleled those in mainstream psychology journals, but there was no evidence that changes to APA style had any direct impact on JEAB. In the future, the onus will continue to be on authors, reviewers and editors to ensure that statistical and graphing practices in JEAB continue to evolve without interfering with characteristics that set the journal apart from other scientific journals.

Applying the Taxonomy of Validity Threats from Mainstream Research Design to Single-Case Experiments in Applied Behavior Analysis

Mainstream research design in the social and behavioral sciences has often been conceptualized using a taxonomy of threats to experimental validity first articulated by Campbell and his colleagues (Campbell & Stanley, 1966; Cook & Campbell, 1979). The most recent update of this framework was published by Shadish, Cook, and Campbell (2002), in which the authors describe different types of validity and numerous threats to each primarily in terms of group-design experiments. In the present article, we apply Shadish et al.'s analysis of threats to internal, external, statistical conclusion, and construct validity to single-case experimental research as it is typically conducted in applied behavior analysis. In doing so, we hope to provide researchers and educators in the field with a translation of the validity-threats taxonomy into terms and considerations relevant to the design and interpretation of applied behavior-analytic research for the purposes of more careful research design and the ability to communicate our designs to individuals outside of behavior analysis, using their own vocabulary.

Results reporting in single case experiments and single case meta-analysis

Single Case Experimental Design is a discipline grounded in applied behavior analysis where the needs of individual clients and the application of scientific inquiry are fundamental tenets. These two principles remain tantamount in the conduct of research using this methodology and the expansion of the method into evidence-based practice determinations. Although recommendations for quality indicators are widespread, implementation is not. Concurrent to the rise of quality indicators is an increasing interest in analysis methodology. Visual analysis has a history of application and validity, newer forms of analysis less so. While some argue for concordance between the two, it may be the differences that are worth exploration in understanding characteristics of trend and variability in much of the published literature. Design choice and visual analysis decisions are rarely fully articulated. Statistical analyses are likewise inadequately justified or described. Recommendations for the explicit language of reporting as derived from prior meta-analysis and a current review of two leading journals provides a scaffold consistent with existing guidelines but additive in detail, exemplars, and justification. This is intended to improve reporting of results for individual studies and their potential use in future meta-analytic work.

Tutorial: Small-N Power Analysis

Power analysis is an overlooked and underreported aspect of study design. A priori power analysis involves estimating the sample size required for a study based on predetermined maximum tolerable Type I and II error rates and the minimum effect size that would be clinically, practically, or theoretically meaningful. Power is more often discussed within the context of large-N group designs, but power analyses can be used in small-N research and within-subjects designs to maximize the probative value of the research. In this tutorial, case studies illustrate how power analysis can be used by behavior analysts to compare two independent groups, behavior in baseline and intervention conditions, and response characteristics across multiple within-subject treatments. After reading this tutorial, the reader will be able to estimate just noticeable differences using means and standard deviations, convert them to standardized effect sizes, and use G*Power to determine the sample size needed to detect an effect with desired power.

The Language of Science

Science is what scientists do and, especially, what they say about what they do. Science is a way of talking about the world that enables the listener to behave more effectively in that world. Understanding science, then, is a matter of understanding the language of science. Scientific verbal practices are codified and recorded so that they can affect the behavior of all scientists, including those without access to the original controlling variables. What we know about the world is simply the way we have learned to talk about the world. We know best what is most useful about the world, in the sense that what we know enables us to behave effectively in the world. Scientists are unique in that they, more so than non-scientists, have the experience of behaving as effectively as possible-they can predict and control. This is what makes all the difference, in the sense that it makes science different from other ways of knowing about the world. Science is not simply one way of knowing about the world, it is arguably the most effective way of knowing about it. Scientific talk leads to effective action.

An Empirical Investigation of Interrater Agreement for Single-Subject Data Using Graphs with and without Trend Lines

The interrater agreement of visual judgments made from single-subject data was examined. Seventy-nine raters were given 21 single-subject graphs. Thirty-nine of the raters examined graphs containing single-subject data arrayed in the traditional format. The remaining 40 subjects reviewed AB graphs that were supplemented by a trend line. As measured by intraclass correlation coefficients, interrater agreement was higher for the trend line group than for the group relying only on visual analysis. There was a statistically significant correlation between the change in slope across the phases of the AB design and a score reflecting disagreement among raters in the visual analysis group. This relationship between change in slope and rater disagreement was not present in the trend line group. The results suggest that the low interrater agreement often associated with visual analysis of single-subject data may be improved by simple supplements to visually inspected charts.

Obstacles in Using Randomization Tests in Single-Case Experimentation

Edgington has just proposed the use of randomization tests for single-case experimental designs. The specific tests presented require rapidly alternating treatments over time for a given subject. Although randomization tests may provide viable statistical options, several features of single-case research ordinarily would preclude their use. Characteristics of the interventions and responses that are encompassed by the substantive areas of single-case research are likely to preclude shifting rapidly from one treatment to another on a randomized schedule. Also, the possibility of multiple-treatment interference may be especially acute with randomization tests. Finally, the use of randomization tests may compete with the logic of single-case experimental designs. The present paper discusses both applied and methodological restrictions on the use of randomization tests.

A Bibliographic Tribute to Jack Michael

"In the late 1950's, Jack Michael, a bright but irritating young psychology instructor, moved from the Universities of Kansas to Houston to Arizona State. Along the way he befriended two nontraditional students, protected them through their Ph.D. programs, and turned them loose on the world: Teodoro Ayllon…and Montrose Wolf…" (Risley, 2001, p. 267). So begins Risley's chapter on the origins of applied behavior analysis. For almost 50 years, Jack Michael provided a model for us to "talk like Skinner" and to analyze behavior as Skinner would. For this, he has been widely respected and revered. The purpose of this bibliography is to explain to new and familiar readers alike Jack's contributions to the field of behavior analysis in areas of his primary focus: (a) behavioral function taxonomy, (b) motivation, (c) reinforcement, (d) response topographies, (e) multiple control, (f) duplic and codic verbal behavior, and (g) teaching. Throughout, we weave his role in the field's history and his leadership in its expansion, as these have been additional areas of significant contributions. Above all, we wish to highlight Jack's work, in bibliographic and narrative form, in a way that expresses a heartfelt tribute on behalf of his students and others whom he influenced to learn about psychology as a natural science and to think and talk like Skinner.

An Analysis of Serial Dependency in Occupational Therapy Research

Single-subject and time-series designs have recently been advocated as a preferred method of examining clinical change in individual patients. Data from single-subject designs are frequently analyzed by means of graphic presentation and visual inspection. The presence of serial dependency or autocorrelation in data collected from a single individual can reduce the reliability and accuracy of visual inferences. Fifty-four data paths from single-subject research published in the occupational therapy literature were reviewed to determine the degree of serial dependency present in each data set. The results revealed that a large portion (41 %) of the data sets contained a significant degree of autocorrelation. The implications of a high degree of serial dependency in relation to data analysis and interpretation are discussed, and methods to reduce the effect of serial dependency are suggested.

Reinforcement Effects on Three Measures of Children's Interest in Math

The effects of contingent reward on children's interest in an academic activity (math) were investigated in a token economy analog. Three measures of interest were examined using an A-B—a design: (1) amount of the activity produced, (2) quality of the activity produced, and (3) time spent engaging in the activity. Reward was delivered contingent upon the first of these measures. Experimental subjects were exposed to baseline, reinforcement, baseline, and follow-up conditions. A control group received baseline procedures throughout. No evidence of substantial undermining of interest occurred on any measure, although two subjects displayed an immediate, transient decrease in postreward performance.

The Relationship Between Visual Analysis and Five Statistical Analyses in a Simple AB Single-Case Research Design

This study explored some practical issues for single-case researchers who rely on visual analysis of graphed data, but who also may consider supplemental use of promising statistical analysis techniques. The study sought to answer three major questions: (a) What is a typical range of effect sizes from these analytic techniques for data from “effective interventions”? (b) How closely do results from these same analytic techniques concur with visual-analysis-based judgments of effective interventions? and (c) What role does autocorrelation play in interpretation of these analytic results? To answer these questions, five analytic techniques were compared with the judgments of 45 doctoral students and faculty, who rated intervention effectiveness from visual analysis of 35 fabricated AB design graphs. Implications for researchers and practitioners using single-case designs are discussed.

Single Subject Designs

Issues surrounding the application of inferential statistical tests to behavior modification research was presented. Four major behavior modification journals were surveyed from their inception through 1974 for their use of inferential statistical tests in empirical research studies. Results indicated that Behaviour Research and Therapy and Behavior Therapy contained the highest percentage of articles using inferential statistical procedures. Over all journals, nearly all studies employing group designs used inferential statistical tests. Parametric analysis-of-variance and various nonparametric tests were generally the most commonly employed statistics within the journals sampled. Implications of the review suggest that the methodological base of behavior modification research is becoming more diverse as reflected in increased use of group designs where inferential tests are commonly employed. While all journals contained some N=1 experiments which relied on some form of inferential statistics, both valid and invalid applications are noted. Issues surrounding the use of statistics are documented. Implications for research and training are discussed.

Characteristics Influencing the Visual Analysis of Single-Subject Data: An Empirical Analysis

A study explored various graphic characteristics associated with the visual analysis of single-subject data. A sample of 20 rehabilitation therapists used visual analysis to rate 24 graphs of data sets using a traditional baseline/treatment (AB) format. Using the intraclass correlation approach, the authors assessed the interrater reliability and found that it ranged from .52 to .66 for each graph. Data analysis revealed that the graphic characteristics of level and mean shift were associated with consistent judgments across the raters, and that changes in slope across the two phases of a graph were associated with substantial rater disagreement. The authors discuss the implications of using visual and statistical procedures to analyze single-subject data, and argue that quantitative adjuncts to visual analysis may facilitate the reliable interpretation of graphed data for a single subject.

The Effects of Celeration Lines on Visual Data Analysis

Previous visual analysis research reported that the overall agreement between visual analysis and statistical analysis was poor. In response, some researchers suggested the use of celeration lines to improve the accuracy and reliability of visual analysis. However, subsequent research reported little or no improvement in accuracy with such lines. The present study presented 5 board-certified behavior analysts with a series of behavioral graphs. The participants were asked to answer questions similar to those posed in previous studies but were also asked to talk aloud as they viewed each graph. Results indicate that the participants made accurate decisions for only 72% of the graphs and that celeration lines did not improve overall accuracy. The verbal protocol analysis suggests that participants were as likely to attend to trend when celeration lines were absent as they were when they were present, with the most differences attributable to varying participant competencies and not graph (i.e., celeration line) characteristics.

Analyzing Two-Phase Single-Case Data with Non-overlap and Mean Difference Indices: Illustration, Software Tools, and Alternatives

Two-phase single-case designs, including baseline evaluation followed by an intervention, represent the most clinically straightforward option for combining professional practice and research. However, unless they are part of a multiple-baseline schedule, such designs do not allow demonstrating a causal relation between the intervention and the behavior. Although the statistical options reviewed here cannot help overcoming this methodological limitation, we aim to make practitioners and applied researchers aware of the available appropriate options for extracting maximum information from the data. In the current paper, we suggest that the evaluation of behavioral change should include visual and quantitative analyses, complementing the substantive criteria regarding the practical importance of the behavioral change. Specifically, we emphasize the need to use structured criteria for visual analysis, such as the ones summarized in the What Works Clearinghouse Standards, especially if such criteria are complemented by visual aids, as illustrated here. For quantitative analysis, we focus on the non-overlap of all pairs and the slope and level change procedure, as they offer straightforward information and have shown reasonable performance. An illustration is provided of the use of these three pieces of information: visual, quantitative, and substantive. To make the use of visual and quantitative analysis feasible, open source software is referred to and demonstrated. In order to provide practitioners and applied researchers with a more complete guide, several analytical alternatives are commented on pointing out the situations (aims, data patterns) for which these are potentially useful.

JEAB Research Over Time: Species Used, Experimental Designs, Statistical Analyses, and Sex of Subjects

We examined the species used as subjects in every article published in the Journal of the Experimental Analysis of Behavior (JEAB) from 1958 through 2013. We also determined the sex of subjects in every article with human subjects (N = 524) and in an equal number of randomly selected articles with nonhuman subjects, as well as the general type of experimental designs used. Finally, the percentage of articles reporting an inferential statistic was determined at 5-year intervals. In all, 35,317 subjects were studied in 3,084 articles; pigeons ranked first and humans second in number used. Within-subject experimental designs were more popular than between-subjects designs regardless of whether human or nonhuman subjects were studied but were used in a higher percentage of articles with nonhumans (75.4 %) than in articles with humans (68.2 %). The percentage of articles reporting an inferential statistic has increased over time, and more than half of the articles published in 2005 and 2010 reported one. Researchers who publish in JEAB frequently depart from Skinner's preferred research strategy, but it is not clear whether such departures are harmful. Finally, the sex of subjects was not reported in a sizable percentage of articles with both human and nonhuman subjects. This is an unfortunate oversight.

Assessing Functional Relations in Single-Case Designs

In the context of the evidence-based practices movement, the emphasis on computing effect sizes and combining them via meta-analysis does not preclude the demonstration of functional relations. For the latter aim, we propose to augment the visual analysis to add consistency to the decisions made on the existence of a functional relation without losing sight of the need for a methodological evaluation of what stimuli and reinforcement or punishment are used to control the behavior. Four options for quantification are reviewed, illustrated, and tested with simulated data. These quantifications include comparing the projected baseline with the actual treatment measurements, on the basis of either parametric or nonparametric statistics. The simulated data used to test the quantifications include nine data patterns in terms of the presence and type of effect and comprise ABAB and multiple-baseline designs. Although none of the techniques is completely flawless in terms of detecting a functional relation only when it is present but not when it is absent, an option based on projecting split-middle trend and considering data variability as in exploratory data analysis proves to be the best performer for most data patterns. We suggest that the information on whether a functional relation has been demonstrated should be included in meta-analyses. It is also possible to use as a weight the inverse of the data variability measure used in the quantification for assessing the functional relation. We offer an easy to use code for open-source software for implementing some of the quantifications.

The influence of the design matrix on treatment effect estimates in the quantitative analyses of single-subject experimental design research

The quantitative methods for analyzing single-subject experimental data have expanded during the last decade, including the use of regression models to statistically analyze the data, but still a lot of questions remain. One question is how to specify predictors in a regression model to account for the specifics of the design and estimate the effect size of interest. These quantitative effect sizes are used in retrospective analyses and allow synthesis of single-subject experimental study results which is informative for evidence-based decision making, research and theory building, and policy discussions. We discuss different design matrices that can be used for the most common single-subject experimental designs (SSEDs), namely, the multiple-baseline designs, reversal designs, and alternating treatment designs, and provide empirical illustrations. The purpose of this article is to guide single-subject experimental data analysts interested in analyzing and meta-analyzing SSED data.

Single-case experimental designs: reflections on conduct and analysis

In this editorial discussion we reflect on the issues addressed by, and arising from, the papers in this special issue on Single-Case Experimental Design (SCED) study methodology. We identify areas of consensus and disagreement regarding the conduct and analysis of SCED studies. Despite the long history of application of SCEDs in studies of interventions in clinical and educational settings, the field is still developing. There is an emerging consensus on methodological quality criteria for many aspects of SCEDs, but disagreement on what are the most appropriate methods of SCED data analysis. Our aim is to stimulate this ongoing debate and highlight issues requiring further attention from applied researchers and methodologists. In addition we offer tentative criteria to support decision-making in relation to the selection of analytical techniques in SCED studies. Finally, we stress that large-scale interdisciplinary collaborations, such as the current Special Issue, are necessary if SCEDs are going to play a significant role in the development of the evidence base for clinical practice.

Factors Affecting Visual Inference in Single-Case Designs

Visual inspection remains the most frequently applied method for detecting treatment effects in single-case designs. The advantages and limitations of visual inference are here discussed in relation to other procedures for assessing intervention effectiveness. The first part of the paper reviews previous research on visual analysis, paying special attention to the validation of visual analysts' decisions, inter-judge agreement, and false alarm and omission rates. The most relevant factors affecting visual inspection (i.e., effect size, autocorrelation, data variability, and analysts' expertise) are highlighted and incorporated into an empirical simulation study with the aim of providing further evidence about the reliability of visual analysis. Our results concur with previous studies that have reported the relationship between serial dependence and increased Type I rates. Participants with greater experience appeared to be more conservative and used more consistent criteria when assessing graphed data. Nonetheless, the decisions made by both professionals and students did not match sufficiently the simulated data features, and we also found low intra-judge agreement, thus suggesting that visual inspection should be complemented by other methods when assessing treatment effectiveness.

The good, the bad, and the aggregate

To evaluate progress and focus goals, scientific disciplines need to identify relations that are robust across many situations. One approach is the literature review, which characterizes generality across studies. Some writers (e.g., Baron & Derenne, 2000) claim that quantitative literature reviews, but not narrative reviews, violate the methodological precepts of behavior analysis by pooling data from nonidentical studies. We argue that it is impossible to assess generality without varying the context in which relationships are studied. Properly chosen data-aggregation strategies can reveal which behavior-environment relations are general and which are procedure dependent. Within behavior analysis, reluctance to conduct quantitative reviews may reflect unsupported assumptions about the consequences of aggregating data across studies. Whether specific data-aggregation techniques help or harm a research program is an empirical issue that cannot be resolved by unstructured discussion. Some examples of how aggregation has been used in identifying behavior-environment relations are examined.

Visual inspection of data revisited: Do the eyes still have it?

In behavior analysis, visual inspection of graphic information is the standard by which data are evaluated. Efforts to supplement visual inspection using inferential statistical procedures to assess intervention effects (e.g., analysis of variance or time-series analysis) have met with opposition. However, when serial dependence is present in the data, the use of visual inspection by itself may prove to be problematic. Previously published reports demonstrate that autocorrelated data influence trained observers' ability to identify level treatment effects and trends that occur in the intervention phase of experiments. In this report, four recent studies are presented in which autoregressive equations were used to produce point-to-point functions to simulate experimental data. In each study, various parameters were manipulated to assess trained observers' responses to changes in point-to-point functions from the baseline condition to intervention. Level shifts over baseline behavior (treatment effect), as well as no change from baseline (no treatment effect or trend), were most readily identified by observers, but trends were rarely recognized. Furthermore, other factors previously thought to augment and improve observers' responses had no impact. Results are discussed in terms of the use of visual inspection and the training of behavior analysts.

A critique of the usefulness of inferential statistics in applied behavior analysis

Researchers continue to recommend that applied behavior analysts use inferential statistics in making decisions about effects of independent variables on dependent variables. In many other approaches to behavioral science, inferential statistics are the primary means for deciding the importance of effects. Several possible uses of inferential statistics are considered. Rather than being an objective means for making decisions about effects, as is often claimed, inferential statistics are shown to be subjective. It is argued that the use of inferential statistics adds nothing to the complex and admittedly subjective nonstatistical methods that are often employed in applied behavior analysis. Attacks on inferential statistics that are being made, perhaps with increasing frequency, by those who are not behavior analysts, are discussed. These attackers are calling for banning the use of inferential statistics in research publications and commonly recommend that behavioral scientists should switch to using statistics aimed at interval estimation or the method of confidence intervals. Interval estimation is shown to be contrary to the fundamental assumption of behavior analysis that only individuals behave. It is recommended that authors who wish to publish the results of inferential statistics be asked to justify them as a means for helping us to identify any ways in which they may be useful.

Reflections on the glass ceiling: women in the experimental analysis of behavior

McSweeney and Swindell (1998) sought to determine whether men and women are treated equitably in the experimental analysis of behavior. They purported to show that women participate less in the Journal of the Experimental Analysis of Behavior than in similar journals and that the participation of women decreases with increases in selectivity. Their data were difficult to interpret, however, because they did not present the variability in the mean data drawn from different individuals over time. My analyses were not in accord with their conclusions. When the percentage of associate editors who are women was considered along with the mean percentages McSweeney and Swindell reported for other measures, participation did not systematically decrease with increases in selectivity in recent years. As quantified in terms of their number of publications in the Journal of the Experimental Analysis of Behavior, women who were editorial board members and associate editors were not more highly selected than their male counterparts. Finally, in the recent period from 1996 to 1998, although women submitted fewer manuscripts to the journal, rejection ratios did not differ for men and women. Efforts to increase the participation of women in the experimental analysis of behavior may best be directed toward recruitment and retention rather than some of the suggestions proposed by McSweeney and Swindell (1998), which could inadvertently create different standards for women's work.

Consistent visual analyses of intrasubject data

Visual inspection of single-case data is the primary method of interpretation of the effects of an independent variable on a dependent variable in applied behavior analysis. The purpose of the current study was to replicate and extend the results of DeProspero and Cohen (1979) by reexamining the consistency of visual analysis across raters. We recruited members of the board of editors and associate editors for the Journal of Applied Behavior Analysis to judge graphs on a 100-point scale of experimental control and by providing a dichotomous response (i.e., "yes" or "no" for experimental control). Results showed high interrater agreement across the three types of graphs, suggesting that visual inspection can lead to consistent interpretation of single-case data among well-trained raters.