Generalizability and Dependability of Direct Behavior Ratings to Assess Social Behavior of Preschoolers

Formative behavioral assessment across eight constructs: Dependability of direct behavior ratings and formative behavior rating measures

Due to their promise as a feasible tool for evaluating the effects of school-based interventions, Direct Behavior Ratings (DBR) have received much research attention over the past 2 decades. Although DBR methodology has demonstrated much promise, favorable psychometric characteristics only have been demonstrated for tools measuring a small number of constructs. Likewise, although a variety of methods of DBR have been proposed, most extant studies have focused on the use of single-item methods. The present study examined the dependability of four methods of formative behavioral assessment (i.e., single-item and multi-item ratings administered either daily [DBR] or weekly [formative behavior rating measures or FBRM]) across eight psychological constructs (i.e., interpersonal skills, academic engagement, organizational skills, disruptive behavior, oppositional behavior, interpersonal conflict, anxious depressed, and social withdrawal). School-based professionals (N = 91; i.e., teachers, paraprofessionals, and intervention specialists) each rated one student across all eight constructs after being assigned to one of the four assessment conditions. Dependability estimates varied substantially across methods and constructs (range = 0.75-0.96), although findings of the present study support the use of the broad set of formative assessment tools evaluated.

A Finite Mixture Item Response Theory Model for Continuous Measurement Outcomes

A mixture extension of Samejima's continuous response model for continuous measurement outcomes and its estimation through a heuristic approach based on limited-information factor analysis is introduced. Using an empirical data set, it is shown that two groups of respondents that differ both qualitatively and quantitatively in their response behavior can be revealed. In addition to the real data application, the effectiveness of the heuristic estimation approach under real data analytic conditions was examined through a Monte Carlo simulation study. The results showed that the heuristic estimation approach provided reliable parameter estimates and the model successfully converged above 80% when the sample size was 250 and above 90% when the sample size was 500 or 1,000 for most conditions.

Examining the Long-Term Stability of a Strengths-Based Screener Over 2 Years

The success of universal screening for effective school mental health programs is dependent on the availability of usable measures as well as empirically based recommendations for use. The current study examined the long-term stability of a strengths-based social-emotional screening tool, the Devereux Student Strengths Assessment-Mini (DESSA-Mini). Elementary teachers rated students ( N = 273; kindergarten and first grade at Time 1) 3 times per year over 2 years to identify students for early intervention. Stability coefficients were moderate to large for continuous and categorical data but lower between years, and a transition matrix demonstrated greater movement across categories compared with prior research. A latent profile analysis with all six time-point T-Scores indicated four stability profiles. Three patterns were stable across all times while one profile improved over time. Profile results were compared with covariates of free and reduced-price lunch, special education, and intervention status as well as outcomes of reading achievement and behavior referrals. Practice implications and areas for future research are discussed.

Generalizability theory: a practical guide to study design, implementation, and interpretation

Generalizability Theory (GT) offers increased utility for assessment research given the ability to concurrently examine multiple sources of variance, inform both relative and absolute decision making, and determine both the consistency and generalizability of results. Despite these strengths, assessment researchers within the fields of education and psychology have been slow to adopt and utilize a GT approach. This underutilization may be due to an incomplete understanding of the conceptual underpinnings of GT, the actual steps involved in designing and implementing generalizability studies, or some combination of both issues. The goal of the current article is therefore two-fold: (a) to provide readers with the conceptual background and terminology related to the use of GT and (b) to facilitate understanding of the range of issues that need to be considered in the design, implementation, and interpretation of generalizability and dependability studies. Given the relevance of this analytic approach to applied assessment contexts, there exists a need to ensure that GT is both accessible to, and understood by, researchers in education and psychology. Important methodological and analytical considerations are presented and implications for applied use are described.

Observations of families in structured interactions: parenting therapists provide reliable ratings of mothers' parenting

The reliability of observations of parenting by parenting therapists was assessed. An important predictor of externalizing behavior in children is quality of parenting. Data were videotapes of structured interactions in families with a child age 8–12 years referred to the evidence based Parent Management Training Oregon (PMTO) treatment program for child behavior problems. The therapists had clinical PMTO training but no training in systematic observation. PMTO observational coders with specific coder training were included as a reference for the therapists. Five therapists and two coders observed videotapes of 10 families and performed global evaluations of mothers’ parenting skills. They used the coder’s impression measure used in PMTO research. Scores were analyzed in a generalizability theory framework for the two groups of observers separately. Both observer types reliably rank-ordered the mothers and assessed the level of parenting skills. PMTO therapists without coder training provided reliable ratings of parenting constructs relevant to the clinical PMTO program in a manner comparable to that of the trained reference coders.

Complacency With Access and the Aggregate? Affirming an Individual Determination of Educational Benefit Under the Individuals With Disabilities Education Act

Although some legal scholars argue that the progression of Congressional intent in laws for children with disabilities reflects a mission to raise the educational standards and their achievement levels, other insist that many students are not realizing equal educational opportunity and not receiving educational benefit. This article proposes that the alignment of the Individuals with disabilities Education Act (IDEA) with the No Child Left Behind (NCLB) Act may implicate a student’s right to a free, appropriate public education (FAPE) and cautions against complacency with mere access to the general curriculum and a satisfaction with aggregate accountability of outcome. An analysis of litigation from circuit courts is presented to confirm individual benefit as the legal standard of appropriateness, and the duties of IEP teams to monitoring individual progress on IEP goals are defined. Two recommendations for the reauthorization of both the NCLB Act and the IDEA are proposed.

The efficiency of behavior rating scales to assess inattentive-overactive and oppositional-defiant behaviors: applying generalizability theory to streamline assessment

Although the efficiency with which a wide range of behavioral data can be obtained makes behavior rating scales particularly attractive tools for the purposes of screening and evaluation, feasibility concerns arise in the context of formative assessment. Specifically, informant load, or the amount of time informants are asked to contribute to the assessment process, likely has a negative impact on the quality of data over time and the informant's willingness to participate. Two important determinants of informant load in progress monitoring are the length of the rating scale (i.e., the number of items) and how frequently informants are asked to provide ratings (i.e., the number of occasions). The purpose of the current study was to investigate the dependability of the IOWA Conners Teacher Rating Scale (Loney & Milich, 1982), which is used to differentiate inattentive-overactive from oppositional-defiant behaviors. Specifically, the facets of items and occasions were examined to identify combinations of these sources of error necessary to reach an acceptable level of dependability for both absolute and relative decisions. Results from D studies elucidated a variety of possible item-occasion combinations reaching the criteria for adequate dependability. Recommendations for research and practice are discussed.

Predictors of Social Skills for Preschool Children at Risk for ADHD: The Relationship Between Direct and Indirect Measurements

The relationship between direct and indirect measurements of social skills and social problem behaviors for preschool children at risk for attention deficit hyperactivity disorder (ADHD) was examined. Participants included 137 preschool children, aged 3 to 5 years, at risk for ADHD, who were participating in a larger study examining the effects of early intervention for young children. Teachers rated the social skills and social problems of the participants. Direct observation data of participants were also collected at preschool during free play. Results support previous research on social skills assessment and suggest that indirect and direct measures may not be measuring the same aspect of social skills. Thus, a variety of evaluation tools are necessary to comprehensively assess the social skills of preschool children with social challenges.