Evaluating the Usability of the Wellness School Assessment Tool Whole School, Whole Community, Whole Child (WellSAT WSCC): A School Wellness Policy Assessment Tool

BACKGROUND

Adoption of the Whole School, Whole Community, Whole Child (WSCC) model has been slowed by a lack of available tools to support implementation. The Wellness School Assessment Tool (WellSAT) WSCC is an online assessment tool that allows schools to evaluate the alignment of their policies with the WSCC model. This study assesses the usability of the WellSAT WSCC.

METHODS

Using a convergent mixed methods design, we collected qualitative and quantitative data from 5 school-based participants with roles in development and evaluation of policy. Participants explored the platform while engaging in a think-aloud procedure and scored a sample policy using the platform. They also completed the System Usability Scale and responded to open-ended questions about the usability of the platform.

RESULTS

Participants rated the WellSAT WSCC as an above-average user experience, but data suggested several areas for improvement, including improved instructions, enhanced visual design of the platform, and guidance for subsequent policy changes.

CONCLUSION

The WellSAT WSCC provides an above-average user experience but can be improved to increase user experience. These improvements increase the potential for greater use to facilitate integration of the WSCC model into school policy.

Formative behavioral assessment across eight constructs: Dependability of direct behavior ratings and formative behavior rating measures

Due to their promise as a feasible tool for evaluating the effects of school-based interventions, Direct Behavior Ratings (DBR) have received much research attention over the past 2 decades. Although DBR methodology has demonstrated much promise, favorable psychometric characteristics only have been demonstrated for tools measuring a small number of constructs. Likewise, although a variety of methods of DBR have been proposed, most extant studies have focused on the use of single-item methods. The present study examined the dependability of four methods of formative behavioral assessment (i.e., single-item and multi-item ratings administered either daily [DBR] or weekly [formative behavior rating measures or FBRM]) across eight psychological constructs (i.e., interpersonal skills, academic engagement, organizational skills, disruptive behavior, oppositional behavior, interpersonal conflict, anxious depressed, and social withdrawal). School-based professionals (N = 91; i.e., teachers, paraprofessionals, and intervention specialists) each rated one student across all eight constructs after being assigned to one of the four assessment conditions. Dependability estimates varied substantially across methods and constructs (range = 0.75-0.96), although findings of the present study support the use of the broad set of formative assessment tools evaluated.

State Variation Regarding Other Health Impairment Eligibility Criteria for Attention Deficit Hyperactivity Disorder

The special education eligibility category that has come to be most commonly associated with Attention-Deficit Hyperactivity Disorder (ADHD) in recent years is Other Health Impairment (OHI). However, the eligibility criteria for the OHI disability category have been criticized for being especially vague, given that the disability category incorporates a wide range of health impairments without providing any additional specificity. Because states have the latitude to utilize more specific eligibility criteria than what is provided at the federal level, the purpose of the current study was to review state-level special education eligibility criteria for OHI, with particular interest in identifying the degree to which eligibility guidance exists specific to students with ADHD and the extent to which this guidance varies across states. Results suggested that wide state variation exists regarding eligibility guidance, with 22% of states utilizing the federal definition and only 14% of states providing elaboration regarding all three components of the federal definition. Whereas it was most common for states to provide additional guidance surrounding what is needed to establish that a student has a health impairment, less than half of states provided specific guidance surrounding the other two components of the federal definition. Implications for policy and practice are discussed.

Exploring Views and Professional Learning Needs of Comprehensive, Integrated, Three-Tiered (Ci3T) Leadership Teams Related to Universal Behavior Screening Implementation

Research conducted to date has highlighted barriers to initial adoption of universal behavior screening in schools. However, little is known regarding the experiences of those implementing these procedures and there have been no studies conducted examining the experiences of educators in different stages of implementing various tiered systems of supports. Universal screening is foundational to a successful Comprehensive, Integrated Three-Tiered (Ci3T) model of prevention-an integrated tiered system addressing academics, behavior, and social and emotional well-being. Therefore, the perspectives of Ci3T Leadership Team members at different stages of Ci3T implementation were solicited through an online survey that sought to understand (1) current school-based screening practices and (2) individual beliefs regarding those practices. A total of 165 Ci3T Leadership Team members representing five school districts from three geographic regions across the United States, all of whom were participating in an Institute of Education Sciences Network grant examining integrated tiered systems, reported the screening procedures were generally well-understood and feasible to implement. At the same time, results highlighted continuing professional learning may be beneficial in the areas of: (1) integrating multiple sources of data (e.g., screening data with other data collected as regular school practices) and (2) using those multiple data sources to determine next steps for intervention. We discuss educational implications, limitations, and directions for future inquiry.

Five direct behavior rating multi-item scales: Sensitivity to the effects of classroom interventions

Direct Behavior Rating (DBR) is a tool designed for the assessment of behavioral changes over time. Unlike methods for summative evaluations, the development of progress monitoring tools requires evaluation of sensitivity to change. The present study aimed to evaluate this psychometric feature of five newly developed DBR Multi-Item Scales (DBR-MIS). Teachers identified students with behaviors interfering with their learning or the learning of others and implemented a Daily Report Card (DRC) intervention in the classroom settings for two months. The analyses were performed on 31 AB single case studies. Change metrics were calculated at an individual level by using Tau-U_{A vs. B + trend B} and Hedges' g and at a scale-level by using Mixed Effect Meta-Analysis, Hierarchical Linear Models (HLMs), and Between-Case Standardized Mean Difference (BC-SMD). HLMs were estimated considering both fixed and random effects of intervention and linear trend within the intervention phase. The results supported sensitivity to change for three DBR-MIS (i.e., Academic Engagement, Organizational Skills, and Disruptive Behavior), and the relative magnitudes were consistent across the metrics. Sensitivity to change of DBR-MIS Interpersonal Skills received moderate support. Conversely, empirical evidence was not provided for sensitivity to change of DBR-MIS Oppositional Behavior. Particular emphasis was placed on the intervention trend in that responses to behavioral interventions might occur gradually or require consistency over time in order to be observed by raters. Implications for the use of the new DBR-MIS in the context of progress monitoring of social-emotional behaviors are discussed.

Exploring the Alignment of Behavior Screening Policies and Practices in US Public School Districts

BACKGROUND

Although recent studies provide information regarding state-level policies and district-level practices regarding social, emotional, and behavioral screening, the degree to which these policies influence screening practices is unknown. As such, the purpose of this exploratory study was to compare state- and district-level policies and reported practices around school-based social, emotional, and behavioral screening.

METHODS

We obtained data for the present study from three sources: (1) a recent systematic review of state department of education websites; (2) a national survey of 1330 US school districts; and (3) a Web search and review of policy manuals published by the 1330 school districts. Comparative analyses were used to identify similarities and differences across state and district policies and practices.

RESULTS

Of the 1330 districts searched, 911 had policy manuals available for review; 87 of these policy manuals, which represented 10 states, met inclusion criteria, and thus, were included in analyses. Discrepancies were found across state and district policies and across state social, emotional, and behavioral screening guidance and district practices, but consistencies did exist across district policies within the same state.

CONCLUSION

District-level guidance around social, emotional, and behavioral screening appears to be limited. Our findings suggest a disconnect between state- and district-level social, emotional, and behavioral screening guidance and district reported practices, which signifies the need to identify the main influences on district- and school-level screening practices.

Konstruktvalidität eines universellen Screenings zur unterrichtsnahen und ökonomischen Diagnostik herausfordernden Verhaltens von Schüler_innen – eine Multitrait-Multimethod-Analyse

Zusammenfassung. Die vorliegende Untersuchung überprüft die Konstruktvalidität einer 16 Items umfassenden Kurzversion der Integrated Teacher Report Form (ITRF), einem universellen und unterrichtsrelevanten Verhaltensscreening zur Diagnostik des externalisierenden Verhaltens von Schüler_innen im Klassenraum. 107 Lehrkräfte bearbeiteten für insgesamt 1048 Schülerinnen und Schüler der ersten bis sechsten Klasse die ITRF sowie zusätzlich jeweils eines von drei im deutschsprachigen Raum etablierten Beurteilungsverfahren. Die Analyse der konvergenten und diskriminanten Validität erfolgt anhand einer Multitrait-Multimethod (MT-MM) Korrelationsmatrix sowie einem strukturprüfenden Correlated Trait-Correlated Method minus 1 [CT-C(M-1)] Modell zur separaten Analyse des Einflusses der Konstrukte (lernbezogene / aufmerksame Verhaltensprobleme, oppositionelle / störende Verhaltensprobleme) und der Methoden (ITRF, zusätzliches Beurteilungsverfahren) auf die erzielten Werte der Beurteilungen. Die Ergebnisse zeigen, dass sich die Stärke der theoretisch postulierten Korrelationen mit den empirischen Daten erwartungskonform abbilden lassen, was auf konvergente und diskriminante Validität hinweist. Die Varianz der ITRF-Werte lässt sich zu einem größeren Anteil durch das zu messende Konstrukt als durch methodenspezifische Einflüsse erklären. Somit liefern unsere Befunde Evidenz für eine angemessene Konstruktvalidität des Verfahrens, weshalb sich die Kurzversion der ITRF für den praktischen Schuleinsatz eignet.

Factorial invariance of the Usage Rating Profile for Supporting Students' Behavioral Needs (URP-NEEDS)

[Correction Notice: An Erratum for this article was reported online in School Psychology on Dec 30 2019 (see record 2019-80953-001). In the fourth paragraph of the "Understanding the Factors That Influence Usage" section and in the "Usage Rating Profile for Supporting Students' Behavioral Needs (URP-NEEDS)" section, the URP-NEEDS was incorrectly reported to have 23 items. This measure consists of 24 items. This item was also missing in the Appendix under the "Understanding" factor: "School personnel understand how goals for social, emotional, and behavioral screening fit with a system of student supports." All versions of this article have been corrected.] Previous research has suggested that multiple factors beyond acceptability alone (e.g., feasibility, external supports) may interact to determine whether consumers will use an intervention or assessment in practice. The Usage Rating Profile for Supporting Students' Behavioral Needs (URP-NEEDS) was developed in order to provide a simultaneous assessment of those factors influencing use of a particular approach to identifying and supporting the social, emotional, and behavioral needs of students. As the measure was intended for use with a range of school-based stakeholders, a first necessary step involved establishing the measurement invariance of the instrument. Participants in the current study included 1,112 district administrators, 431 building administrators, and 1,355 teachers who were asked to identify the approach used within their school district to identify and support the social, emotional, and behavioral needs of students, and then to complete the URP-NEEDS in reference to this identified approach. Results supported the measurement invariance of the URP-NEEDS across stakeholder groups. In addition, measurement invariance was found across self-identified approaches to social, emotional, and behavioral risk identification within the district administrator and teacher groups. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

Appraisal of comparative single-case experimental designs for instructional interventions with non-reversible target behaviors: Introducing the CSCEDARS ("Cedars")

Evidence-based practice as a process requires the appraisal of research as a critical step. In the field of developmental disabilities, single-case experimental designs (SCEDs) figure prominently as a means for evaluating the effectiveness of non-reversible instructional interventions. Comparative SCEDs contrast two or more instructional interventions to document their relative effectiveness and efficiency. As such, these designs have great potential to inform evidence-based decision-making. To harness this potential, however, interventionists and authors of systematic reviews need tools to appraise the evidence generated by these designs. Our literature review revealed that existing tools do not adequately address the specific methodological considerations of comparative SCEDs that aim to compare instructional interventions of non-reversible target behaviors. The purpose of this paper is to introduce the Comparative Single-Case Experimental Design Rating System (CSCEDARS, "cedars") as a tool for appraising the internal validity of comparative SCEDs of two or more non-reversible instructional interventions. Pertinent literature will be reviewed to establish the need for this tool and to underpin the rationales for individual rating items. Initial reliability information will be provided as well. Finally, directions for instrument validation will be proposed.

Measurement Invariance of a Universal Behavioral Screener Across Samples From the USA and Germany

Abstract. The current study examines the item and scalar equivalence of an abbreviated school-based universal screener that was cross-culturally translated and adapted from English into German. The instrument was designed to assess student behavior problems that impact classroom learning. Participants were 1,346 K-6 grade students from the US (n = 390, Mage = 9.23, 38.5% female) and Germany (n = 956, Mage = 8.04, 40.1% female). Measurement invariance was tested by multigroup confirmatory factor analysis (CFA) across students from the US and Germany. Results support full scalar invariance between students from the US and Germany (df = 266, χ2 = 790.141, Δχ2 = 6.9, p < .001, CFI = 0.976, ΔCFI = 0.000, RMSEA = 0.052, ΔRMSEA = −0.003) indicating that the factor structure, the factor loadings, and the item thresholds are comparable across samples. This finding implies that a full cross-cultural comparison including latent factor means and structural coefficients between the US and the German version of the abbreviated screener is possible. Therefore, the tool can be used in German schools as well as for cross-cultural research purposes between the US and Germany.

A universal behavioral screener linked to personalized classroom interventions: Psychometric characteristics in a large sample of German schoolchildren

The current study represents the first psychometric evaluation of an American English into German translation of a school-based universal screening measure designed to assess academic and disruptive behavior problems. This initial study examines the factor structure and diagnostic accuracy of the newly translated measure in a large sample of 1009 German schoolchildren attending grades 1-6 in Western Germany. Confirmatory factor analysis supported a two-factor model for both male- and female- students. Configural invariance was supported between male- and female-samples. However scalar invariance was not supported, with higher thresholds for ratings of female students. Results of receiver operating characteristic (ROC) curve analyses were indicative of good to excellent diagnostic accuracy with areas under the curve ranging from 0.89 to 0.93. Optimal cut-off scores were 10, 5, and 13 for the Academic Productivity/Disorganization, Oppositional/Disruptive, and the Total Problems Composite scores of the Integrated System Teacher Rating Form respectively. This initial study of the newly translated measure supports further investigations into its utility for universal screening in German speaking schools.

Classification accuracy and acceptability of the Integrated Screening and Intervention System Teacher Rating Form

This study examines the classification accuracy and teacher acceptability of a problem-focused screener for academic and disruptive behavior problems, which is directly linked to evidence-based intervention. Participants included 39 classroom teachers from 2 public school districts in the Northeastern United States. Teacher ratings were obtained for 390 students in Grades K-6. Data from the screening instrument demonstrate favorable classification accuracy, and teacher ratings of feasibility and acceptability support the use of the measure for universal screening in elementary school settings. Results indicate the novel measure should facilitate classroom intervention for problem behaviors by identifying at-risk students and informing targets for daily behavior report card interventions. (PsycINFO Database Record

Multi-item direct behavior ratings: Dependability of two levels of assessment specificity

Direct Behavior Rating-Multi-Item Scales (DBR-MIS) have been developed as formative measures of behavioral assessment for use in school-based problem-solving models. Initial research has examined the dependability of composite scores generated by summing all items comprising the scales. However, it has been argued that DBR-MIS may offer assessment of 2 levels of behavioral specificity (i.e., item-level, global composite-level). Further, it has been argued that scales can be individualized for each student to improve efficiency without sacrificing technical characteristics. The current study examines the dependability of 5 items comprising a DBR-MIS designed to measure classroom disruptive behavior. A series of generalizability theory and decision studies were conducted to examine the dependability of each item (calls out, noisy, clowns around, talks to classmates and out of seat), as well as a 3-item composite that was individualized for each student. Seven graduate students rated the behavior of 9 middle-school students on each item over 3 occasions. Ratings were based on 10-min video clips of students during mathematics instruction. Separate generalizability and decision studies were conducted for each item and for a 3-item composite that was individualized for each student based on the highest rated items on the first rating occasion. Findings indicate favorable dependability estimates for 3 of the 5 items and exceptional dependability estimates for the individualized composite.

The influence of student characteristics on the dependability of behavioral observation data

Although generalizability theory has been used increasingly in recent years to investigate the dependability of behavioral estimates, many of these studies have relied on use of general education populations as opposed to those students who are most likely to be referred for assessment due to problematic classroom behavior (e.g., inattention, disruption). The current study investigated the degree to which differences exist in terms of the magnitude of both variance component estimates and dependability coefficients between students nominated by their teachers for Tier 2 interventions due to classroom behavior problems and a general classroom sample (i.e., including both nominated and non-nominated students). The academic engagement levels of 16 (8 nominated, 8 non-nominated) middle school students were measured by 4 trained observers using momentary time-sampling procedures. A series of G and D studies were then conducted to determine whether the 2 groups were similar in terms of the (a) distribution of rating variance and (b) number of observations needed to achieve an adequate level of dependability. Results suggested that the behavior of students in the teacher-nominated group fluctuated more across time and that roughly twice as many observations would therefore be required to yield similar levels of dependability compared with the combined group. These findings highlight the importance of constructing samples of students that are comparable to those students with whom the measurement method is likely to be applied when conducting psychometric investigations of behavioral assessment tools.

The influence of observation length on the dependability of data

Although direct observation is one of the most frequently used assessment methods by school psychologists, studies have shown that the number of observations needed to obtain a dependable estimate of student behavior may be impractical. Because direct observation may be used to inform important decisions about students, it is crucial that data be reliable. Preliminary research has suggested that dependability may be improved by extending the length of individual observations. The purpose of the current study was, therefore, to examine how changes in observational duration affect the dependability of student engagement data. Twenty seventh grade students were each observed for 30-min across 2 days during math instruction. Generalizability theory was then used to calculate reliability-like coefficients for the purposes of intraindividual decision making. Across days, acceptable levels of dependability for progress monitoring (i.e., .70) were achieved through two 30-min observations, three 15-min observations, or four to five 10-min observations. Acceptable levels of dependability for higher stakes decisions (i.e., .80) required over an hour of cumulative observation time. Within a given day, a 15 minute observation was found to be adequate for making low-stakes decisions whereas an hour long observation was necessary for high-stakes decision making. Limitations of the current study and implications for research and practice are discussed.

Assessing influences on intervention implementation: revision of the usage rating profile-intervention

Although treatment acceptability was originally proposed as a critical factor in determining the likelihood that a treatment will be used with integrity, more contemporary findings suggest that whether something is likely to be adopted into routine practice is dependent on the complex interplay among a number of different factors. The Usage Rating Profile-Intervention (URP-I; Chafouleas, Briesch, Riley-Tillman, & McCoach, 2009) was recently developed to assess these additional factors, conceptualized as potentially contributing to the quality of intervention use and maintenance over time. The purpose of the current study was to improve upon the URP-I by expanding and strengthening each of the original four subscales. Participants included 1005 elementary teachers who completed the instrument in response to a vignette depicting a common behavior intervention. Results of exploratory and confirmatory factor analyses, as well as reliability analyses, supported a measure containing 29 items and yielding 6 subscales: Acceptability, Understanding, Feasibility, Family-School Collaboration, System Climate, and System Support. Collectively, these items provide information about potential facilitators and barriers to usage that exist at the level of the individual, intervention, and environment. Information gleaned from the instrument is therefore likely to aid consultants in both the planning and evaluation of intervention efforts.

Generating Usable Knowledge

In this study, a self-report measure of intervention usage, the Usage Rating Profile for Interventions (URP-I) is developed and empirically examined with regard to factor structure and internal consistency. Results supported that intervention usage is associated with at least 4 different constructs, and that a measure consisting of 25 items may provide a reliable index of the 4 factors. The 4 factors identified included Acceptability, Knowledge, Feasibility, and Integrity. Findings extend and integrate existing work-related items to acceptability research aimed at predicting usage. Implications, limitations, and future directions are discussed.