Using cluster-robust standard errors when analyzing group-randomized trials with few clusters

Evaluating two small-sample corrections for fixed-effects standard errors and inferences in multilevel models with heteroscedastic, unbalanced, clustered data

Multilevel modeling (MLM) is commonly used in psychological research to model clustered data. However, data in applied research usually violate one of the essential assumptions of MLM-homogeneity of variance. While the fixed-effect estimates produced by the maximum likelihood method remain unbiased, the standard errors for the fixed effects are misestimated, resulting in inaccurate inferences and inflated or deflated type I error rates. To correct the bias in fixed effects standard errors and provide valid inferences, small-sample corrections such as the Kenward-Roger (KR) adjustment and the adjusted cluster-robust standard errors (CR-SEs) with the Satterthwaite approximation for t tests have been used. The current study compares KR with random slope (RS) models and the adjusted CR-SEs with ordinary least squares (OLS), random intercept (RI) and RS models to analyze small, heteroscedastic, clustered data using a Monte Carlo simulation. Results show the KR procedure with RS models has large biases and inflated type I error rates for between-cluster effects in the presence of level 2 heteroscedasticity. In contrast, the adjusted CR-SEs generally yield results with acceptable biases and maintain type I error rates close to the nominal level for all examined models. Thus, when the interest is only in within-cluster effect, any model with the adjusted CR-SEs could be used. However, when the interest is to make accurate inferences of the between-cluster effect, researchers should use the adjusted CR-SEs with RS to have higher power and guard against unmodeled heterogeneity. We reanalyzed an example in Snijders & Bosker (2012) to demonstrate the use of the adjusted CR-SEs with different models.

The Impact of Restorative Practices on the Use of Out-of-School Suspensions: Results from a Cluster Randomized Controlled Trial

The overuse of exclusionary discipline practices, such as out-of-school suspensions (OSS), has consistently been documented over several decades. The resulting racial discipline disparities and the negative outcomes related to OSS have led policy makers and educators to consider other approaches to school discipline. One such approach, which has gained increasing use in the USA, is restorative practices (RP). However, despite its popularity, the experimental evidence base of the effects of RP and suspensions is extremely limited. To add to this knowledge base, we present findings from a cluster randomized controlled trial with 18 K-12 schools in an urban district in the US Northeast with 5878 students. Multilevel regression results, after 1 year of the intervention, showed no differences in the likelihood of suspension between students in the intervention and control schools and that the effects of the intervention did not vary by race/ethnicity, gender, or student disability status. However, for students in the intervention group, we show reductions in the likelihood of receiving an OSS for students who had previously been suspended.

Les Misérables: An analysis of low SWB across the world

Global trends indicate that the prevalence of low subjective wellbeing is on the rise, though not all regions are equal in terms of both absolute levels and their trajectories. In this paper, we explore the relative importance of individual- and country-level factors in predicting low SWB. Put differently, we ask if a person found themselves behind a veil of ignorance, should they want to know who they will be or what country they will live in to better understand their risk of having low wellbeing. To answer this question, we leverage data from the most extensive wellbeing survey in the world-the Gallup World Poll. We explore people's likelihood of reporting low evaluative wellbeing (that their life is close to the worst possible life on the Cantril ladder) and low experiential wellbeing (reporting having felt angry, sad, stressed, and worried for most of the day yesterday). Using multilevel models on both measures, we show that individual factors have the greatest explanatory power across both measures, but that country level factors are almost four times more important in explaining the variation in low evaluative wellbeing than low experiential wellbeing around the world. We also present evidence that individual and country-level factors interact, suggesting that a complex system of people and places determines people's likelihood of reporting low SWB.

Evaluation of the whole school restorative practices project: One-year impact on discipline incidents

The current study addressed the need for experimental research on Restorative Practices (RP) in its evaluation of the Morningside Center for Teaching Social Responsibility's Whole School RP Project. The study was conducted in a large Northeastern city using a cluster randomized controlled design in 18 elementary, middle, and high schools. In a single year, before the COVID-19 pandemic, and with data from 5878 students, the study found that overall, students in the RP Project schools were less likely to receive a discipline incident record (11.1%) as compared to students in the comparison schools (18.2%). However, differential treatment effects based on sex, race/ethnicity, and disability status were not found. The findings suggest prevention efforts are a cornerstone of practice/policy reforms to reduce the use of exclusionary discipline. Findings also suggest multi-year initiatives are needed to address discipline disparities.