It's not as simple as it sounds: Problems and solutions in accessing and using administrative child welfare data for evaluating the impact of early childhood interventions

Why Are Child and Youth Welfare Support Services Initiated? A First-Time Analysis of Administrative Data on Child and Youth Welfare Services in Austria

Even if numerous children and young people are looked after by child and youth welfare, there are only a few scientific studies on the reasons for this support. The aim of this retrospective descriptive study was to examine the reasons why child and youth welfare was initiated. Therefore, administrative data, collected by the Lower Austrian Child and Youth Welfare Service, from the year 2021 will be presented. On the one hand, the frequencies of the different justifications provided by the social workers and, on the other hand, whether these are primarily based on problems of the parents/caregivers or the children are reported. In 2021, a total of 7760 clarifications of child welfare endangerments were initiated. The descriptive statistical analyses showed that the most frequent concerns were parental overload (49%), behavioral issues (10%), and difficult economic conditions (9%). Although a classification according to the caregiver or child level cannot always be clearly distinguished, there is a trend that in many cases (84% to 99% depending on the type of support) the problems lie at the caregiver level. Further studies are necessary so that the care of such vulnerable groups of people will be better supported by scientific findings.

Leveraging integrated data for program evaluation: Recommendations from the field

Use of administrative data to inform decision making is now commonplace throughout the public sector, including program and policy evaluation. While reuse of these data can reduce costs, improve methodologies, and shorten timelines, challenges remain. This article informs evaluators about the growing field of Integrated Data Systems (IDS), and how to leverage cross-sector administrative data in evaluation work. This article is informed by three sources: a survey of current data integration efforts in the United States (U.S.) (N=63), informational interviews with experts, and internal knowledge cultivated through Actionable Intelligence for Social Policy's (AISP) 12+ years of work in the field. A brief discussion of the U.S. data integration context and history is provided, followed by discussion of tangible recommendations for evaluators, examples of evaluations relying on integrated data, and a list of U.S. IDS sites with publicly available processes for external data requests. Despite the challenges associated with reusing administrative data for program evaluation, IDS offer evaluators a new set of tools for leveraging data across institutional silos.

A contemporary case file analysis of child sexual abuse in institutional settings in England and Wales

BACKGROUND

Despite increased institutional safeguards and regularly updated statutory guidance that stresses safeguarding is 'everybody's responsibility', children continue to be sexually abused in institutional contexts in England and Wales. There also remains a lack of contemporary knowledge about institutions' responses to concerns about [risk of] CSA.

OBJECTIVE

Reviewing Disclosure and Barring Service (DBS) discretionary-decision case files, as a detailed source of information about CSA in institutions occurring in the last 5 years, the aim of this paper was to understand what institutions knew about CSA and inappropriate behaviours that indicated children could be at risk of harm prior to formal disclosures to the DBS and how institutions did, or did not, respond to this knowledge.

PARTICIPANTS AND SETTING

Cases involved 32 male and 11 female barred persons and 19 male and 51 female sexually abused children.

METHOD

Thematic analysis was applied to 43 files where decisions were made by the DBS to add individuals to the Children's Barred list (which prevents them by law from working with children) between 2017 and 2020.

RESULTS

In 79% of cases there was some level of awareness of concerns relating to inappropriate behaviour or CSA, through rumours and gossip, concerns about professionalism, and observed changes in the behaviour of subsequently barred individuals or victims. There was wide variation in the degree to which institutions responded. In a small number of cases action was taken promptly; however, this was not typical.

CONCLUSIONS

Institutions and professionals failed in their duty of care by not taking any action at all, or responding slowly in ways that did not prevent CSA.

The Sobriety Treatment and Recovery Teams program for families with parental substance use: Comparison of child welfare outcomes through 12 months post-intervention

BACKGROUND

The 2018 Family First Prevention Services Act (FFPSA) shifted child welfare funding to interventions proven effective in preserving families with parental substance use and child welfare involvement. The Sobriety Treatment and Recovery Teams (START) program serves this population with FFPSA aligned goals.

OBJECTIVE

This study was the first to test the sustained effects of START from the initial CPS report through 12-months post-intervention.

PARTICIPANTS AND SETTING

Children (n = 784) receiving START services in four sites were compared to 784 children receiving child welfare treatment as usual (TAU).

METHODS

Using child welfare administrative data, children in START were matched to children in TAU using propensity score matching. Outcomes were tested during the intervention period, and at six- and 12-months post-intervention using comparative statistics and multilevel logistic regression.

RESULTS

The odds of START children being placed in out-of-home care (OOHC) during the intervention period were half those of children in TAU (20.3% vs. 35.2%, p < .001, OR = 0.47, 95% CI [0.37, 0.59]). When placed in OOHC, START children were more likely to be reunified with their parents (p = .042, OR = 1.44, 95% CI [0.99, 1.62]). At 12-months post-intervention, 68.5% of START and 56.0% of TAU-served children remained free from both OOHC placement and child abuse and neglect (after multilevel clustering adjustment: p < .001, OR = 1.85, 95% CI [1.41, 2.43]).

CONCLUSION

The primary impact of START was significantly reduced rates of OOHC placement, with results sustained through 12-months post-intervention and after accounting for family clusters and site differences.

Comparative outcomes for Black children served by the Sobriety Treatment and Recovery Teams program for families with parental substance abuse and child maltreatment

INTRODUCTION

An urgent need exists for child welfare and substance use disorder (SUD) interventions that safely preserve Black families, engage parents in treatment services, and improve child and parent outcomes. The Title IV-E Prevention Services Clearinghouse rated The Sobriety Treatment and Recovery Teams (START) as a promising practice for families with parental substance use and child maltreatment. This study is the first to test the effects of START on Black families.

METHODS

This study compared child welfare and parent outcomes for 894 children and their 567 primary parents in three groups: Black children served by START, Black children served in treatment as usual (TAU), and White children served in START. This was a quasi-experimental study using a propensity score matched dataset of START-served children to TAU children. Comparisons included placement in state custody and repeat child abuse or neglect (CA/N) during the intervention period, and at 12 months post-intervention. Generalized linear models accounted for the effects of clustering and unbalanced covariates on outcomes.

RESULTS

In this study, 51.8% of children were neonates or infants at the CPS report. At 12-months post-intervention, 80.6% of Black children served by START, but only 56.0% of Black children in TAU, remained free of both placement in state custody and CA/N (p < .001, OR = 3.27, 95% CI [2.14, 4.98]); these effects held after controlling for family clusters. Black and White families in START received equal SUD treatment and community-based services. For START-served families, parental use of opioids (p = .005, OR = 3.52, 95% CI [1.46, 8.48]) and mental health issues (p = .002, OR = 1.90, 95% CI [1.27, 2.86]), rather than race, predicted child placement in state custody. Parent mental health issues or opioid use doubled or quadrupled, respectively, the odds of failing to achieve early recovery by case closure.

CONCLUSIONS

START is a potent intervention, co-implemented with SUD treatment providers, that kept Black children safely with their families through the intervention and 12-months post-intervention periods. Scaling up effective programs, like START, that align with the goals of the Family First Prevention Services Act might reduce racial disparities and improve child welfare and SUD treatment outcomes.

A Process for Identifying Indicators With Public Data: An Example From Sexual Violence Prevention

Despite advances in the sexual violence (SV) prevention field, practitioners still face challenges with identifying indicators to measure the impact of their prevention strategies. Public data, such as existing administrative and surveillance system data, may be a good option for organizations to examine trends in indicators for the purpose of program evaluation. In this article, we describe a framework and a process for identifying indicators with public data. Specifically, we present the SV Indicator Framework and a five-step indicator review process, which we used to identify indicators for a national SV prevention program. We present the findings of the indicator review and explain how the process could be used by evaluators and program planners within other developing topic areas. Tracking indicators with public data, in conjunction with other evaluation methods, may be a viable option for state-level program evaluations. We discuss limitations and implications for practice and research.

The Cumulative Effect of Prior Maltreatment on Emotional and Physical Health of Children in Informal Kinship Care

OBJECTIVE

Millions of children cared for by their grandparents or other kin without a biological parent present are not part of the foster care system. Maltreatment may have precipitated out-of-home care arrangements, but most children in informal kinship care are not being tracked or receiving services. Importantly, the extent of previous child welfare involvement and its association with well-being among this population are not well known.

METHODS

Kinship caregivers known to social service and community agencies were recruited for this study. Caregivers who agreed to participate rated the physical and emotional health of children under their care. Maternal mental health and demographic data were collected, and matched records of children were retrieved from the state child welfare database. The nested structure of children within families was addressed using a generalized estimating equation model.

RESULTS

The sample included 365 children from 274 families. Only 25% of the children were in private kinship care with no known Child Protective Services (CPS) involvement. An average of 2.37 open CPS cases were observed for those in voluntary kinship care. One out of 4 children scored below the clinical level of emotional health. The number of CPS cases and kin caregiver's parenting stress were inversely associated with child's physical and emotional health. The presence of maternal mental health problem was significantly associated with poor child emotional well-being.

CONCLUSION

Many children in informal kinship care experienced multiple episodes of maltreatment. This study's results add further evidence of the cumulative negative effects of child maltreatment on child well-being. Children in informal kinship care are in need of services to mitigate their traumatic experiences.

Improving Child Welfare Outcomes: Balancing Investments in Prevention and Treatment

To provide objective analyses about the effects of prevention and treatment programs on child welfare outcomes, RAND researchers built a quantitative model that simulated how children enter and flow through the nation's child welfare system. They then used the model to project how different policy options (preventive services, family preservation treatment efforts, kinship care treatment efforts, and a policy package that combined preventive services and kinship care) would affect a child's pathway through the system, costs, and outcomes in early adulthood. This study is the first attempt to integrate maltreatment risk, detection, pathways through the system, and consequences in a comprehensive quantitative model that can be used to simulate the impact of policy changes. This research suggests that expanding both prevention and treatment is needed to achieve the desired policy objectives: Combining options that intervene at different points in the system and increasing both prevention and treatment generates stronger effects than would any single option. The simulation model identifies ways to increase both targeted prevention and treatment while achieving multiple objectives: reducing maltreatment and the number of children entering the system, improving a child's experience moving through the system, and improving outcomes in young adulthood. These objectives can all be met while also reducing total child welfare system costs. A policy package combining expanded prevention and kinship supports pays for itself: There is a net cost reduction in the range of 3 to 7 percent of total spending (or approximately $5.2 billion to $10.5 billion saved against the current baseline of $155.9 billion) for a cohort of children born over a five-year period.

Visualizing and Describing Foster Care Placement Pathways

This paper introduces a flowchart-based methodology for describing the movement of foster youth in and out of placements of differing types and durations. This longitudinal methodology is designed to be sufficiently simple to appeal to policymakers and administrators seeking to chart the movement of groups of youth over time and the sequencing of their placements, and sufficiently descriptive to be of use to researchers seeking to predict the placement trajectories of subgroups of foster youth. The paper provides an example of the use of the method drawing upon state administrative data from a large study of preadolescent and adolescent youth in foster care situated in Oregon. Implications for the application of the methodology to different issues of interest to researchers, policymakers, and administrators are discussed.

New Methods to Address Old Challenges: The Use of Administrative Data for Longitudinal Replication Studies of Child Maltreatment

Administrative data are crucial to the “big data” revolution of social science and have played an important role in the development of child maltreatment research. These data are also of value to administrators, policy makers, and clinicians. The focus of this paper is the use of administrative data to produce and replicate longitudinal studies of child maltreatment. Child protection administrative data have several advantages. They are often population-based, and allow longitudinal examination of child maltreatment and complex multi-level analyses. They also allow comparison across subgroups and minority groups, remove burden from individuals to disclose traumatic experiences, and can be less biased than retrospective recall. Finally, they can be linked to data from other agencies to explore comorbidity and outcomes, and are comparatively cost and time effective. The benefits and challenges associated with the use of administrative data for longitudinal child maltreatment research become magnified when these data are used to produce replications. Techniques to address challenges and support future replication efforts include developing a biographical understanding of the systems from which the data are drawn, using multiple data sources to contextualize the data and research results, recognizing and adopting various approaches to replication, and documenting all data coding and manipulation processes. These techniques are illustrated in this paper via a case study of previous replication work.

Summary of comparison between FFT-CW® and Usual Care sample from Administration for Children's Services

This evaluation compared the efficiency and effectiveness of Functional Family Therapy-Child Welfare (FFT-CW^®, n=1625) to Usual Care (UC: n=2250) in reducing child maltreatment. FFT-CW^® is a continuum of care model based on the family's risk status. In a child welfare setting, families received either UC or FFT-CW^® in a quasi-experimental, stepped wedge design across all five boroughs of New York City. The families were matched using stratified propensity scoring on their pre-service risk status and followed for 16 months. The ethnically diverse sample included African American (36%), Asian (4%); Hispanic (49%), and Non-Hispanic White (6%) or Other (6%) participants. Referral reasons included abuse or neglect (57.4%), child service needs (56.9%) or child health and safety concerns (42.8%). Clinical process variables included staff fidelity, service duration, and number of contacts. Positive outcomes included whether all clinical goals were met and negative outcomes included transfers, outplacement, recurring allegations and service participation within 16 months of the case open date. Families receiving FFT-CW^® completed treatment more quickly than UC and they were significantly more likely to meet all of the planned service goals. Higher treatment fidelity was associated with more favorable outcomes. Fewer FFT-CW^® families were transferred to another program at closing, and they had fewer recurring allegations. FFT-CW^® had fewer out-of-home placements in families with higher levels of risk factors. The FFT-CW^® program was more efficient in completing service, and more effective than UC in meeting treatment goals while also avoiding adverse outcomes.

Assessing child maltreatment prevention via administrative data systems: A case example of reproducibility

Critical issues about scientific reproducibility have been raised about biomedical research, including the reliability of data and analyses within a given study. The case example in this article examined a reproducibility issue pertaining to the use of administrative data systems for evaluation of child maltreatment (CM) prevention, making use of a prevention study conducted over a decade ago that provided a unique opportunity. The place-randomization study, which randomized counties to condition, found that community-wide implementation of a parenting and family support intervention produced positive impact on county-wide rates for substantiated CM cases and out-of-home placements, documented through a state information system. The key consideration is whether and to what extent the administrative record data re-examined retroactively a decade later for the original study's time period would yield comparable results to those based on data acquired at the time of the study. The results indicated that despite small changes over time, the same data patterns and statistical effects were reproducible for the two archival outcome variables. For substantiated CM, the reproduced analyses reflected higher effect sizes and a clear pattern of reduction as a function of intervention. For out-of-home placements, effect sizes were quite comparable to the original ones, reflecting preventive impact. Overall, this case study illustrated the verifiability of data reproducibility in the context of a population outcome evaluation, which underscores the importance of reliable population-prevalence measurement as an essential part of a comprehensive public health strategy aimed at the prevention of CM.

Case file coding of child maltreatment: Methods, challenges, and innovations in a longitudinal project of youth in foster care

State social service agency case files are a common mechanism for obtaining information about a child's maltreatment history, yet these documents are often challenging for researchers to access, and then to process in a manner consistent with the requirements of social science research designs. Specifically, accessing and navigating case files is an extensive undertaking, and a task that many researchers have had to maneuver with little guidance. Even after the files are in hand and the research questions and relevant variables have been clarified, case file information about a child's maltreatment exposure can be idiosyncratic, vague, inconsistent, and incomplete, making coding such information into useful variables for statistical analyses difficult. The Modified Maltreatment Classification System (MMCS) is a popular tool used to guide the process, and though comprehensive, this coding system cannot cover all idiosyncrasies found in case files. It is not clear from the literature how researchers implement this system while accounting for issues outside of the purview of the MMCS or that arise during MMCS use. Finally, a large yet reliable file coding team is essential to the process, however, the literature lacks training guidelines and methods for establishing reliability between coders. In an effort to move the field toward a common approach, the purpose of the present discussion is to detail the process used by one large-scale study of child maltreatment, the Studying Pathways to Adjustment and Resilience in Kids (SPARK) project, a longitudinal study of resilience in youth in foster care. The article addresses each phase of case file coding, from accessing case files, to identifying how to measure constructs of interest, to dealing with exceptions to the coding system, to coding variables reliably, to training large teams of coders and monitoring for fidelity. Implications for a comprehensive and efficient approach to case file coding are discussed.