An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention

Pragmatic fidelity measurement in youth service settings

Background

Fidelity measurement is critical for developing, evaluating, and implementing evidence-based treatments (EBTs). However, traditional fidelity measurement tools are often not feasible for community-based settings. We developed a short fidelity rating form for the Collaborative Problem Solving (CPS) approach from an existing manualized coding system that requires extensive training. We examined the reliability and accuracy of this short form when completed by trained observers, untrained observers, and self-reporting providers to evaluate multiple options for reducing barriers to fidelity measurement in community-based settings.

Methods

Community-based treatment providers submitted recordings of youth service sessions in which they did, or did not, use CPS. For 60 recordings, we compared short-form fidelity ratings assigned by trained observers and untrained observers to those provided by trained observers on the manualized coding system. For 141 recordings, we compared providers' self-reported fidelity on the short form to ratings provided by trained observers on the manualized coding system and examined providers' accuracy as a function of their global fidelity.

Results & Conclusions

The short form was reliable and accurate for trained observers. An assigned global integrity score and a calculated average of component scores on the short form, but not component scores themselves, were reliable and accurate for observers who had CPS expertise but no specific training on rating CPS fidelity. When providers self-reported fidelity on the short form, their global integrity score was a reliable estimate of their CPS integrity; however, providers with better CPS fidelity were most accurate in their self-reports. We discuss the costs and benefits of these more pragmatic fidelity measurement options in community-based settings.

A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care

Background

Evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. When programs have been taken to scale, declines in the quality of implementation diminish intervention effects. Gold-standard methods of implementation monitoring are cost-prohibitive and impractical in resource-scarce delivery systems. Technological developments using computational linguistics and machine learning offer an opportunity to assess fidelity in a low burden, timely, and comprehensive manner.

Methods

In this study, we test two natural language processing (NLP) methods [i.e., Term Frequency-Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers (BERT)] to assess the delivery of the Family Check-Up 4 Health (FCU4Health) program in a type 2 hybrid effectiveness-implementation trial conducted in primary care settings that serve primarily Latino families. We trained and evaluated models using 116 English and 81 Spanish-language transcripts from the 113 families who initiated FCU4Health services. We evaluated the concurrent validity of the TF-IDF and BERT models using observer ratings of program sessions using the COACH measure of competent adherence. Following the Implementation Cascade model, we assessed predictive validity using multiple indicators of parent engagement, which have been demonstrated to predict improvements in parenting and child outcomes.

Results

Both TF-IDF and BERT ratings were significantly associated with observer ratings and engagement outcomes. Using mean squared error, results demonstrated improvement over baseline for observer ratings from a range of 0.83-1.02 to 0.62-0.76, resulting in an average improvement of 24%. Similarly, results demonstrated improvement over baseline for parent engagement indicators from a range of 0.81-27.3 to 0.62-19.50, resulting in an approximate average improvement of 18%.

Conclusions

These results demonstrate the potential for NLP methods to assess implementation in evidence-based parenting programs delivered at scale. Future directions are presented.

Trial registration

NCT03013309 ClinicalTrials.gov.

Health behaviour outcomes of a family based intervention for paediatric obesity in primary care: A randomized type II hybrid effectiveness-implementation trial

BACKGROUND

Paediatric obesity is a multifaceted public health problem. Family based behavioural interventions are the recommended approach for the prevention of excess weight gain in children and adolescents, yet few have been tested under "real-world" conditions.

OBJECTIVES

To evaluate the effectiveness of a family based intervention, delivered in coordination with paediatric primary care, on child and family health outcomes.

METHODS

A sample of 240 families with racially and ethnically diverse (86% non-White) and predominantly low-income children (49% female) ages 6 to 12 years (M = 9.5 years) with body mass index (BMI) ≥85th percentile for age and gender were identified in paediatric primary care. Participants were randomized to either the Family Check-Up 4 Health (FCU4Health) program (N = 141) or usual care plus information (N = 99). FCU4Health, an assessment-driven individually tailored intervention designed to preempt excess weight gain by improving parenting skills was delivered for 6 months in clinic, at home and in the community. Child BMI and body fat were assessed using a bioelectrical impedance scale and caregiver-reported health behaviours (eg, diet, physical activity and family health routines) were obtained at baseline, 3, 6 and 12 months.

RESULTS

Change in child BMI and percent body fat did not differ by group assignment. Path analysis indicated significant group differences in child health behaviours at 12 months, mediated by improved family health routines at 6 months.

CONCLUSION

The FCU4Health, delivered in coordination with paediatric primary care, significantly impacted child and family health behaviours that are associated with the development and maintenance of paediatric obesity. BMI did not significantly differ.

Motivational Interviewing and Caregiver Engagement in the Family Check-Up 4 Health

Motivational interviewing (MI) is a therapeutic style in which a provider elicits client motivation and helps strengthen commitment to change (Miller and Rollnick 2002). The original Family Check-Up (FCU; Dishion and Stormshak 2007)-and the adapted version for improving health behaviors in primary care, the Family Check-Up 4 Health (FCU4Health; Smith et al. 2018a)-are brief, assessment-driven, and family-centered preventive interventions that use MI to improve parent engagement in services to improve parenting and prevent negative child outcomes. This study examines the role of MI in the Raising Healthy Children project, a randomized trial to test the effectiveness of the FCU4Health for the prevention of obesity in pediatric primary care, with data from the 141 families assigned to receive the FCU4Health. Families were eligible for the study if the child was between 5.5 and 12 years of age at the time of identification and had a BMI ≥ 85th percentile for age and gender at the most recent visit to their primary care provider. MI skills at the first session predicted caregiver in-session active engagement, attendance at follow-up parenting sessions, and improvements in motivation to address child health and behavior goals. Baseline characteristics of the family (i.e., child health diagnosis, caregiver baseline depression, motivation, and Spanish language preference) had differential associations with responsiveness and MI skills. This study has implications for program development, provider training, and fidelity monitoring.

The Family Check-Up 4 Health: Study protocol of a randomized type II hybrid effectiveness-implementation trial in integrated primary care (the healthy communities 4 healthy students study)

BACKGROUND

Parenting interventions like the Family Check-Up have demonstrated effects on child physical and behavioral health outcomes. However, access to these programs is limited, particularly for populations experiencing health disparities. Primary care settings have become recognized as a potential delivery system in which these programs may be implemented at scale. The purpose of this trial is to test the effectiveness of the Family Check-Up 4 Health (FCU4Health) program, an adaptation of the FCU for primary care, and assess program implementation in an integrated primary care setting.

METHODS

We will conduct a hybrid type 2 effectiveness-implementation trial in partnership with a primary care clinic in a low-income, majority Latino community. Families with 2- to 5-year-old children will be eligible to participate. Families will be randomized to receive the intervention (n = 130) or services as usual (n = 70) and will be assessed annually over three years. Outcomes are informed by the RE-AIM framework (i.e., reach, effectiveness, adoption, implementation, and maintenance). Effectiveness outcomes include child health behaviors (e.g., Dietary Screener Questionnaire), behavioral health (e.g., Strengths and Difficulties Questionnaire), and parenting (e.g., Proactive Parenting). Early stage implementation outcomes are also included (e.g., cost, acceptability, appropriateness, and feasibility). Effectiveness outcomes will be assessed via intent-to-treat (ITT) analyses. Implementation outcomes will be primarily descriptive with comparisons to prior trials of FCU4Health and the original FCU.

PROJECTED OUTCOMES

This trial will provide evidence related to the potential of integrated primary care settings to deliver evidence-based preventive interventions with a dual focus on behavioral and physical health.

Effectiveness and Efficiency of Observationally Assessing Fidelity to a Family-Centered Child Intervention: A Quasi-Experimental Study

Assessment of fidelity that is effective, efficient, and differentiates from usual practices is critical for effectively implementing evidence-based programs for families. This quasi-experiemntal study sought to determine whether observational ratings of fidelity to the Family Check-Up (FCU) could differentiate between levels of clinician training in the model, and from services as usual, and whether rating segments of sessions could be equivalent to rating complete sessions. Coders rated 75 videotaped sessions-complete and 20-min segments-for fidelity, using a valid and reliable rating system across three groups: (a) highly trained in FCU with universal, routine monitoring; (b) minimally trained in FCU with optional, variable monitoring; and (c) services as usual with no training in the FCU. We hypothesized that certain dimensions of fidelity would differ by training, whereas others would not. The results indicated that, as expected, one dimension of fidelity to the FCU, Conceptually accurate to the FCU, was reliably different between the groups (χ² = 44.63, p < .001). The differences observed were in the expected direction, showing higher scores for therapists with more training. The rating magnitude of session segments largely did not differ from those of complete session ratings; however, interrater reliabilities were low for the segments. Although observational ratings were shown to be sensitive to the degree of training in the FCU on a unique and theoretically critical dimension, observational coding of complete sessions is resource intensive and limits scalability. Additional work is needed to reduce the burden of assessing fidelity to family-centered programs.

George SM, Prado G, Mauricio AM, Chiapa A, Bruening MM, Dishion TJ. The Family Check-Up 4 Health (FCU4Health): Applying Implementation Science Frameworks to the Process of Adapting an Evidence-Based Parenting Program for Prevention of Pediatric Obesity and Excess Weight Gain in Primary Care

Implementation experts have recently argued for a process of "scaling out" evidence-based interventions, programs, and practices (EBPs) to improve reach to new populations and new service delivery systems. A process of planned adaptation is typically required to integrate EBPs into new service delivery systems and address the needs of targeted populations while simultaneously maintaining fidelity to core components. This process-oriented paper describes the application of an implementation science framework and coding system to the adaptation of the Family Check-Up (FCU), for a new clinical target and service delivery system-prevention of obesity and excess weight game in primary care. The original FCU has demonstrated both short- and long-term effects on obesity with underserved families across a wide age range. The advantage of adapting such a program is the existing empirical evidence that the intervention improves the primary mediator of effects on the new target outcome. We offer a guide for determining the levels of evidence to undertake the adaptation of an existing EBP for a new clinical target. In this paper, adaptation included shifting the frame of the intervention from one of risk reduction to health promotion; adding health-specific assessments in the areas of nutrition, physical activity, sleep, and media parenting behaviors; family interaction tasks related to goals for health and health behaviors; and coordinating with community resources for physical health. We discuss the multi-year process of adaptation that began by engaging the FCU developer, community stakeholders, and families, which was then followed by a pilot feasibility study, and continues in an ongoing randomized effectiveness-implementation hybrid trial. The adapted program is called the Family Check-Up 4 Health (FCU4Health). We apply a comprehensive coding system for the adaptation of EBPs to our process and also provide a side-by-side comparison of behavior change techniques for obesity prevention and management used in the original FCU and in the FCU4Health. These provide a rigorous means of classification as well as a common language that can be used when adapting other EBPs for context, content, population, or clinical target. Limitations of such an approach to adaptation and future directions of this work are discussed.

An individually tailored family-centered intervention for pediatric obesity in primary care: study protocol of a randomized type II hybrid effectiveness-implementation trial (Raising Healthy Children study)

BACKGROUND

Pediatric obesity is a multi-faceted public health concern that can lead to cardiovascular diseases, cancers, and early mortality. Small changes in diet, physical activity, or BMI can significantly reduce the possibility of developing cardiometabolic risk factors. Family-based behavioral interventions are an underutilized, evidence-based approach that have been found to significantly prevent excess weight gain and obesity in children and adolescents. Poor program availability, low participation rates, and non-adherence are noted barriers to positive outcomes. Effective interventions for pediatric obesity in primary care are hampered by low family functioning, motivation, and adherence to recommendations.

METHODS

This (type II) hybrid effectiveness-implementation randomized trial tests the Family Check-Up 4 Health (FCU4Health) program, which was designed to target health behavior change in children by improving family management practices and parenting skills, with the goal of preventing obesity and excess weight gain. The FCU4Health is assessment driven to tailor services and increase parent motivation. A sample of 350 families with children aged 6 to 12 years who are identified as overweight or obese (BMI ≥ 85th percentile for age and gender) will be enrolled at three primary care clinics [two Federally Qualified Healthcare Centers (FQHCs) and a children's hospital]. All clinics serve predominantly Medicaid patients and a large ethnic minority population, including Latinos, African Americans, and American Indians who face disparities in obesity, cardiometabolic risk, and access to care. The FCU4Health will be coordinated with usual care, using two different delivery strategies: an embedded approach for the two FQHCs and a referral model for the hospital-based clinic. To assess program effectiveness (BMI, body composition, child health behaviors, parenting, and utilization of support services) and implementation outcomes (such outcomes as acceptability, adoption, feasibility, appropriateness, fidelity, and cost), we use a multi-method and multi-informant assessment strategy including electronic health record data, behavioral observation, questionnaires, interviews, and cost capture methods.

DISCUSSION

This study has the potential to prevent excess weight gain, obesity, and health disparities in children by establishing the effectiveness of the FCU4Health and collecting information critical for healthcare decision makers to support sustainable implementation of family-based programs in primary care.

TRIAL REGISTRATION

NCT03013309 ClinicalTrials.gov.