A Case Study of the Development of a Valid and Pragmatic Implementation Science Measure: The Barriers and Facilitators in Implementation of Task-Sharing Mental Health Interventions (BeFITS-MH) Measure

Background

Few implementation science (IS) measures have been evaluated for validity, reliability and utility - the latter referring to whether a measure captures meaningful aspects of implementation contexts. In this case study, we describe the process of developing an IS measure that aims to assess Barriers and Facilitators in Implementation of Task-Sharing in Mental Health services (BeFITS-MH), and the procedures we implemented to enhance its utility.

Methods

We summarize conceptual and empirical work that informed the development of the BeFITS-MH measure, including a description of the Delphi process, detailed translation and local adaptation procedures, and concurrent pilot testing. As validity and reliability are key aspects of measure development, we also report on our process of assessing the measure's construct validity and utility for the implementation outcomes of acceptability, appropriateness, and feasibility.

Results

Continuous stakeholder involvement and concurrent pilot testing resulted in several adaptations of the BeFITS-MH measure's structure, scaling, and format to enhance contextual relevance and utility. Adaptations of broad terms such as "program," "provider type," and "type of service" were necessary due to the heterogeneous nature of interventions, type of task-sharing providers employed, and clients served across the three global sites. Item selection benefited from the iterative process, enabling identification of relevance of key aspects of identified barriers and facilitators, and what aspects were common across sites. Program implementers' conceptions of utility regarding the measure's acceptability, appropriateness, and feasibility were seen to cluster across several common categories.

Conclusions

This case study provides a rigorous, multi-step process for developing a pragmatic IS measure. The process and lessons learned will aid in the teaching, practice and research of IS measurement development. The importance of including experiences and knowledge from different types of stakeholders in different global settings was reinforced and resulted in a more globally useful measure while allowing for locally-relevant adaptation. To increase the relevance of the measure it is important to target actionable domains that predict markers of utility (e.g., successful uptake) per program implementers' preferences. With this case study, we provide a detailed roadmap for others seeking to develop and validate IS measures that maximize local utility and impact.

Assessment of the Perceived Importance of Religion in Couple Relationships in Christians, Muslims, Buddhists, and the Nonreligious

Couple relationship standards (beliefs about what makes for a satisfying couple relationship) have not included standards held about religion, which is surprising given how important religion is in many parts of the world. In the current study, we developed the Importance of Religion in Couple Relationships Scale (IRCRS) with the aim of having a scale suitable for use across different cultural and religious groups. The IRCRS was administered to three samples: 354 Pakistani residents (178 females, 176 males) who identified as Muslim; 274 Thai residents (157 females, 117 males) who identified as Buddhist; and 165 Westerners (resident in Australia or the United States, 60 males, 105 females) who identified as either not religious (n = 74) or Christian (n = 91). We developed a 13-item measure with a two level structure yielding an overall importance of religion score. The items in the IRCRS had acceptable cross-cultural structural invariance in a multi-group confirmatory factor analysis of the Pakistani Muslims, Thai Buddhists, and Westerners. Pakistani Muslims endorsed IRCRS standards most strongly, Western Christians next most strongly, Thai Buddhists next, and Westerners with no religion least strongly. There were no gender differences, and only very small differences by relationship status. The IRCRS can be used in future research to investigate the association of religious relationship standards with couple relationship satisfaction and might be a useful clinical tool to assess the importance of religion to couples.

Missing Piece of the Puzzle: Evaluating a General Meaning of Sex Measure

The presence of and search for meaning has divergent and salient outcomes for individuals' physical, mental, and emotional well-being. Although numerous domains of meaning have been examined, the literature on the meaning of sex is sparse and lacks a quantitative measure. In this study, we evaluated a new general Meaning of Sex (MOS) Measure that captured the presence of (MOS-P) and search for (MOS-S) the meaning of sex by analyzing how these subscales were associated with important relational and sexual outcomes (i.e., relationship stability and satisfaction and sexual satisfaction) in two samples. For Sample 1, all participants (N = 865) reported engaging in what they considered to be sexual intercourse and were either single or in a relationship. For Sample 2, all participants (N = 884) reported being in a committed relationship for at least two years. With both samples, confirmatory factor analyses showed good construct validity as the two subscales were distinct from one another and had good reliability. The analyses also demonstrated divergent validity between both scales and meaning in life, sexual motivations, sexual beliefs, and religiosity. The MOS-S had a strong (and negative) association with relationship stability in both samples. The MOS-P had a strong (and positive) association with relationship satisfaction in both samples.

Are Healthcare Organizations Ready for Change? Comment on "Development and Content Validation of a Transcultural Instrument to Assess Organizational Readiness for Knowledge Translation in Healthcare Organizations: The OR4KT"

Worldwide most health systems are facing a series of common challenges characterized by the increasing burden of chronic diseases and multimorbidity, and the accelerated pace of biomedical and technological innovations, on the other side. There is a growing recognition that many changes are needed at the macro, meso and micro management levels to tackle these challenges. Therefore, knowing if healthcare organizations are ready for change is a key issue, as high organizational readiness for change (ORC) has been positively related with higher organizational effort and staff motivation for overcoming barriers and setbacks in change endeavours. In practice, readiness for change is not commonly measured and there is a need of adequate metrics for it. In this commentary, a new tool for measuring readiness change is reviewed, the OR4KT. It has been developed based on a solid theoretical background and with the involvement of experts and potential users in the design and it has been tested and validated in three languages and in different organizational settings. Although its generalizability needs to be further tested, it seems to be a promising and useful tool to diagnose if organizations are ready to implement evidence-informed changes. A broader recognition of the key role that the science of implementation can play in the success of much needed transformations in healthcare provides a good opportunity for the dissemination of the OR4KT.

Patient-Reported Outcome Measures in the Food and Drug Administration Pilot Compendium: Meeting Today's Standards for Patient Engagement in Development?

BACKGROUND

In 2016, the Food and Drug Administration (FDA) released a Pilot Clinical Outcome Assessment Compendium (COA Compendium) intended to foster patient-focused drug development (PFDD). However, it is unclear whether patient perspectives were solicited during development or validation of the included patient-reported outcome (PRO) measures.

OBJECTIVE

To examine the pedigree of a sample of measures included in the COA Compendium.

METHODS

PROs included in chapters 1 or 2 of the COA Compendium were extracted and three reviewers independently searched PubMed and Google to identify information on measure pedigree. Data on method and stage of measure development where patient engagement took place were documented.

RESULTS

Among the 26 evaluated PRO measures, we were unable to identify information on development or validation on nearly half the sample (n = 12). Among the remaining 14 measures, 5 did not include any evidence of patient engagement; 2 engaged patients during concept elicitation only; 1 engaged patients during psychometric validation only; and 6 engaged patients during both concept elicitation and cognitive interviewing. Measures either previously qualified or submitted for qualification were more likely to include patient engagement.

CONCLUSIONS

For the FDA Pilot COA Compendium to fulfill its purpose of fostering PFDD, it needs fine-tuning to reflect today's standards, improving transparency and facilitating clear identification of included measures so that the level of patient engagement, among other factors, can be properly assessed. Suggested improvements include identifying clinical trials that correspond to the COA Compendium's use in drug development; more clearly identifying which measure is referred to; and including only those measures that already qualified or undergoing qualification.

Development and Pilot Test of the Workplace Readiness Questionnaire, a Theory-Based Instrument to Measure Small Workplaces' Readiness to Implement Wellness Programs

PURPOSE

To develop a theory-based questionnaire to assess readiness for change in small workplaces adopting wellness programs.

DESIGN

In developing our scale, we first tested items via "think-aloud" interviews. We tested the revised items in a cross-sectional quantitative telephone survey.

SETTING

The study setting comprised small workplaces (20-250 employees) in low-wage industries.

SUBJECTS

Decision-makers representing small workplaces in King County, Washington (think-aloud interviews, n = 9), and the United States (telephone survey, n = 201) served as study subjects.

MEASURES

We generated items for each construct in Weiner's theory of organizational readiness for change. We also measured workplace characteristics and current implementation of workplace wellness programs.

ANALYSIS

We assessed reliability by coefficient alpha for each of the readiness questionnaire subscales. We tested the association of all subscales with employers' current implementation of wellness policies, programs, and communications, and conducted a path analysis to test the associations in the theory of organizational readiness to change.

RESULTS

Each of the readiness subscales exhibited acceptable internal reliability (coefficient alpha range, .75-.88) and was positively associated with wellness program implementation ( p < .05). The path analysis was consistent with the theory of organizational readiness to change, except change efficacy did not predict change-related effort.

CONCLUSION

We developed a new questionnaire to assess small workplaces' readiness to adopt and implement evidence-based wellness programs. Our findings also provide empirical validation of Weiner's theory of readiness for change.