Framework for the development and evaluation of complex interventions: gap analysis, workshop and consultation-informed update

BACKGROUND

The Medical Research Council published the second edition of its framework in 2006 on developing and evaluating complex interventions. Since then, there have been considerable developments in the field of complex intervention research. The objective of this project was to update the framework in the light of these developments. The framework aims to help research teams prioritise research questions and design, and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods.

METHODS

There were four stages to the update: (1) gap analysis to identify developments in the methods and practice since the previous framework was published; (2) an expert workshop of 36 participants to discuss the topics identified in the gap analysis; (3) an open consultation process to seek comments on a first draft of the new framework; and (4) findings from the previous stages were used to redraft the framework, and final expert review was obtained. The process was overseen by a Scientific Advisory Group representing the range of relevant National Institute for Health Research and Medical Research Council research investments.

RESULTS

Key changes to the previous framework include (1) an updated definition of complex interventions, highlighting the dynamic relationship between the intervention and its context; (2) an emphasis on the use of diverse research perspectives: efficacy, effectiveness, theory-based and systems perspectives; (3) a focus on the usefulness of evidence as the basis for determining research perspective and questions; (4) an increased focus on interventions developed outside research teams, for example changes in policy or health services delivery; and (5) the identification of six 'core elements' that should guide all phases of complex intervention research: consider context; develop, refine and test programme theory; engage stakeholders; identify key uncertainties; refine the intervention; and economic considerations. We divide the research process into four phases: development, feasibility, evaluation and implementation. For each phase we provide a concise summary of recent developments, key points to address and signposts to further reading. We also present case studies to illustrate the points being made throughout.

LIMITATIONS

The framework aims to help research teams prioritise research questions and design and conduct research with an appropriate choice of methods, rather than to provide detailed guidance on the use of specific methods. In many of the areas of innovation that we highlight, such as the use of systems approaches, there are still only a few practical examples. We refer to more specific and detailed guidance where available and note where promising approaches require further development.

CONCLUSIONS

This new framework incorporates developments in complex intervention research published since the previous edition was written in 2006. As well as taking account of established practice and recent refinements, we draw attention to new approaches and place greater emphasis on economic considerations in complex intervention research. We have introduced a new emphasis on the importance of context and the value of understanding interventions as 'events in systems' that produce effects through interactions with features of the contexts in which they are implemented. The framework adopts a pluralist approach, encouraging researchers and research funders to adopt diverse research perspectives and to select research questions and methods pragmatically, with the aim of providing evidence that is useful to decision-makers.

FUTURE WORK

We call for further work to develop relevant methods and provide examples in practice. The use of this framework should be monitored and the move should be made to a more fluid resource in the future, for example a web-based format that can be frequently updated to incorporate new material and links to emerging resources.

FUNDING

This project was jointly funded by the Medical Research Council (MRC) and the National Institute for Health Research (Department of Health and Social Care 73514).

A scoping review of evaluation frameworks and their applicability to real-world physical activity and dietary change programme evaluation

BACKGROUND

Physical activity and dietary change programmes play a central role in addressing public health priorities. Programme evaluation contributes to the evidence-base about these programmes; and helps justify and inform policy, programme and funding decisions. A range of evaluation frameworks have been published, but there is uncertainty about their usability and applicability to different programmes and evaluation objectives, and the extent to which they are appropriate for practitioner-led or researcher-led evaluation. This review appraises the frameworks that may be applicable to evaluation of physical activity and/or dietary change programmes, and develops a typology of the frameworks to help guide decision making by practitioners, commissioners and evaluators.

METHODS

A scoping review approach was used. This included a systematic search and consultation with evaluation experts to identify evaluation frameworks and to develop a set of evaluation components to appraise them. Data related to each framework's general characteristics and components were extracted. This was used to construct a typology of the frameworks based on their intended programme type, evaluation objective and format. Each framework was then mapped against the evaluation components to generate an overview of the guidance included within each framework.

RESULTS

The review identified 71 frameworks. These were described variously in terms of purpose, content, or applicability to different programme contexts. The mapping of frameworks highlighted areas of overlap and strengths and limitations in the available guidance. Gaps within the frameworks which may warrant further development included guidance on participatory approaches, non-health and unanticipated outcomes, wider contextual and implementation factors, and sustainability.

CONCLUSIONS

Our typology and mapping signpost to frameworks where guidance on specific components can be found, where there is overlap, and where there are gaps in the guidance. Practitioners and evaluators can use these to identify, agree upon and apply appropriate frameworks. Researchers can use them to identify evaluation components where there is already guidance available and where further development may be useful. This should help focus research efforts where it is most needed and promote the uptake and use of evaluation frameworks in practice to improve the quality of evaluation and reporting.

The structure of qualitative studies: a bibliometric pattern of biomedical literature

The lack of knowledge in the biomedical literature regarding the validity of qualitative studies might be related to the lower number of qualitative studies that have been published. The criticisms range from a lack of theoretical depth to the superficial discussions of empirical findings. The aim of this study was to explore the bibliometric entities and the trends in the structure of qualitative research in the biomedical literature. A bibliometric analysis and mapping of the biomedical literature were used. The number of studies selected was 1,725. The heath themes with the most publications included Health Management (12%) and Women's Health (9.8%), while the authors of the studies had academic affiliation in 76 different countries. The sample sizes were between 11 and 20 participants (27.13%) and the Grounded Theory framework (9.04%) stood out. The improved structuring of a qualitative research extends the effective communication between health providers and researchers, and support in the management of clinical situations.

Integrating program-tailored universal trauma screening into HIV care: an evidence-based participatory approach

Trauma is prevalent among HIV-infected populations and associated with negative HIV care outcomes. While several agencies have called for integrating trauma-informed care into HIV services, uptake has been limited and effective methods have not been delineated. To develop comprehensive trauma screening strategies tailored to a large, urban HIV care center, we aimed to first understand provider and staff perceptions of the center's current level of trauma screening and barriers and facilitators to universal screening uptake. We used a mixed-methods convergent parallel design: quantitative assessments with 31 providers and staff and in-depth interviews with 19 providers and staff. Quantitative assessments highlighted center strengths (i.e. assessing effects of trauma, explaining care plans to patients, providing opportunities for questions) and gaps (i.e. asking about head injuries, emotional safety). Qualitative interviews suggested center screening practices were highly-variable and limited, identified gaps in interdepartmental communication regarding results of trauma screening that led to repeated screenings and potential patient re-traumatization, screening barriers (i.e. limited time, competing clinical priorities) and facilitators (i.e. provider-driven, standardized, non-disruptive screening with clear cut-points and follow-up steps). This study provides tangible strategies for the center to begin integrating universal trauma screening, many of which could be adapted by other U.S. HIV clinics.

Aplicação do Estudo de Avaliabilidade na área da saúde: uma revisão integrativa

RESUMO Objetivou-se analisar a produção científica sobre a aplicação dos Estudos de Avaliabilidade na área da saúde. Trata-se de uma revisão integrativa da literatura realizada de setembro a outubro de 2017, com busca de pesquisas avaliativas com aplicação de um Estudo de Avaliabilidade, na Biblioteca Virtual em Saúde, Scopus e Web of Science. Analisaram-se 21 artigos, 57,1% publicados nos últimos cinco anos. Os resultados demonstram que há incorporação de aspectos importantes da teoria dos Estudos de Avaliabilidade nos estudos, tais como modelo teórico norteador, uso das etapas próprias de um Estudo de Avaliabilidade, realização da modelagem do programa e associação de fontes de evidências. Conclui-se que o uso do Estudo de Avaliabilidade vem aumentando na área da saúde, entretanto, há necessidade de melhorar sua qualidade, especialmente quanto ao objetivo e questão de pesquisa, descrição do envolvimento dos autores e autorização do local de pesquisa, envolvimento dos stakeholders, descrição dos procedimentos de análise dos dados, estratégias para garantir validade e confiabilidade dos dados, e contribuição para aumento do conhecimento acadêmico, explicitando as contribuições potenciais do Estudo de Avaliabilidade e da avaliação em saúde.

An Impact Mapping Method to Generate Robust Qualitative Evaluation of Community-Based Research Programs for Youth and Adults

Ripple Effect Mapping (REM) is an evaluation approach that has traditionally been used in community settings to visually map the impact of programming and community interventions. This manuscript utilizes the Community Capitals Framework (CCF) to inform REM and to better highlight the changes and impact between various levels of a community, following a childhood obesity prevention intervention. The addition of in-depth qualitative analyses makes this approach particularly useful for the evaluation of interventions with a research-community partnership focus. The objective of this study was to describe a CCF-informed REM approach with detailed protocol, training, and application to the community-based, childhood obesity prevention intervention, iCook 4-H, which targeted youth and adult pairs. This protocol includes the steps required to prepare for REM sessions of, ideally, six youth and adult pairs, one facilitator, and one or two evaluators/note takers. REM sessions typically begin with an icebreaker and appreciative inquiry activities that inform the REM mapping process that follows. In-depth qualitative analysis of the notes and map images captured during REM sessions ensure the rigor required for research-related interventions. Researchers, community members, and participants can use CCF-informed REM collectively as a robust evaluation tool to demonstrate, through visual mapping, the positive effects of community-partnered research programs.

Building Capacity for Evidence-Based Public Health: Reconciling the Pulls of Practice and the Push of Research

Timely implementation of principles of evidence-based public health (EBPH) is critical for bridging the gap between discovery of new knowledge and its application. Public health organizations need sufficient capacity (the availability of resources, structures, and workforce to plan, deliver, and evaluate the preventive dose of an evidence-based intervention) to move science to practice. We review principles of EBPH, the importance of capacity building to advance evidence-based approaches, promising approaches for capacity building, and future areas for research and practice. Although there is general agreement among practitioners and scientists on the importance of EBPH, there is less clarity on the definition of evidence, how to find it, and how, when, and where to use it. Capacity for EBPH is needed among both individuals and organizations. Capacity can be strengthened via training, use of tools, technical assistance, assessment and feedback, peer networking, and incentives. Modest investments in EBPH capacity building will foster more effective public health practice.