Theories of change for e-health interventions targeting HIV/STIs and sexual risk, substance use and mental ill health amongst men who have sex with men: systematic review and synthesis

Analyzing the usage of theories of change for routine immunization programs -- a review of impact evaluations from LMICs

BACKGROUND

In this article we analyzed the extent of the usage of Theories of Change (TOCs) and causal pathways in the evaluation of immunization programs to identify the challenges to generating evidence on how interventions improve immunization.

METHODS

We analyzed the use of the TOC in impact evaluations (IEs) of immunization interventions published after 2010, and its associated articles. The review includes studies from Evidence Gap Map and Yale review that were conducted in May and March of 2020, respectively. We synthesized data on six domains using NVIVO - program theory, context, assumptions, usage of TOC, use in evaluation, and description causal pathways.

RESULTS

Our review included 47 large-scale and 45 small-to medium-scale interventions. Of the included studies, 19% used a TOC, 56% described a causal pathway or used a conceptual diagram with varying degrees of detail, and 25% of the IEs did not provide any information on how their intervention was expected to affect change. Only 19 of the 92 IEs explicitly outlined any assumptions associated with the implementation of the interventions. Forty studies measured the outputs or intermediate outcomes leading to improved immunization coverage.

CONCLUSION

Future implementers and evaluators need to develop clear TOCs that are based on established theory and have clearly articulated the underlying assumptions. Large-scale health system strengthening initiatives implemented by governments, also need to build TOCs and integrate them into their results frameworks. Additionally, there is a need to combine both impact and process evaluations to understand the how context affects the causal pathways.

Whole-school interventions promoting student commitment to school to prevent substance use and violence, and improve educational attainment: a systematic review

Background

Whole-school interventions modify the school environment to promote health. A subset of these interventions promotes student commitment to school to prevent substance (tobacco, alcohol, other drugs) use and/or violence. A previous review identified the theory of human functioning and school organisation as a comprehensive theory of such interventions, and found evidence that these interventions reduce substance use and/or violence.

Objectives

The objectives were to search for, appraise and synthesise evidence to address the following questions: (1) What whole-school interventions promoting student commitment to school to prevent substance use and/or violence have been evaluated, what intervention subtypes are apparent and how closely do these align with the theory of human functioning and school organisation? (2) What factors relating to setting, population and intervention affect implementation? (3) What are the effects on student substance use, violence and educational attainment? (4) What is the cost-effectiveness of such interventions? (5) Are intervention effects mediated by student commitment to school or moderated by setting or population?

Data sources

A total of 56 information sources were searched (in January 2020), then an updated search of 48 of these was carried out (in May 2021). Reference lists were also searched and experts were contacted.

Review methods

Eligible studies were process/outcome evaluations of whole-school interventions to reduce student violence or substance use among students aged 5-18 years attending schools, via actions aligning with the theory of human functioning and school organisation: modifying teaching to increase engagement, enhancing student-staff relationships, revising school policies, encouraging volunteering or increasing parental involvement. Data extraction and quality assessments used existing tools. Theory and process reports were synthesised qualitatively. Outcome and economic data were synthesised narratively; outcome data were meta-analysed.

Results

Searches retrieved 63 eligible reports on 27 studies of 22 interventions. We identified four intervention subtypes focused on student participation in school-wide decisions, improving staff-student relationships, increasing engagement in learning and involving parents. The theories of change of most intervention subtypes aligned closely with the theory of human functioning and school organisation, and informed refinement of an intervention theory of change. Theories of change for interventions increasing learning engagement did not align with this theory, aiming instead to increase school commitment primarily via social skills curricula. Factors influencing the implementation included whether or not interventions were tailorable, workable and well explained. Interventions with action groups comprising staff/students, etc. and providing local data were well implemented. Implementation was also affected by whether or not schools accepted the need for change and staff had the resources for delivery. Meta-analyses suggest small, but significant, intervention effects in preventing violence victimisation and perpetration, and substance use. There was sparse and inconsistent evidence of moderation and some evidence of mediation by student commitment to school. Two economic evaluations suggested that there is the potential for the interventions to be cost-effective.

Limitations

The quality of the studies was variable and the economic synthesis was limited to two studies.

Conclusions

Whole-school interventions aiming to promote student commitment to school share similar theories of change and factors affecting implementation. They have the potential to contribute to preventing violence and substance use among young people. Future trials should aim to optimise intervention effectiveness by better theorisation, and assess implementation and effect moderators and mediators.

Study registration

This study is registered as PROSPERO CRD42019154334.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Public Health Research programme (NIHR award ref: 17/151/05) and is published in full in Public Health Research; Vol. 12, No. 2. See the NIHR Funding and Awards website for further award information.

Integration of evidence into Theory of Change frameworks in the healthcare sector: A rapid systematic review

BACKGROUND

Theory of Change (ToC) has become an established approach to design and evaluate interventions. While ToC should-in line with the growing international focus on evidence-informed health decision-making-consider explicit approaches to incorporate evidence, there is limited guidance on how this should be done. This rapid review aims to identify and synthesize the available literature on how to systematically use research evidence when developing or adapting ToCs in the health sector.

METHODS

A rapid review methodology using a systematic approach, was designed. Eight electronic databases were consulted to search for peer-reviewed and gray publications detailing tools, methods, and recommendations promoting the systematic integration of research evidence in ToCs. The included studies were compared, and the findings summarized qualitatively into themes to identify key principles, stages, and procedures, guiding the systematic integration of research evidence when developing or revising a ToC.

RESULTS

This review included 18 studies. The main sources from which evidence was retrieved in the ToC development process were institutional data, literature searches, and stakeholder consultation. There was a variety of ways of finding and using evidence in ToC. Firstly, the review provided an overview of existing definitions of ToC, methods applied in ToC development and the related ToC stages. Secondly, a typology of 7 stages relevant for evidence integration into ToCs was developed, outlining the types of evidence and research methods the included studies applied for each of the proposed stages.

CONCLUSION

This rapid review adds to the existing literature in two ways. First, it provides an up-to-date and comprehensive review of the existing methods for incorporating evidence into ToC development in the health sector. Second, it offers a new typology guiding any future endeavors of incorporating evidence into ToCs.

Associations of Momentary Mindfulness With Affect and Cannabis Desire in a Trial of Cannabis Use Interventions With and Without Momentary Assessment

PURPOSE

Mindfulness, awareness resulting from attending to the present without judgment, has been associated with improved health. When considered as a time-varying momentary state, mindfulness is associated with other momentary states such as affect. We examined whether momentary mindfulness, specifically mindful attention and awareness (MAA), changed after counseling interventions to reduce cannabis use that included ecological momentary assessment (EMA) and explored associations with negative affect, positive affect, and cannabis desire.

METHODS

Outpatients 15-24 years using cannabis ≥3x/week were randomized to one of the three interventions, each including two motivational enhancement therapy (MET) sessions. For two interventions, MET was followed by 2 weeks of EMA (with/without messaging). Momentary MAA, negative affect, positive affect, and cannabis desire were assessed over 1 week of EMA at baseline and 3-month follow-up (N = 1,971 reports, 68 participants). We examined changes in momentary MAA from baseline to follow-up overall and by group (MET + EMA, MET-only) using generalized linear mixed effects models. We tested associations of momentary MAA with momentary affect and cannabis desire using generalized estimating equations.

RESULTS

Momentary MAA increased from baseline to follow-up after MET counseling plus EMA (β = 0.237), but did not change with MET counseling alone. Higher momentary MAA was associated with lower negative affect (β = -0.526) and cannabis desire (β = -0.521), but not with positive affect.

DISCUSSION

Among youth using cannabis frequently, momentary MAA was increased 3 months after interventions with EMA after counseling and was inversely associated with momentary negative affect and momentary cannabis desire. Mindfulness may be a useful target for momentary intervention.

Web-Based Service Provision of HIV, Viral Hepatitis, and Sexually Transmitted Infection Prevention, Testing, Linkage, and Treatment for Key Populations: Systematic Review and Meta-analysis

BACKGROUND

Despite the growth of web-based interventions for HIV, viral hepatitis (VH), and sexually transmitted infections (STIs) for key populations, the evidence for the effectiveness of these interventions has not been reported.

OBJECTIVE

This study aimed to inform the World Health Organization guidelines for HIV, VH, and STI prevention, diagnosis, and treatment services for key populations by systematically reviewing the effectiveness, values and preferences, and costs of web-based outreach, web-based case management, and targeted web-based health information for key populations (men who have sex with men, sex workers, people who inject drugs, trans and gender-diverse people, and people in prisons and other closed settings).

METHODS

We searched CINAHL, PsycINFO, PubMed, and Embase in May 2021 for peer-reviewed studies; screened abstracts; and extracted data in duplicate. The effectiveness review included randomized controlled trials (RCTs) and observational studies. We assessed the risk of bias using the Cochrane Collaboration tool for RCTs and the Evidence Project and Risk of Bias in Non-randomized Studies of Interventions tools for non-RCTs. Values and preferences and cost data were summarized descriptively.

RESULTS

Of 2711 records identified, we included 13 (0.48%) articles in the effectiveness review (3/13, 23% for web-based outreach; 7/13, 54% for web-based case management; and 3/13, 23% for targeted web-based health information), 15 (0.55%) articles in the values and preferences review, and 1 (0.04%) article in the costs review. Nearly all studies were conducted among men who have sex with men in the United States. These articles provided evidence that web-based approaches are as effective as face-to-face services in terms of reaching new people, use of HIV, VH, and STI prevention services, and linkage to and retention in HIV care. A meta-analysis of 2 RCTs among men who have sex with men in China found increased HIV testing after web-based outreach (relative risk 1.39, 95% CI 1.21-1.60). Among men who have sex with men in the United States, such interventions were considered feasible and acceptable. One cost study among Canadian men who have sex with men found that syphilis testing campaign advertisements had the lowest cost-per-click ratio on hookup platforms compared with more traditional social media platforms.

CONCLUSIONS

Web-based services for HIV, VH, and STIs may be a feasible and acceptable approach to expanding services to key populations with similar outcomes as standard of care, but more research is needed in low-resource settings, among key populations other than men who have sex with men, and for infections other than HIV (ie, VH and STIs).

What factors influence implementation of whole-school interventions aiming to promote student commitment to school to prevent substance use and violence? Systematic review and synthesis of process evaluations

BACKGROUND

Whole-school interventions go beyond classroom health education to modify the school environment to promote health. A sub-set aim to promote student commitment to school to reduce substance use and violence (outcomes associated with low commitment). It is unclear what factors influence implementation of such interventions.

METHODS

We conducted a systematic review including synthesis of evidence from process evaluations examining what factors affect implementation. Meta-ethnographic synthesis was informed by May's General Theory of Implementation.

RESULTS

Sixteen reports, covering 13 studies and 10 interventions were included in our synthesis. In terms of May's concept of 'sense-making', we found that school staff were more likely to understand what was required in implementing an intervention when provided with good-quality materials and support. Staff could sometimes wilfully or unintentionally misinterpret interventions. In terms of May's concept of 'cognitive participation', whereby staff commit to implementation, we found that lack of intervention adaptability could in particular undermine implementation of whole-school elements. Interventions providing local data were reported as helping build staff commitment. School leaders were more likely to commit to an intervention addressing an issue they already intended to tackle. Collaborative planning groups were reported as useful in ensuring staff 'collective action' (May's term for working together) to enact interventions. Collective action was also promoted by the presence of sufficient time, leadership and relationships. Implementation of whole-school interventions took time to build. Considering May's concept of 'reflexive monitoring' (formal or informal review of progress), this was important in assessing and enhancing implementation. 'Quick wins' could help maintain collective impetus to implement further intervention activities.

CONCLUSION

We identified novel factors influencing implementation of whole-school elements such as: local adaptability of interventions; providing local data to build commitment; interventions addressing an issue already on school leaders' agenda; collaborative planning groups; and 'reflexive monitoring' as an explicit intervention component.

eHealth Interventions to Address HIV and Other Sexually Transmitted Infections, Sexual Risk Behavior, Substance Use, and Mental Ill-health in Men Who Have Sex With Men: Systematic Review and Meta-analysis

Background

Men who have sex with men experience disproportionately high levels of HIV and other sexually transmitted infections (STIs), sexual risk behavior, substance use, and mental ill-health. These experiences are interrelated, and these interrelations are potentiated by structural conditions of discrimination, stigma, and unequal access to appropriate health services, and they magnify each other and have intersecting causal pathways, worsening both risk for each condition and risk for the negative sequelae of each condition. eHealth interventions could address these issues simultaneously and thus have wide-ranging and greater effects than would be for any 1 outcome alone.

Objective

We systematically reviewed the evidence for the effectiveness of eHealth interventions in addressing these outcomes separately or together.

Methods

We searched 19 databases for randomized trials of interactive or noninteractive eHealth interventions delivered via mobile phone apps, internet, or other electronic media to populations consisting entirely or principally of men who have sex with men to prevent HIV, STIs, sexual risk behavior, alcohol and drug use, or common mental illnesses. We extracted data and appraised each study, estimated meta-analyses where possible by using random effects and robust variance estimation, and assessed the certainty of our findings (closeness of the estimated effect to the true effect) by using GRADE (Grading of Recommendations, Assessment, Development and Evaluations).

Results

We included 14 trials, of which 13 included active versus control comparisons; none reported mental health outcomes, and all drew from 12 months or less of follow-up postintervention. Findings for STIs drew on low numbers of studies and did not suggest consistent short-term (<3 months postintervention; d=0.17, 95% CI –0.18 to 0.52; I²=0%; 2 studies) or midterm (3-12 months postintervention, no meta-analysis, 1 study) evidence of effectiveness. Eight studies considering sexual risk behavior outcomes suggested a short-term, nonsignificant reduction (d=–0.14, 95% CI –0.30 to 0.03) with very low certainty, but 6 studies reporting midterm follow-ups suggested a significant impact on reducing sexual risk behavior (d=–0.12, 95% CI –0.19 to –0.05) with low certainty. Meta-analyses could not be undertaken for alcohol and drug use (2 heterogeneous studies) or for HIV infections (1 study for each of short-term or midterm follow-up), and alcohol outcomes alone were not captured in the included studies. Certainty was graded as low to very low for most outcomes, including all meta-analyses.

Conclusions

To create a comprehensive eHealth intervention that targets multiple outcomes, intervention evaluations should seek to generalize both mechanisms and components that are successfully used to achieve change in 1 outcome over multiple outcomes. However, additional evaluations of interventions seeking to address outcomes other than sexual risk behavior are needed before development and evaluation of a joined-up intervention.

E-health interventions targeting STIs, sexual risk, substance use and mental health among men who have sex with men: four systematic reviews

Background

Human immunodeficiency virus/sexually transmitted infections, sexual risk, substance (alcohol and other legal and illegal drugs) use and mental ill health constitute a ‘syndemic’ of mutually reinforcing epidemics among men who have sex with men. Electronic health (e-health) interventions addressing these epidemics among men who have sex with men might have multiplicative effects. To our knowledge, no systematic review has examined the effectiveness of such interventions on these epidemics among men who have sex with men.

Objective

The objective was to synthesise evidence addressing the following: (1) What approaches and theories of change do existing e-health interventions employ to prevent human immunodeficiency virus/sexually transmitted infections, sexual risk, alcohol/drug use or mental ill health among men who have sex with men? (2) What factors influence implementation? (3) What are the effects of such interventions on the aforementioned epidemics? (4) Are such interventions cost-effective?

Data sources

A total of 24 information sources were searched initially (October–November 2018) [the following sources were searched: ProQuest Applied Social Sciences Index and Abstracts; Campbell Library; EBSCO Cumulative Index to Nursing and Allied Health Literature Plus, Wiley Online Library The Cochrane Library; Centre for Reviews and Dissemination databases (the Database of Abstracts of Reviews of Effects and the NHS Economic Evaluation Database); the Health Technology Assessment database; Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) database of health promotion research (Bibliomap); ProQuest Dissertations & Theses Global; OvidSP EconLit; OvidSP EMBASE; OvidSP Global Health; OvidSP Health Management Information Consortium; ProQuest International Bibliography of the Social Sciences; Ovid MEDLINE ALL; OvidSP PsycINFO; Web of Science Science Citation Index Expanded; Elsevier Scopus; OvidSP Social Policy & Practice; Web of Science Social Sciences Citation Index Expanded; ProQuest Sociological Abstracts; ClinicalTrials.gov; World Health Organization International Clinical Trials Registry Platform; EPPI-Centre Trials Register of Promoting Health Interventions; and the OpenGrey database], and an updated search of 19 of these was conducted in April 2020. Reference lists of included reports were searched and experts were contacted.

Review methods

Eligible reports presented theories of change and/or process, outcome and/or economic evaluations of e-health interventions offering ongoing support to men who have sex with men to prevent human immunodeficiency virus/sexually transmitted infections, sexual risk behaviour, alcohol/drug use and/or common mental illnesses. References were screened by title/abstract, then by full text. Data extraction and quality assessments used existing tools. Theory and process reports were synthesised using qualitative methods. Outcome and economic data were synthesised narratively; outcome data were meta-analysed.

Results

Original searches retrieved 27 eligible reports. Updated searches retrieved 10 eligible reports. Thirty-seven reports on 28 studies of 23 interventions were included: 33 on theories of change, 12 on process evaluations, 16 on outcome evaluations and one on an economic evaluation. Research question 1: five intervention types were identified – ‘online modular’, ‘computer games’ and ‘non-interactive’ time-limited/modular interventions, and open-ended interventions with ‘content organised by assessment’ and ‘general content’. Three broad types of intervention theories of change were identified, focusing on ‘cognitive/skills’, ‘self-monitoring’ and ‘cognitive therapy’. Research question 2: individual tailoring based on participant characteristics was particularly acceptable, and participants valued intervention content reflecting their experiences. Research question 3: little evidence was available of effects on human immunodeficiency virus or sexually transmitted infections. The analysis did not suggest that interventions were effective in reducing instances of human immunodeficiency virus or sexually transmitted infections. The overall meta-analysis for sexually transmitted infections reported a small non-significant increase in sexually transmitted infections in the intervention group, compared with the control group. Meta-analyses found a significant impact on sexual risk behaviour. The findings for drug use could not be meta-analysed because of study heterogeneity. Studies addressing this outcome did not present consistent evidence of effectiveness. Trials did not report effects on alcohol use or mental health. Research question 4: evidence on cost-effectiveness was limited.

Limitations

The quality of the eligible reports was variable and the economic synthesis was limited to one eligible study.

Conclusions

There is commonality in intervention theories of change and factors affecting receipt of e-health interventions. Evidence on effectiveness is limited.

Future work

Future trials should assess the impact of interventions on multiple syndemic factors, among them sexual risk, substance use and mental health; incorporate sufficient follow-up and sample sizes to detect the impact on human immunodeficiency virus/sexually transmitted infections; and incorporate rigorous process and economic evaluations.

Study registration

This study is registered as PROSPERO CRD42018110317.

Funding

This project was funded by the National Institute for Health Research (NIHR) Public Health Research programme and will be published in full in Public Health Research; Vol. 10, No. 4. See the NIHR Journals Library website for further project information.

eHealth Interventions to Address Sexual Health, Substance Use, and Mental Health Among Men Who Have Sex With Men: Systematic Review and Synthesis of Process Evaluations

Background

Men who have sex with men (MSM) face disproportionate risks concerning HIV and other sexually transmitted infections, substance use, and mental health. These outcomes constitute an interacting syndemic among MSM; interventions addressing all 3 together could have multiplicative effects. eHealth interventions can be accessed privately, and evidence from general populations suggests these can effectively address all 3 health outcomes. However, it is unclear how useable, accessible, or acceptable eHealth interventions are for MSM and what factors affect this.

Objective

We undertook a systematic review of eHealth interventions addressing sexual risk, substance use, and common mental illnesses among MSM and synthesized evidence from process evaluations.

Methods

We searched 19 databases, 3 trials registers, OpenGrey, and Google, and supplemented this by reference checks and requests to experts. Eligible reports were those that discussed eHealth interventions offering ongoing support to MSM aiming to prevent sexual risk, substance use, anxiety or depression; and assessed how intervention delivery or receipt varied with characteristics of interventions, providers, participants, or context. Reviewers screened citations on titles, abstracts, and then full text. Reviewers assessed quality of eligible studies, and extracted data on intervention, study characteristics, and process evaluation findings. The analysis used thematic synthesis.

Results

A total of 12 reports, addressing 10 studies of 8 interventions, were eligible for process synthesis. Most addressed sexual risk alone or with other outcomes. Studies were assessed as medium and high reliability (reflecting the trustworthiness of overall findings) but tended to lack depth and breadth in terms of the process issues explored. Intervention acceptability was enhanced by ease of use; privacy protection; use of diverse media; opportunities for self-reflection and to gain knowledge and skills; and content that was clear, interactive, tailored, reflective of MSM’s experiences, and affirming of sexual-minority identity. Technical issues and interventions that were too long detracted from acceptability. Some evidence suggested that acceptability varied by race or ethnicity and educational level; findings on variation by socioeconomic status were mixed. No studies explored how intervention delivery or receipt varied by provider characteristics.

Conclusions

Findings suggest that eHealth interventions targeting sexual risk, substance use, and mental health are acceptable for MSM across sociodemographic groups. We identified the factors shaping MSM’s receipt of such interventions, highlighting the importance of tailored content reflecting MSM’s experiences and of language affirming sexual-minority identities. Intervention developers can draw on these findings to increase the usability and acceptability of integrated eHealth interventions to address the syndemic of sexual risk, substance use, and mental ill health among MSM. Evaluators of these interventions can draw on our findings to plan evaluations that explore the factors shaping usability and acceptability.