Joint international consensus statement on crowdsourcing challenge contests in health and medicine: results of a modified Delphi process

Consensus on Adolescent and Young Adult HIV Research Consent in Low- and Middle-Income Countries

Importance

Many adolescents and young adults (AYAs) in low- and middle-income countries (LMICs) are excluded from HIV research because of challenges with informed consent for study participation, which makes it difficult to understand and improve the lives of AYAs living with HIV and AIDS in a wide variety of settings.

Objective

To help increase the inclusion of AYAs in HIV research, we developed a consensus statement on practical strategies for improving AYA consent in HIV research in LMICs.

Evidence Review

The VOICE (HIV Youth Informed Consent & Ethics in Research) Working Group included AYAs, researchers, community organizers, advocates, research ethics committee members, parents of AYAs, and bioethicists who drafted initial statement items using data from a global open call and scoping review. An adapted Delphi process was then used to develop consensus statement items. The process involved 3 rounds of online Likert-scale questionnaires and a hybrid (online and in-person) consensus summit in Lagos, Nigeria, in 2022, with the total study period lasting from August 23, 2021, to February 10, 2023.

Findings

Thirty-five people participated in the final round of the Delphi process, including 14 individuals younger than 35 years (40.0%), 25 HIV researchers (71.4%), and 32 people who worked in an LMIC (91.4%). Twenty-five items reached a predefined threshold for consensus (≥80% agreement). Strong consensus emerged for formal mechanisms (eg, cocreation, crowdsourcing, or youth advisory boards) for AYA engagement in and education about research as well as for strategies to enhance parental and guardian involvement in HIV research when safe and appropriate. Capacity strengthening can allow AYAs to review research protocols, join ethical review committees, and advocate for regulatory change. Two items in the statement (alternatives to parental consent and raising awareness among research ethics committees about AYA-independent consent) required further refinement to reach the agreement threshold for inclusion.

Conclusions and Relevance

Greater engagement of both AYAs and parents may help to enhance consent processes and increase the inclusion of AYAs in LMIC HIV research studies. The resulting consensus statement provides practical strategies for implementing improved consent processes for AYA research participation at the organizational, community, and policy levels, which may help foster greater inclusion of AYAs in HIV research and address existing data gaps.

PrEP your step: Implementing an online crowdsourcing contest to engage young people in HIV prevention in Washington DC, USA

HIV incidence among young people (Black and Latinx women and men who have sex with men ages 16-24 years), in the United States is high. Traditional top-down approaches for pre-exposure prophylaxis (PrEP) social marketing are not effectively reaching this population. Crowdsourcing is a promising approach to engaging young people in the development of innovative solutions to raise awareness and use of PrEP among those at highest risk of HIV. This study engaged young people in the design and evaluation of an online crowdsourcing contest to promote PrEP among Washington, DC youth. The contest used standard methods recommended by the World Health Organization and feedback from our community partners. Online recruitment using social media elicited online votes and survey responses. We analyzed cross-sectional surveys using descriptive statistics, and semi-structured interviews with contest participants using thematic coding to explore barriers and facilitators to contest engagement. Approximately 82% of entries were from young people in DC. A convenience sample of 181 people voted on their favorite crowdsourced PrEP messages and shared their awareness and attitudes about PrEP. The contest website received 2,500 unique visitors and 4,600 page views. Themes from semi-structured interviews (n = 16) included the need for more community engagement in developing PrEP messaging and positive attitudes towards crowdsourcing. Survey data (n = 887) showed that the crowdsourced messages were well-liked and resonated with the community. Most preferred to see PrEP messages in social media (23%), email (17%) and videos (14%). Approximately 70% of survey participants reported that after viewing the crowdsourced message they would talk to their sexual partner or medical provider (63%) about PrEP, use PrEP (58%), and learn more about PrEP (56%). Crowdsourced messages solicit substantial online viewership. More implementation research is needed to understand the public health impact of integrating social media, crowdsourcing, and community engagement to develop PrEP promotional messages.

How to (or how not to) implement crowdsourcing for the development of health interventions: lessons learned from four African countries

Crowdsourcing strategies are useful in the development of public health interventions. Crowdsourcing engages end users in a co-creation process through challenge contests, designathons or online collaborations. Drawing on our experience of crowdsourcing in four African countries, we provide guidance on designing crowdsourcing strategies across seven steps: deciding on the type of crowdsourcing strategy, convening a steering committee, developing the content of the call for ideas, promotion, evaluation, recognizing finalists and sharing back ideas or implementing the solutions.

A designathon to co-create HPV screening and vaccination approaches for mothers and daughters in Nigeria: findings from a community-led participatory event

BACKGROUND

Oncogenic types of human Papillomavirus (HPV) infection cause substantial morbidity and mortality in Nigeria. Nigeria has low cervical cancer screening and vaccination rates, suggesting the need for community engagement to enhance reach and uptake. We organised a designathon to identify community-led, innovative approaches to promote HPV screening and vaccination for women and girls, respectively, in Nigeria. A designathon is a three-phase participatory process informed by design thinking that includes the preparation phase that includes soliciting innovative ideas from end-users, an intensive collaborative event to co-create intervention components, and follow-up activities.

METHODS

We organised a three-phase designathon for women (30-65yrs) and girls (11-26yrs) in Nigeria. First, we launched a national crowdsourcing open call for ideas on community-driven strategies to support HPV screening among women and vaccination among girls. The open call was promoted widely on social media and at in-person gatherings. All eligible entries were graded by judges and 16 exceptional teams (with 4-6members each). All six geo-political zones of Nigeria were invited to join an in-person event held over three days in Lagos to refine their ideas and present them to a panel of expert judges. The ideas from teams were reviewed and scored based on relevance, feasibility, innovation, potential impact, and mother-daughter team dynamics. We present quantitative data on people who submitted and themes from the textual submissions.

RESULTS

We received a total of 612 submissions to the open call from mother-daughter dyads. Participants submitted ideas via a website designated for the contest (n = 392), in-person (n = 99), email (n = 31), or via an instant messaging application (n = 92). Overall, 470 were eligible for judging after initial screening. The average age of participants for daughters was 19 years and 39 years for mothers. Themes from the top 16 proposals included leveraging local leaders (5/16), faith-based networks (4/16), educational systems (4/16), and other community networks (7/16) to promote awareness of cervical cancer prevention services. After an in-person collaborative event, eight teams were selected to join an innovation training boot camp, for capacity building to implement ideas.

CONCLUSIONS

Innovative strategies are needed to promote HPV screening for mothers and vaccination for girls in Nigeria. Our designathon was able to facilitate Nigerian mother-daughter teams to develop cervical cancer prevention strategies. Implementation research is needed to assess the effectiveness of these strategies.

Catching lightning in a bottle: the STI and HIV 2023 World Congress Participatory Designathon

The International Society for STD Research (ISSTDR) STI/HIV 2023 World Congress convened a participatory designathon to engage attendees in a problem-solving crowdsourcing event with the mission to design innovative solutions for improving sexually transmitted infection (STI) control. Designathons are three-phase crowdsourcing events consisting of a pre-planning phase, an active and intensive collaborative phase, and denouement phase for implementation and dissemination. Given escalating STI concerns, the Congress organisers recognised the opportunity to harness the collective expertise of the attendees by actively engaging them to generate innovative solutions for STI control by hosting a designathon during the scientific meeting. Designathon activities occurred during the Congress, and innovative solutions were presented during the closing plenary. Organisers prioritised participant diversity and engagement by creating four distinct groups based on lived experiences (Silver, Early Career, Traveller, and Community). Although competing priorities through the Congress were a challenge, participation from the attendees was high. Dedicated time and space for the event allowed for a successful designathon event, and the lack of restrictions, as might be encountered from funders or other stakeholder agencies, allowed participants to creatively develop innovative solutions for STI control. This designathon serves as an exemplar for successfully hosting a designathon at a large scientific meeting to engage attendees and contribute their intellect and voice to collectively solving significant problems.

A designathon to co-create HPV screening and vaccination approaches for mothers and daughters in Nigeria: findings from a community-led participatory event

Background: Oncogenic types of human Papillomavirus (HPV) infection cause substantial morbidity and mortality in Nigeria. Nigeria has low cervical cancer screening and vaccination rates, suggesting the need for community engagement to enhance reach and uptake. We organised a designathon to identify community-led, innovative approaches to promote HPV screening and vaccination for women and girls, respectively, in Nigeria. A designathon is a three-phase participatory process informed by design thinking that includes the preparation phase that includes soliciting innovative ideas from end-users, an intensive collaborative event to co-create intervention components, and follow-up activities. Methods: We organised a three-phase designathon for women (30-65yrs) and girls (11-26yrs) in Nigeria. First, we launched a national crowdsourcing open call for ideas on community-driven strategies to support HPV screening among women and vaccination among girls. The open call was promoted widely on social media and at in-person gatherings. All eligible entries were graded by judges and 16 exceptional teams (with 4-6members each). All six geo-political zones of Nigeria were invited to join an in-person event held over three days in Lagos to refine their ideas and present them to a panel of expert judges. The ideas from teams were reviewed and scored based on relevance, feasibility, innovation, potential impact, and mother-daughter team dynamics. We present quantitative data on people who submitted and themes from the textual submissions. Results: We received a total of 612 submissions to the open call from mother-daughter dyads. Participants submitted ideas via a website designated for the contest (n=392), in-person (n=99), email (n=31), or via an instant messaging application (n=92). Overall, 470 were eligible for judging after initial screening. The average age of participants for daughters was 19 years and 39 years for mothers. Themes from the top 16 proposals included leveraging local leaders (5/16), faith-based networks (4/16), educational systems (4/16), and other community networks (7/16) to promote awareness of cervical cancer prevention services. After an in-person collaborative event, eight teams were selected to join an innovation training boot camp, for capacity building to implement ideas. Conclusions: Innovative strategies are needed to promote HPV screening for mothers and vaccination for girls in Nigeria. Our designathon was able to facilitate Nigerian mother-daughter teams to develop cervical cancer prevention strategies. Implementation research is needed to assess the effectiveness of these strategies.

Health research mentorship in low-income and middle-income countries: a global qualitative evidence synthesis of data from a crowdsourcing open call and scoping review

INTRODUCTION

Research mentorship is critical for advancing science, but there are few practical strategies for cultivating mentorship in health research resource-limited settings. WHO/TDR Global commissioned a group to develop a practical guide on research mentorship. This global qualitative evidence synthesis included data from a crowdsourcing open call and scoping review to identify and propose strategies to enhance research mentorship in low/middle-income country (LMIC) institutions.

METHODS

The crowdsourcing open call used methods recommended by WHO/TDR and solicited descriptions of strategies to enhance research mentorship in LMICs. The scoping review used the Cochrane Handbook and predefined the approach in a protocol. We extracted studies focused on enhancing health research mentorship in LMICs. Textual data describing research mentorship strategies from the open call and studies from the scoping review were coded into themes. The quality of evidence supporting themes was assessed using the Confidence in the Evidence from Reviews of Qualitative research approach.

RESULTS

The open call solicited 46 practical strategies and the scoping review identified 77 studies. We identified the following strategies to enhance research mentorship: recognising mentorship as an institutional responsibility that should be provided and expected from all team members (8 strategies, 15 studies; moderate confidence); leveraging existing research and training resources to enhance research mentorship (15 strategies, 49 studies; moderate confidence); digital tools to match mentors and mentees and sustain mentorship relations over time (14 strategies, 11 studies; low confidence); nurturing a culture of generosity so that people who receive mentorship then become mentors to others (7 strategies, 7 studies; low confidence); peer mentorship defined as informal and formal support from one researcher to another who is at a similar career stage (16 strategies, 12 studies; low confidence).

INTERPRETATION

Research mentorship is a collective institutional responsibility, and it can be strengthened in resource-limited institutions by leveraging already existing resources. The evidence from the crowdsourcing open call and scoping review informed a WHO/TDR practical guide. There is a need for more formal research mentorship programmes in LMIC institutions.

Enhancing Community Participation in Dengue Control Through Digital Crowdsourcing: An Analysis of a World Mosquito Program Digital Open Call in Sri Lanka

BACKGROUND

Two crowdsourcing open calls were created to enhance community engagement in dengue control in Sri Lanka. We analyzed the process and outcomes of these digital crowdsourcing open calls.

METHODS

We used standard World Health Organization methods to organize the open calls, which used exclusively digital methods because of coronavirus disease 2019 (COVID-19). We collected and analyzed sociodemographic information and digital engagement metrics from each submission. Submissions in the form of textual data describing community-led strategies for mosquito release were coded using grounded theory.

RESULTS

The open calls received 73 submissions. Most people who submitted ideas spoke English, lived in Sri Lanka, and were 18 to 34 years old. The total Facebook reach was initially limited (16 161 impressions), prompting expansion to a global campaign, which reached 346 810 impressions over 14 days. Diverse strategies for the distribution of Wolbachia-infected mosquito boxes were identified, including leveraging traditional festivals, schools, and community networks. Fifteen submissions (21%) suggested the use of digital tools for monitoring and evaluation, sharing instructions, or creating networks. Thirteen submissions (18%) focused on social and economic incentives to prompt community engagement and catalyze community-led distribution.

CONCLUSIONS

Our project demonstrates that digital crowdsourcing open calls are an effective way to solicit creative and innovative ideas in a resource-limited setting.

Crowdsourcing to support training for public health: A scoping review

Crowdsourcing is an interactive process that has a group of individuals attempt to solve all or part of a problem and then share solutions with the public. Crowdsourcing is increasingly used to enhance training through developing learning materials and promoting mentorship. This scoping review aims to assess the literature on crowdsourcing for training in public health. We searched five medical and public health research databases using terms related to crowdsourcing and training. For this review, the concept of crowdsourcing included open calls, designathons, and other activities. We used a PRISMA checklist for scoping reviews. Each full-text was assessed by two independent reviewers. We identified 4,071 citations, and 74 studies were included in the scoping review. This included one study in a low-income country, 15 studies in middle-income countries, 35 studies in high-income countries, and 11 studies conducted in multiple countries of varying income levels (the country income level for 12 studies could not be ascertained). Nine studies used open calls, 35 used a hackathon, designathon or other "a-thon" event, and 30 used other crowdsourcing methods, such as citizen science programs and online creation platforms. In terms of crowdsourcing purpose, studies used crowdsourcing to educate participants (20 studies), develop learning materials (17 studies), enhance mentorship (13 studies) and identify trainees (9 studies). Fifteen studies used crowdsourcing for more than one training purpose. Thirty-four studies were done in-person, 31 were conducted virtually and nine used both meeting options for their crowdsourcing events. Seventeen studies generated open access materials. Our review found that crowdsourcing has been increasingly used to support public health training. This participatory approach can be a useful tool for training in a variety of settings and populations. Future research should investigate the impact of crowdsourcing on training outcomes.

Pay-it-forward gonorrhea and chlamydia testing among men who have sex with men and male STD patients in China: the PIONEER pragmatic, cluster randomized controlled trial protocol

BACKGROUND

Gonorrhea and chlamydia are the most common sexually transmitted diseases (STDs) among men who have sex with men (MSM) in China. Previous studies have shown pay-it-forward (PIF) interventions to be associated with a substantial increase in gonorrhea and chlamydia test uptake compared to standard-of-care. We propose a 'pay-it-forward' gonorrhea and chlamydia testing randomized controlled trial (PIONEER). The trial would evaluate the effectiveness of two pay-it-forward strategies in promoting testing uptake compared to the standard of care (in which men pay for their tests out-of-pocket) among MSM and male STD patients in China.

METHODS

PIONEER will be a three-armed, pragmatic cluster randomized controlled trial (RCT), conducted across 12 clinics (six MSM-led and six public STD clinics) to compare the effectiveness of three implementation strategies. Each facility will be randomized to a standard pay-it-forward intervention of gonorrhea/ chlamydia testing with minimal encouragement for testing, a community-engaged pay-it-forward arm, or a control arm where men pay for their tests out-of-pockets. The primary outcome will be dual gonorrhea/chlamydia test uptake. Secondary outcomes will include syphilis testing, amount donated in pay-it-forward, number of positive gonorrhea and chlamydia tests, and measures of antimicrobial resistance. A sequential transformative mixed methods design will be used to evaluate the implementation process in type 2 effectiveness-implementation hybrid design. Data sources will include survey on acceptability, and feelings and attitudes towards the interventions among participants; testing and treatment uptake data from clinic records, WeChat records, and qualitative data to gain insights into men's perceptions and attitudes towards the pay-it-forward, mechanisms driving uptake, and donating behaviors. Implementers and organizers will be interviewed about fidelity and adherence to protocol, sustainability of pay-it-forward intervention, and barriers and facilitators of implementing the intervention.

DISCUSSION

PIONEER will substantially increase gonorrhea/chlamydia testing among MSM in China, providing an innovative and new financial mechanism to sustain STD screening among sexual minorities in low- and middle-income countries. This study will answer compelling scientific questions about how best to implement pay-it-forward and the individual and organizational characteristics that moderate it.

TRIAL REGISTRATION

The study with identification number NCT05723263 has been registered on clinicaltrials.gov/.

Economic evaluation of improving HIV self-testing among MSM in China using a crowdsourced intervention: a cost-effectiveness analysis

OBJECTIVES

Crowdsourcing, which taps into the wisdom of crowds, has been successful in generating strategies to enhance HIV self-testing (HIVST) uptake. We determined the cost-effectiveness of a crowdsourced intervention (one-off or annual) compared with a control scenario (no crowdsourcing) among MSM living in China.

DESIGN

Economic evaluation.

METHODS

We used data from our cluster randomized controlled trial of MSM (NCT02796963). We used a micro-costing approach to measure direct health costs ($USD2017) from a health provider perspective. Using outputs from a dynamic transmission model over a 20-year time horizon, we estimated the incremental cost-effectiveness ratios using cost per disability-adjusted life years (DALYs) averted with 3% discounting. An intervention was considered highly cost-effective if it was less than one gross domestic product (GDP, $8823) per DALY averted.

RESULTS

Across all cities, the crowdsourced intervention was highly cost-effective compared with the control scenario (incremental cost-effectiveness ratios ranged from $2263 to 6152 per DALY averted for annual crowdsourcing; $171 to 204 per DALY averted for one-off crowdsourcing). The one-off intervention was cost-saving in Guangzhou and Qingdao. Sensitivity analyses confirmed the robustness of the findings; specifically, changes in discounting, costs of the crowdsourced intervention, costs of HIV testing and cost of antiretroviral therapy did not alter our conclusions.

CONCLUSION

Scaling up a one-off or annual crowdsourced HIV prevention intervention in four cities in China was very likely to be cost-effective. Further research is warranted to evaluate the feasibility of scaling up crowdsourced HIV prevention interventions in other settings and populations.