A survey of Delphi panelists after core outcome set development revealed positive feedback and methods to facilitate panel member participation

Delphi studies in social and health sciences-Recommendations for an interdisciplinary standardized reporting (DELPHISTAR). Results of a Delphi study

BACKGROUND

While different proposals exist for a guideline on reporting Delphi studies, none of them has yet established itself in the health and social sciences and across the range of Delphi variants. This seems critical because empirical studies demonstrate a diversity of modifications in the conduction of Delphi studies and sometimes even errors in the reporting. The aim of the present study is to close this gap and formulate a general reporting guideline.

METHOD

In an international Delphi procedure, Delphi experts were surveyed online in three rounds to find consensus on a reporting guideline for Delphi studies in the health and social sciences. The respondents were selected via publications of Delphi studies. The preliminary reporting guideline, containing 65 items on five topics and presented for evaluation, had been developed based on a systematic review of the practice of Delphi studies and a systematic review of existing reporting guidelines for Delphi studies. Starting in the second Delphi round, the experts received feedback in the form of mean values, measures of dispersion, a summary of the open-ended responses and their own response in the previous round. The final draft of the reporting guideline contains the items on which at least 75% of the respondents agreed by assigning scale points 6 and 7 on a 7-point Likert scale.

RESULTS

1,072 experts were invited to participate. A total of 91 experts completed the first Delphi round, 69 experts the second round, and 56 experts the third round. Of the 65 items in the first draft of the reporting guideline, consensus was ultimately reached for 38 items addressing the five topics: Title and Abstract (n = 3), Context (n = 7), Method (n = 20), Results (n = 4) and Discussion (n = 4). Items focusing on theoretical research and on dissemination were either rejected or remained subjects of dissent.

DISCUSSION

We assume a high level of acceptance and interdisciplinary suitability regarding the reporting guideline presented here and referred to as the "Delphi studies in social and health sciences-recommendations for an interdisciplinary standardized reporting" (DELPHISTAR). Use of this reporting guideline can substantially improve the ability to compare and evaluate Delphi studies.

Core socioDemographic data variables in ICU Trials (CoDe-IT): a protocol for generating core data variables using a Delphi consensus process

INTRODUCTION

Sociodemographic variables influence health outcomes, either directly (ie, gender identity) or indirectly (eg, structural/systemic racism based on ethnoracial group). Identification of how sociodemographic variables can impact the health of critically ill adults is important to guide care and research design for this population. However, despite the growing recognition of the importance of collecting sociodemographic measures that influence health outcomes, insufficient and inconsistent data collection of sociodemographic variables persists in critical care studies. We aim to develop a set of core data variables (CoDaV) for social determinants of health specific to studies involving critically ill adults.

METHODS AND ANALYSIS

We will conduct a scoping review to generate a list of possible sociodemographic measures to be used for round 1 of the modified Delphi processes. We will engage relevant knowledge users (previous intensive care unit patients and family members, critical care researchers, critical care clinicians and research co-ordinators) to participate in the modified Delphi consensus survey to identify the CoDaV. A final consensus meeting will be held with knowledge user representatives to discuss the final CoDaV, how each sociodemographic variable will be collected (eg, level of granularity) and how to disseminate the CoDaV for use in critical care studies.

ETHICS AND DISSEMINATION

The University of Calgary conjoint health research ethics board has approved this study protocol (REB22-1648).

Achieving consensus on the key elements of a nurse-led multidisciplinary chronic heart failure service in Thailand: An e-Delphi study

AIMS

To describe the outcomes of an e(electronic)-Delphi survey used to achieve consensus on the essential elements that should be included in a multidisciplinary, nurse-led service for patients with chronic heart failure (CHF).

DESIGN

The study design was based on a three-round e-Delphi survey.

METHODS

A series of three survey rounds were used to gather expert opinions and achieve consensus on the key elements that should be included in a CHF disease management program from a sample of healthcare professionals and organizational leaders with expertise in existing CHF services. Consensus for each item was defined as at least 90% agreement.

RESULTS

There were 20 participants (18 healthcare professionals and 2 organizational leaders) in round-1, 22 (20 healthcare professionals and 2 organizational leaders) in round-2, and 17 (15 healthcare professionals and 2 organizational leaders) in round-3. Fifteen participants attended a final online meeting (13 healthcare professionals and 2 organizational leaders). Consensus was obtained on five essential components: (i) consumer education, (ii) treatment optimization using a multidisciplinary approach, (iii) discharge planning, (iv) provision of supportive community care and (v) mechanisms to respond to early symptoms of decompensated CHF.

CONCLUSION

Participants focused on mechanisms to improve treatment effectiveness, patient and family knowledge, communication between healthcare providers and supportive care in the community. The proposed model of care may be useful to other health service providers who are designing or adapting CHF models of care for the South-East Asian environment.

IMPACT

This research provides a solid basis for using a Delphi method to resolve the challenges and issues of transferring global model-based recommendations in CHF knowledge. The Delphi method proved successful as an important step in developing a culturally acceptable model of chronic care that meets the goals of local healthcare providers.

What to Measure in Aneurysmal Subarachnoid Haemorrhage Research-An International Delphi Survey

Aneurysmal subarachnoid haemorrhage (aSAH) is a devastating condition with high mortality and morbidity. The outcome measures used in aSAH clinical research vary making it challenging to compare and combine different studies. Additionally, there may be a mismatch between the outcomes prioritized by patients, caregivers, and health care providers and those selected by researchers. We conducted an international, online, multiple round Delphi study to develop consensus on domains (where a domain is a health concept or aspect) prioritized by key stakeholders including those with lived experience of aSAH, health care providers, and researchers, funders, or industry professionals. One hundred seventy-five people participated in the survey, 59% of whom had lived experience of aSAH. Over three rounds, 32 domains reached the consensus threshold pre-defined as 70% of participants rating the domain as being critically important. During the fourth round, participants ranked the importance of each of these 32 domains. The top ten domains ranked highest to lowest were (1) Cognition and executive function, (2) Aneurysm obliteration, (3) Cerebral infarction, (4) Functional outcomes including ability to walk, (5) Delayed cerebral ischemia, (6) The overall quality of life as reported by the SAH survivor, (7) Changes to emotions or mood (including depression), (8) The basic activities of daily living, (9) Vasospasm, and (10) ICU complications. Our findings confirm that there is a mismatch between domains prioritized by stakeholders and outcomes used in clinical research. Our future work aims to address this mismatch through the development of a core outcome set in aSAH research.

What mobility factors are critical to include in a comprehensive mobility discharge assessment framework for older adults transitioning from hospital-to-home in the community? An international e-Delphi study

PURPOSE

To prioritize and achieve consensus on mobility determinant factors [cognitive, financial, environmental, personal, physical, psychological, social] considered critical to include in the COmprehensive Mobility Discharge Assessment Framework (COMDAF) for older adults transitioning from hospital-to-home.

MATERIALS AND METHODS

We conducted a three-round modified e-Delphi process with 60 international experts (seven older adults, nine family caregivers, 24 clinicians, and 20 researchers) from nine countries with universal or near-universal health coverage. Expert members rated 91 factors identified from scoping reviews using a 9-point scale: not important (1-3), important (4-6), and critical (7-9).

RESULT

A total of 41 of 91 factors (45.1%) met the a-priori consensus criterion after three rounds: five cognitive, five environmental, two personal, 19 physical, six psychological, and six social factors. No financial factors reached a consensus. The older adult steering committee member recommended the addition of two environmental factors, resulting in 43 mobility factors included in the COMDAF.

CONCLUSIONS

We advanced a comprehensive mobility framework by developing, through consensus, 43 mobility factors to be assessed as part of a COMDAF. However, its use in hospital-to-home may not be feasible. Therefore, future research will determine the core mobility factors for COMDAF and which measurement instruments best measure these factors.

RELEVANCE

An interdisciplinary discharge rehabilitation team can utilize the COMDAF during hospital-to-home transition.Implications for rehabilitationMobility assessment following a hospital discharge is a complex process requiring an interdisciplinary discharge rehabilitation team.This study provided a comprehensive list of 91 factors across all seven mobility determinants (cognitive, environmental, financial, personal, physical, psychological, and social) for clinicians in other care settings to use as a starting point to determine which mobility factor should be assessed during older adults' hospital-to-home transition.This international e-Delphi study identified 43 factors within mobility determinants (cognitive, environmental, personal, physical, psychological, and social) to be included in a Comprehensive Mobility Discharge Assessment Framework to assess older adults' mobility during the hospital to home transition.Using these 43 factors, clinicians can identify which assessment tool is best suited to assess the factors while reflecting on the logistics and feasibility; this is the next phase of this project.

Assessing the safety of physical rehabilitation in critically ill patients: a Delphi study

BACKGROUND

Physical rehabilitation of critically ill patients is implemented to improve physical outcomes from an intensive care stay. However, before rehabilitation is implemented, a risk assessment is essential, based on robust safety data. To develop this information, a uniform definition of relevant adverse events is required. The assessment of cardiovascular stability is particularly relevant before physical activity as there is uncertainty over when it is safe to start rehabilitation with patients receiving vasoactive drugs.

METHODS

A three-stage Delphi study was carried out to (a) define adverse events for a general ICU cohort, and (b) to define which risks should be assessed before physical rehabilitation of patients receiving vasoactive drugs. An international group of intensive care clinicians and clinician researchers took part. Former ICU patients and their family members/carers were involved in generating consensus for the definition of adverse events. Round one was an open round where participants gave their suggestions of what to include. In round two, participants rated their agreements with these suggestions using a five-point Likert scale; a 70% consensus agreement threshold was used. Round three was used to re-rate suggestions that had not reached consensus, whilst viewing anonymous feedback of participant ratings from round two.

RESULTS

Twenty-four multi-professional ICU clinicians and clinician researchers from 10 countries across five continents were recruited. Average duration of ICU experience was 18 years (standard deviation 8) and 61% had publications related to ICU rehabilitation. For the adverse event definition, five former ICU patients and one patient relative were recruited. The Delphi process had a 97% response rate. Firstly, 54 adverse events reached consensus; an adverse event tool was created and informed by these events. Secondly, 50 risk factors requiring assessment before physical rehabilitation of patients receiving vasoactive drugs reached consensus. A second tool was created, informed by these suggestions.

CONCLUSIONS

The adverse event tool can be used in studies of physical rehabilitation to ensure uniform measurement of safety. The risk assessment tool can be used to inform clinical practise when risk assessing when to start rehabilitation with patients receiving vasoactive drugs. Trial registration This study protocol was retrospectively registered on https://www.researchregistry.com/ (researchregistry2991).

Building consensus on priority areas for Sub-Saharan Africa's ageing population research: An e-Delphi study protocol

BACKGROUND

Improvement in medico-social services has increased life expectancy and population ageing in Sub-Saharan Africa (SSA). It was estimated that about 163 million people aged 65 and older will be resident in SSA by 2050. There is inadequate ageing research capacity in SSA which necessitates this study to (a) identify a decade-long ageing research opportunities, challenges, and solutions, and (b) prioritize critical ageing research areas and methodologies relevant to the SSA.

METHODS

We designed an e-Delphi protocol following the Reporting Guideline for Priority Setting of Health Research with Stakeholder. The stakeholders will be researchers, practitioners, older adults, and caregivers purposively selected through snowballing quota sampling to complete three rounds of e-Delphi surveys. Round 1 will involve open-ended questions derived from the study objectives. Responses from round 1 will be prepared as a checklist for stakeholders to rate during rounds 2 & 3, using a 9-point scale: low priority (1-3), moderate priority (4-6), and high priority (7-9). The criterion for reaching a consensus will be ≥ 70% of stakeholders rating an item "high priority" and ≤ 15% as "low priority." Quantitative data will be analysed using descriptive statistics, Wilcoxon matched-pairs signed-rank test will be used to assess the stability of stakeholders' responses, and qualitative comments will be analysed using content analysis.

DISCUSSION AND IMPLICATIONS

Setting aging research/practice priorities will help maximize the benefits of research investment and provide valuable direction for allocating public and private research funds to areas of strategic importance.

Evaluation in design science: A framework to support project studies in the context of University Research Centres

This paper proposes a discussion of the evaluation of an artefact developed under the Design Science paradigm using the Delphi method. It evaluates the Ecosystem framework of University Research Centres in Project Studies, considering a set of criteria pre-established in the literature. The Delphi method is an evaluation implemented in an electronic platform involving twenty-one participants, among whom were academics, practitioners, and PhD candidates in the field of project management. It reached consensus and stability in two rounds: the results indicate a consensus among the participants in the applicability, novelty, simplicity, completeness, fidelity to modelled phenomena, consistency and internal coherence, scalability, flexibility, interest, elegance, and reusability criteria. Usability was the only criterion that did not attain the predefined percentage of consensus among the participants (70%). Given the framework's characteristics, Delphi participants indicated the need to produce complementary guidelines for its implementation.

Assessment of Critical Health and Safety Risks in Homes where Hoarding is Prevalent

Hoarding behaviour sometimes requires intervention from community agencies to reduce risks to residents and the nearby community. Human services professionals from a wide range of disciplines are called upon to address hoarding concerns, often in collaboration with each other. No guidelines currently exist to guide staff from those community agencies in a shared understanding of common health and safety risks that occur with severe hoarding behaviour. Using a modified Delphi method, we aimed to generate consensus among a panel of 34 service-provider experts from a range of disciplines on essential risks in the home that would require intervention for health or safety reasons. This process identified 31 environmental risk factors that experts agreed are critical to assess in cases of hoarding. Panelist comments outlined the debates that commonly occur in the field, the complexity of hoarding, and the difficulty with conceptualizing risks in the home. The multi-disciplinary consensus achieved on these risks will facilitate better collaboration between agencies by providing a minimum standard of what to evaluate in hoarded homes to ensure health and safety standards are being met. This can improve communication between agencies, specify the core hazards that should be incorporated into training for professionals who work with hoarding, and facilitate more standardized assessment of health and safety risks in hoarded homes.

Implementation of a text messaging intervention to patients on warfarin therapy in Brazilian primary care units: a quasi-experimental study

Background

Warfarin remains the most affordable oral anticoagulant in many countries. However, it may have serious side effects, and the success of the therapy depends on the patient’s understanding of the medication and their adherence to treatment. The use of short messages services (SMS) is a strategy that can be used to educate patients, but there are no studies evaluating this intervention in patients taking warfarin. Therefore, we aimed to develop, implement, and assess the feasibility of an intervention using SMS to primary care patients taking warfarin in a medium-sized Brazilian city.

Methods

A bank of 79 SMS was drafted and validated by an expert panel. During 6 months, three times a week, patients received messages about anticoagulation with warfarin. At baseline and after 3 months, we assessed their knowledge and adherence with validated instruments. At the end of the follow-up, participants answered a satisfaction questionnaire. Subsequently, a scale-up phase was conducted, with another round of the intervention including 82 participants (29 from the first phase and 53 newly recruited). Seven months after the end of the scale-up, we asked the patients for their insights about the long-term effects of this program. All patients signed informed consent. The study was approved by the Research and Ethics committee of the Universidade Federal de Minas Gerais.

Results

In the pilot, 33 (89.2%) patients completed the follow-up. Among the participants who answered the satisfaction questionnaire (n = 29), 86.2% considered that the intervention motivated a healthy lifestyle and improved their understanding of warfarin therapy. All patients were willing to continue receiving the messages. Adherence measured by the Measure of Adherence to Treatment (MAT) test was high in the pre-intervention assessment and remained high (96.7% vs. 93.3%; p = 1.0000). The proportion of patients who achieved > 75% correct answers on the Oral Anticoagulation Knowledge (OAK) test increased from 6.5% to 25.6, p = 0.0703. In the scale-up, 23 patients answered the long-term assessment questionnaire. The main long-term knowledge reported was dietary information. Nine patients received the messages but did not remember their content.

Conclusion

The intervention was well-accepted and had a positive impact on patient’s knowledge about oral anticoagulation therapy. The scale-up assessment reinforced the need to constantly monitor digital interventions.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12875-022-01647-5.

[The Delphi technique: Methodology, variants and usage examples]

In the field of medicine and health sciences, Delphi methods are applied mainly in the exploratory or evaluative phases of a research process. Explicit and implicit knowledge of respected experts from research and practice is systematically synthesized. Originally developed as a method for structuring a group communication process, Delphi techniques have been established in the health sector as a consensus method. The findings are used to improve the evidence and acceptance of planned interventions or necessary standards or guidelines and to increase the probability of successful implementation in practice. However, different variants of Delphi methods have been developed in recent years, which are systematically contrasted and reflected in this paper with regard to key epistemological and methodological research activities. Based on this overview, researchers should be enabled to select the most suitable Delphi technique for their own research questions and research endeavors.

A comprehensive mobility discharge assessment framework for older adults transitioning from hospital-to-home in the community—What mobility factors are critical to include? Protocol for an international e-Delphi study

Background

Mobility deficits have been identified as an independent risk factor for hospital readmission for adults ≥65 years. Despite evidence indicating how determinants additively influence and predict mobility, no hospital-to-home care transition models comprehensively assess all seven mobility determinants, cognitive, financial, environmental, personal, physical, psychological, and social. There is currently a lack of clarity regarding what factors clinicians and researchers should evaluate for each mobility determinant. The purpose of this e-Delphi study is to prioritize and reach consensus on the factors for each mobility determinant that are critical to assess as part of the Comprehensive Mobility Discharge Assessment Framework (CMDAF) when older adults are discharged from hospital-to-home.

Methods

This protocol paper is an international modified e-Delphi study following the Recommendations for the Conducting and Reporting of Delphi Studies. International researchers, clinicians, older adults and family caregivers residing in a country with universal or near-universal health coverage will be invited to participate as ‘experts’ in three e-Delphi rounds administered through DelphiManager©. The e-Delphi Round 1 questionnaire will be developed based on scoping review findings and will be pilot tested. For each round, experts will be asked to rate factors for each determinant that are critical to assess as part of the CMDAF using a 9-point scale: Not Important (1–3), Important but Not Critical (4–6), and Critical (7–9). The scale will include a selection option of "unable to score" and experts will also be asked to provide a rationale for their scoring and suggest missing factors. Experts will receive feedback summaries in Rounds 2 and 3 to guide them in reflecting on their initial responses and re-rating of factors that have not reached consensus. The criteria for reaching consensus will be if ≥70% of experts rate a factor as "critical" (scores ≥7) and ≤ 15% of experts rate a factor as "not important" (scores≤ 3). Quantitative data will be analyzed using median values, frequencies, percentages, interquartile range, and bar graphs; Wilcoxon matched-pairs signed-rank test will be used to assess the stability of participants’ responses. Rationale (qualitative data) provided in the open-ended comments section will be analyzed using content analysis.

Conclusion

This study is a first step in developing the CMDAF and will be used to guide a subsequent e-Delphi survey to decide on the tools that should be used to measure the examples of each factor included in our framework.

Development of a performance standard for physiotherapists delivering exercise and mobilisation to the critically ill: A modified Delphi consensus study

BACKGROUND

The provision of early mobilisation to critically ill patients has the potential to improve long term outcomes, but, is complex to deliver. There is minimal literature detailing the training and expertise required to deliver these interventions safely and effectively.

OBJECTIVE

The objective of this study was to determine the key elements of a performance standard for assessment of physiotherapists delivering exercise and mobilisation interventions to the critically ill.

METHOD

This is a modified eDelphi expert consensus study. Fifty-one physiotherapists from Australia and New Zealand with relevant clinical, educational, or research experience were included on the expert panel. Background information and the initial pool of items were developed from review of relevant literature. Five survey rounds were administered across two study phases to determine the elements, performance criteria, and assessment scale of the performance standard. Items were modified, amalgamated, and added based upon panel comments.

RESULTS

Consensus was achieved for 69 mandatory, and two supplementary performance criteria which were arranged under 15 elements encompassing knowledge, assessment, analysis, intervention, and professional behaviours. A 3-point rating scale was selected to assess item achievement and global performance.

CONCLUSION

Binational expert consensus was reached to define the assessment criteria for physiotherapists delivering exercise and mobilisation interventions to the critically ill. This standard can be utilised in clinical, educational, and research practice environments to guide training, assessment, and skill recognition in critical care physiotherapy.

Reaching Consensus on Politicized Topics: A Convening of Public Health Professionals to Discuss Appropriate Abortion Activities for US Health Departments

CONTEXT

Public health professionals, particularly those in state and local health departments, do not always have clear understandings of their roles related to politically controversial public health topics. A process of consensus development among public health professionals that considers the best available evidence may be able to guide decision making and lay out an appropriate course of action.

APPROACH

In May 2020, a group of maternal and child health and family planning professionals working in health departments, representatives of schools of public health, and members of affiliated organizations convened to explore values and principles relevant to health departments' engagement in abortion and delineate activities related to abortion that are appropriate for health departments. The convening followed a structured consensus process that included multiple rounds of input and opportunities for feedback and revisions.

OUTCOMES

Convening participants came to consensus on principles to guide engagement in activities related to abortion, a set of activities related to abortion that are appropriate for health departments, and next steps to support implementation of such activities.

LESSONS LEARNED

The experience of the convening indicates that consensus processes can be feasible for politically controversial public health topics such as abortion.

A Mapping Review on the Uptake of the COVID-19 Vaccine among Adults in Africa Using the 5A's Vaccine Taxonomy

Uptake of a vaccine is complete if individuals are aware of the associated risks of the vaccine, accept the vaccine, and respond positively to the nudges (activation) to increase the uptake, and respond when the vaccine is made accessible and affordable. We mapped systematically the existing literature concerning the 5As-acceptability, accessibility, affordability, awareness, and activation-of COVID-19 vaccination among adults and, specifically, older adults (55 years and older) in Africa. We searched multiple databases from 2020 to December 2021. Using predefined inclusion and exclusion criteria, two reviewers screened citations, conducted title and abstract screening, and extracted data independently. We included 68 articles conducted in 33 African countries, primarily cross-sectional studies (n = 49, 72%). None of the articles focused on older adults only, but 22 articles (32%) included at least one older adult (55 years and older) in their sample size. Acceptance (n = 58, 85%) was the most commonly researched aspect of vaccine uptake, followed by accessibility (n = 17, 25%), awareness (n = 13, 19%), and affordability (n = 5, 7.0%). We found only one report on activation. Factors affecting acceptance of the COVID-19 vaccine in Africa were grouped into sociodemographic factors; knowledge-, attitude-, and belief-related factors; a COVID-19 vaccine efficacy and safety concern factor; and trust in government and public health authorities. The governments of African nations should focus on strategies to influence the modifiable factors identified in this review. More studies are needed to evaluate the impact of nudges (activation) to improve COVID-19 vaccine uptake in African nations.

[Information tool about labor and delivery designed by Delphi method]

OBJECTIVE

The integration of couples in decision making in obstetrics is necessary to improve the experience of giving birth. The objective of this work is to elaborate a new tool to provide information to health professionals.

METHODS

A 5-round Delphi method was used to define the information points. The 20 multi-disciplinary experts answered 5 surveys on the information to deliver concerning childbirth. The selected items were organized with mind mapping.

RESULTS

54 items were identified on the first round. 5 were retained after the second survey while the rest were reformulated and submitted for the third round. Amongst the 47 items submitted, 19 were retained, the rest were removed, merged or reworded. The fourth survey included 22 items of which 11 were retained. The fifth survey concerned the 11 remaining items and 2 new ones. During this round, 7 items were retained. This round made it possible to reach the consensus. In total, 42 items reached the consensus of the experts. They were then split into 5 mind maps allowing their visual organization to form the information tool.

CONCLUSION

The tool elaborated in a multi-disciplinary manner offers complementary information to that already given during childbirth preparation courses. It is a framework for the information to be given that leaves the content up to each professional.

Identifying competencies for integrated knowledge translation: a Delphi study

Background

Considerable progress has been made to advance the field of knowledge translation to address the knowledge-to-action gap in health care; however, there remains a growing concern that misalignments persist between research being conducted and the issues faced by knowledge users, such as clinicians and health policy makers, who make decisions in the health care context. Integrated knowledge translation (IKT) is a collaborative research model that has shown promise in addressing these concerns. It takes advantage of the unique and shared competencies amongst researchers and knowledge users to ensure relevance of the research process and its outcomes. To date, core competencies have already been identified to facilitate training in knowledge translation more generally but they have yet to be prioritized for IKT more specifically. The primary aim of this study was to recruit a group of researchers and knowledge users to identify and prioritize core competencies for researchers and knowledge users to engage with IKT.

Methods

We recruited health care knowledge users (KUs) and researchers with experience and knowledge of IKT for a quantitative, cross-sectional study. We employed a modified Delphi approach consisting of three e-survey rounds to establish consensus on competencies important to IKT for KUs and researchers based on mean rating of importance and agreement between participants.

Results

Nineteen (73%) of the initial 26 participants were researchers (response rate = 41% in the first round; retention in subsequent rounds > 80%). Participants identified a total of 46 competencies important for IKT (18 competencies for KUs, 28 competencies for researchers) under 3 broad domains. Technical research skills were deemed extremely important for researchers, while both groups require teamwork and knowledge translation skills.

Conclusions

This study provides important insight into distinct and overlapping IKT competencies for KUs and researchers. Future work could focus on how these can be further negotiated and contextualized for a wide range of IKT contexts, projects and teams. Greater attention could also be paid to establishing competencies of the entire team to support the research co-production process.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-07107-7.

Inclusion of participants from low-income and middle-income countries in core outcome sets development: a systematic review

OBJECTIVE

Our study aims to describe differences or similarities in the scope, participant characteristics and methods used in core outcome sets (COS) development when only participants from high-income countries (HICs) were involved compared with when participants from low-income and middle-income countries (LMICs) were also involved.

DESIGN

Systematic review.

DATA SOURCES

Annual Core Outcome Measures in Effectiveness Trials systematic reviews of COS which are updated based on SCOPUS and MEDLINE, searches. The latest systematic review included studies published up to the end of 2019.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

We included studies reporting development of a COS for use in research regardless of age, health condition or setting. Studies reporting the development of a COS for patient-reported outcomes or adverse events or complications were also included.

DATA EXTRACTION AND SYNTHESIS

Data were extracted in relation to scope of the COS study, participant categories and the methods used in outcome selection.

RESULTS

Studies describing 370 COS were identified in the database. Of these, 75 (20%) included participants from LMICs. Only four COS were initiated from an LMIC setting. More than half of COS with LMIC participants were developed in the last 5 years. Cancer and rheumatology were the dominant disease domains. Overall, over 259 (70%) of COS explicitly reported including clinical experts; this was higher where LMIC participants were also included 340 (92%). Most LMIC participants were from China, Brazil and South Africa. Mixed methods for consensus building were used across the two settings.

CONCLUSION

Progress has been made in including LMIC participants in the development of COS, however, there is a need to explore how to enable initiation of COS development from a range of LMIC settings, how to ensure prioritisation of COS that better reflects the burden of disease in these contexts and how to improve public participation from LMICs.

Development of the CORE-Kids core set of outcome domains for studies of childhood limb fractures

Aims

The aim of this study is to develop a core set of outcome domains that should be considered and reported in all future trials of childhood limb fractures.

Methods

A four-phase study was conducted to agree a set of core outcome domains. Identification of candidate outcome domains were identified through systematic review of trials, and outcome domains relevant to families were identified through semi-structured interviews with 20 families (parent-child pairing or group). Outcome domains were prioritized using an international three-round Delphi survey with 205 panellists and then condensed into a core outcome set through a consensus workshop with 30 stakeholders.

Results

The systematic review and interviews identified 85 outcome domains as relevant to professionals or families. The Delphi survey prioritized 30 upper and 29 lower limb outcome domains at first round, an additional 17 upper and 18 lower limb outcomes at second round, and four additional outcomes for upper and lower limb at the third round as important domains. At the consensus workshop, the core outcome domains were agreed as: 1) pain and discomfort; 2) return to physical and recreational activities; 3) emotional and psychosocial wellbeing; 4) complications from the injury and treatment; 5) rturn to baseline activities daily living; 6) participation in learning; 7) appearance and deformity; and 8) time to union. In addition, 9a) recovery of mobility and 9b) recovery of manual dexterity was recommended as a core outcome for lower and upper limb fractures, respectively.

Conclusion

This set of core outcome domains is recommended as a minimum set of outcomes to be reported in all trials. It is not an exhaustive set and further work is required to identify what outcome tools should be used to measure each of these outcomes. Adoption of this outcome set will improve the consistency of research for these children that can be combined for more meaningful meta-analyses and policy development.

Cite this article: Bone Joint J 2021;103-B(12):1821–1830.

Applying real-time Delphi methods: development of a pain management survey in emergency nursing

The modified Delphi technique is widely used to develop consensus on group opinion within health services research. However, digital platforms are offering researchers the capacity to undertake a real-time Delphi, which provides novel opportunities to enhance the process. The aim of this case study is to discuss and reflect on the use of a real-time Delphi method for researchers in emergency nursing and cognate areas of practice. A real-time Delphi method was used to develop a national survey examining knowledge, perceptions and factors influencing pain assessment and management practices among Australian emergency nurses. While designing and completing this real-time Delphi study, a number of areas, emerged that demanded careful consideration and provide guidance to future researchers.

Uniform Registration Agreements on Cholesteatoma Care: A Nationwide Consensus Procedure

Background:

To coordinate and align the content for registration of cholesteatoma care.

Methods:

Systematic Delphi consensus procedure, consisting three rounds: two written sessions followed by a face-to-face meeting. Before this procedure, input on important patient outcomes was obtained. Consensus was defined as at least 80% agreement by participants. Hundred-thirty-six adult patients who had undergone cholesteatoma surgery and all ENT surgeons of the Dutch ENT Society were invited. The consensus rounds were attended by ENT surgeons with cholesteatoma surgery experience. Feasibility and acceptability of outcome measures and reporting agreements were assessed in round 1 by 150 ENT surgeons. In round 2 definitions were narrowed and context information to interpret outcome measure were questioned. In round 3, the results, amendments, and the open-ended points were discussed to reach agreement.

Results:

Most important outcome measures are: 1) the presence or absence of a cholesteatoma in the first 5 years after surgical removal of cholesteatoma, 2) hearing level after surgical removal of cholesteatoma, and 3) the documented assessment of patient's complaints with a validated patient reported outcome measures questionnaire (PROM). Furthermore, consensus was reached on the registration of cholesteatoma type (residual/recurrent), localization of cholesteatoma, and reporting of the presence of cholesteatoma in the follow-up.

Conclusion:

Consensus was reached on the content and method of registration of cholesteatoma care based on patient's and ENT surgeons input. Three outcome measures were defined. National agreements on the method and content of registration will facilitate monitoring and feedback to the ENT surgeon about the cholesteatoma care.

"Vicarious thinking" was a key driver of score change in Delphi surveys for COS development and is facilitated by feedback of results

OBJECTIVE

The objectives of this nested study were to (1) assess whether changes in scores between rounds altered the final degree of consensus achieved in three Delphi surveys conducted as part of COS development projects (anal, gastric, and prostate cancer), and (2) explore participants' reasons for changing scores between rounds.

STUDY DESIGN AND SETTING

All Delphi surveys were conducted online using DelphiManager software and included healthcare professionals and participating patients. Participants were invited to give a free-text reason whenever they changed their score across an important threshold on a 1-9 Likert scale (1-3 not important, 4-5 important, 7-9 critically important). Reasons for score change were coded by four researchers independently using an inductive-iterative approach.

RESULTS

In all three Delphi surveys, the number of outcomes reaching criteria for consensus was greater in R2 than R1. Twelve themes and 23 subthemes emerged from 2298 discrete reasons given for score change. The most common reasons for the change were "time to reflect" (482 responses, 23%) and vicarious thinking (424, 21%), with 68% (291) of vicarious thinking attributed to seeing other participant's scores.

CONCLUSION

Our findings support conducting a Delphi survey over the use of a single questionnaire where building consensus is the objective. Time to reflect and vicarious thinking, facilitated by seeing other participant's scores, were important drivers of score change. How results are presented to participants between rounds and the duration of and time between rounds in a Delphi survey may, therefore, influence the results and should be clearly reported.

Two different invitation approaches for consecutive rounds of a Delphi survey led to comparable final outcome

OBJECTIVES

There are two different approaches to involve participants in consecutive rounds of a Delphi survey: (1) invitation to every round independent of response to the previous round ("all-rounds") and (2) invitation only when responded to the previous round ("respondents-only"). This study aimed to investigate the effect of invitation approach on the response rate and final outcome of a Delphi survey.

STUDY DESIGN AND SETTING

Both experts (N = 188) and patients (N = 188) took part in a Delphi survey to update the core outcome set (COS) for axial spondyloarthritis. A study with 1:1 allocation to two experimental groups (ie, "all-rounds" [N = 187] and "respondents-only" [N = 189]) was built-in.

RESULTS

The overall response rate was lower in the "respondents-only group" (46%) compared to the "all-rounds group" (61%). All domains that were selected for inclusion in the COS by the "respondents-only group" were also selected by the "all-rounds group." Additionally, the four most important domains were identical between groups after the final round, with only minor differences in the other domains.

CONCLUSION

Inviting panel members who missed a round to a subsequent round will lead to a better representation of opinions of the originally invited panel and reduces the chance of false consensus, while it does not influence the final outcome of the Delphi.

Delphi Technique in Health Sciences: A Map

Objectives: In health sciences, the Delphi technique is primarily used by researchers when the available knowledge is incomplete or subject to uncertainty and other methods that provide higher levels of evidence cannot be used. The aim is to collect expert-based judgments and often to use them to identify consensus. In this map, we provide an overview of the fields of application for Delphi techniques in health sciences in this map and discuss the processes used and the quality of the findings. We use systematic reviews of Delphi techniques for the map, summarize their findings and examine them from a methodological perspective.

Methods: Twelve systematic reviews of Delphi techniques from different sectors of the health sciences were identified and systematically analyzed.

Results: The 12 systematic reviews show, that Delphi studies are typically carried out in two to three rounds with a deliberately selected panel of experts. A large number of modifications to the Delphi technique have now been developed. Significant weaknesses exist in the quality of the reporting.

Conclusion: Based on the results, there is a need for clarification with regard to the methodological approaches of Delphi techniques, also with respect to any modification. Criteria for evaluating the quality of their execution and reporting also appear to be necessary. However, it should be noted that we cannot make any statements about the quality of execution of the Delphi studies but rather our results are exclusively based on the reported findings of the systematic reviews.

Core outcomes sets for studies evaluating critical illness and patient recovery

PURPOSE OF REVIEW

Given the growing body of critical care clinical research publications, core outcome sets (COSs) are important to help mitigate heterogeneity in outcomes assessed and measurement instruments used, and have potential to reduce research waste. This article provides an update on COS projects in critical care medicine, and related resources and tools for COS developers.

RECENT FINDINGS

We identified 28 unique COS projects, of which 15 have published results as of May 2020. COS topics relevant to critical care medicine include mechanical ventilation, cardiology, stroke, rehabilitation, and long-term outcomes (LTOs) after critical illness. There are four COS projects for coronavirus disease 2019 (COVID-19), with a 'meta-COS' summarizing common outcomes across these projects. To help facilitate COS development, there are existing resources, standards, guidelines, and tools available from the Core Outcome Measures in Effectiveness Trials Initiative (www.comet-initiative.org/) and the National Institutes of Health-funded Improve LTO project (www.improvelto.com/).

SUMMARY

Many COS projects have been completed in critical care, with more on-going COS projects, including foci from across the spectrum of acute critical care, COVID-19, critical care rehabilitation, and patient recovery and LTOs. Extensive resources are accessible to help facilitate rigorous COS development.

Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response

Background

A proper application of the Delphi technique is essential for obtaining valid research results. Medical researchers regularly use Delphi studies, but reports often lack detailed information on methodology and controlled feedback: in the medical literature, papers focusing on Delphi methodology issues are rare. Since the introduction of electronic surveys, details on response times remain scarce. We aim to bridge a number of gaps by providing a real world example covering methodological choices and response times in detail.

Methods

The objective of our e(lectronic)-Delphi study was to determine minimum standards for emergency departments (EDs) in the Netherlands. We opted for a two-part design with explicit decision rules. Part 1 focused on gathering and defining items; Part 2 addressed the main research question using an online survey tool. A two-person consensus rule was applied throughout: even after consensus on specific items was reached, panellists could reopen the discussion as long as at least two panellists argued similarly. Per round, the number of reminders sent and individual response times were noted. We also recorded the methodological considerations and evaluations made by the research team prior to as well as during the study.

Results

The study was performed in eight rounds and an additional confirmation round. Response rates were 100% in all rounds, resulting in 100% consensus in Part 1 and 96% consensus in Part 2. Our decision rules proved to be stable and easily applicable. Items with negative advice required more rounds before consensus was reached. Response delays were mostly due to late starts, but once panellists started, they nearly always finished the questionnaire on the same day. Reminders often yielded rapid responses. Intra-individual differences in response time were large, but quick responders remained quick.

Conclusions

We advise those considering Delphi study to follow the CREDES guideline, consider a two-part design, invest in personal commitment of the panellists, set clear decision rules, use a consistent lay-out and send out your reminders early. Adopting this overall approach may assist researchers in future Delphi studies and may help to improve the quality of Delphi designs in terms of improved rigor and higher response rates.

Participating in core outcome set development via Delphi surveys: qualitative interviews provide pointers to inform guidance

OBJECTIVES

To explore participants' views of Delphi surveys in core outcome set (COS) development.

STUDY DESIGN AND SETTING

Patients and health professionals (n=24) from seven recently concluded COS studies that had involved a Delphi survey took part in semistructured qualitative interviews (telephone and email exchange). Interviews explored participants' understanding of COS and their experiences of the Delphi survey. Analysis was thematic.

RESULTS

Several interviewees had previously participated in two or more COS or Delphi surveys. Those with multiple experiences of participation generally understood the purpose of COS and were satisfied with the Delphi survey. However, some interviewees who were first-time participants struggled to understand the purpose of COS and aspects of the Delphi survey, which limited their contribution and satisfaction with the study. Interviewees also differed in how they interpreted and subsequently used the written documentation provided to COS participants. Some interviewees wanted guidance regarding whose perspective to take into account when scoring outcomes and on how to use the scoring system. Interviewees reported being motivated to take part by the international and expert consensus aspects of the Delphi survey. A few interviewees reported experiencing either positive or negative emotional impacts arising from when they reviewed outcomes and stakeholder feedback.

CONCLUSION

This study identifies important information that should be communicated to COS Delphi study participants. It also indicates the importance of communicating about COS Delphi studies in ways that are accessible and salient to participants, to enhance their experience of participation and make the process more meaningful for all.

A clinimetric analysis of the Pressure Ulcer Risk Primary or Secondary Evaluation Tool: PURPOSE-T

The assessment of patients' risk for developing pressure ulcers is a routine and fundamental nursing process undertaken to prevent avoidable harm to patients in all care settings. Many risk assessment tools are currently used in clinical practice, however no individual tool is recommended by advisory bodies such as the National Institute for Health and Care Excellence or the European Pressure Ulcer Advisory Panel. The evidence base on the value of structured risk assessment tools in reducing the incidence or severity of pressure ulcers is poor. This purpose of this article is to provide a clinimetric analysis of the recently developed Pressure Ulcer Risk Primary or Secondary Evaluation Tool (PURPOSE-T) and identify areas for future research to improve the utility of structured risk assessment in identifying patients at risk of developing pressure ulcers.

Delphi procedure in core outcome set development: rating scale and consensus criteria determined outcome selection

OBJECTIVE

The objective of this study was to compare two different rating scales within one Delphi study for defining consensus in core outcome set development and to explore the influence of consensus criteria on the outcome selection.

STUDY DESIGN AND SETTING

Randomized controlled parallel group trial with 1:1 allocation within the first Delphi round of the Core Outcome Set in the Incontinence-Associated Dermatitis project. Outcomes were rated on a three-point or nine-point Likert scale. Decisions about which outcomes to retain were determined by commonly used consensus criteria (i.e., [combinations of] proportions with restricted ranges, central tendency within a specific range, and decrease in variance).

RESULTS

Fifty-seven participants (group 1 = 28, group 2 = 29) rated 58 outcomes. The use of the nine-point scale resulted in almost twice as many outcomes being rated as "critical" compared with the three-point scale (24 vs. 13). Stricter criteria and combining criteria led to less outcomes being identified as "critical".

CONCLUSION

The format of rating scales in Delphi studies for core outcome set development and the definition of the consensus criteria influence outcome selection. The use of the nine-point scale might be recommended to inform the consensus process for a subsequent rating or face-to-face meeting. The three-point scale might be preferred when determining final consensus.

Consensus Building in OMERACT: Recommendations for Use of the Delphi for Core Outcome Set Development

Objective.

Developing international consensus on outcome measures for clinical trials is challenging. The following paper will review consensus building in Outcome Measures in Rheumatology (OMERACT), with a focus on the Delphi.

Methods.

Based on the literature and feedback from delegates at OMERACT 2018, a set of recommendations is provided in the form of the OMERACT Delphi Consensus Checklist.

Results.

The OMERACT delegates generally supported the use of the checklist as a guide. The checklist provides guidance for clearly outlining the multiple aspects of the Delphi process.

Conclusion.

OMERACT is deeply committed to consensus building and these recommendations should be considered a work in progress.

Higher number of items associated with significantly lower response rates in COS Delphi surveys

OBJECTIVES

The Delphi method is commonly used to achieve consensus in core outcome set (COS) development. It is important to try to maximize response rates to Delphi studies and minimize attrition rates and potential for bias. The factors that impact response rates in a Delphi study used for COS development are unknown. The objective of this study was to explore the impact of design characteristics on response rates in Delphi surveys within COS development.

METHODS

Published and ongoing studies that included Delphi to develop a COS were eligible. Second round voting response rates were analyzed, and multilevel linear regression was conducted to investigate whether design characteristics were associated with the response rate.

RESULTS

Thirty-one studies were included. Two characteristics were significantly associated with a lower response rate: larger panels and studies with more items included.

CONCLUSION

COS developers should pay attention to methods when designing a COS development study; in particular, the size of the panels and the size of the list of outcomes. We identified other potential design characteristics that might influence response rates but were unable to explore them in this analysis. These should be reported in future reports to allow for further investigation.