A method for studying decision-making by guideline development groups

Guideline panel social dynamics influence the development of clinical practice recommendations: a mixed-methods systematic review

OBJECTIVES

To synthesize empirical studies that investigate the cognitive and social processes involved in the deliberation process of guideline development meetings and determine the distribution of deliberated topics.

STUDY DESIGN AND SETTING

We conducted a mixed-method systematic review using a convergent segregated approach. We searched for empirical studies that investigate the intragroup dynamics of guideline development meetings indexed in bibliographic databases.

RESULTS

Of the 5,899 citations screened, 12 studies from six countries proved eligible. Chairs, cochairs, and methodologists contributed to at least one-third of the discussion time in guideline development meetings; patient partners contributed the least. In interdisciplinary groups, male gender and occupation as a physician were positively associated with the amount of contribution. Compared to groups that used the Grading of Recommendations Assessment, Development and Evaluation approach, for groups that did not, when faced with insufficient or low-quality evidence, relied more on their clinical experience. The presence of a cognitive "yes" bias was apparent in meetings: panelists tended to acquiesce with positive statements that required less cognitive effort than negative statements.

CONCLUSION

The social dynamics of the discussions were linked to each panelist's activity role, professional background, and gender, all of which influenced the level of contributions they made in guideline development meetings.

A few panel members dominated guideline development meeting discussions: Social network analysis

OBJECTIVES

To identify patterns of interactions that may influence guideline panels' decision-making.

STUDY DESIGN AND SETTING

Social network analysis (SNA) to describe the conversation network in a guideline development meeting in United States.

RESULTS

We analyzed one two-day guideline panel meeting that included 20 members who developed a guideline using the GRADE (Grading of Recommendations Assessment, Development and Evaluation) approach. The conversation pattern of the guideline panel indicated a well-connected network (density=0.59, clustering coefficient=0.82). GRADE topics on quality of evidence and benefits versus harms accounted for 46%; non-GRADE factors accounted for 30% of discussion. The chair, co-chair and methodologist initiated 53% and received 60% of all communications in the meeting; 42% of their communications occurred among themselves. SNA metrics (eigenvector, betweenness and closeness) indicated that these individuals also exerted highest influence on discussion, controlled information flow and were at the center of all communications. Members were more likely to continue previous discussion with the same individuals after both morning breaks (r=0.54, P<0.005; r=0.17, P=0.04), and after the last break on day 2 (r=0.44, P=0.015).

CONCLUSION

Non-GRADE factors such as breaks, and the members' roles, affect guideline development more than previously recognized. Collectively, the chair, co-chair and methodologist dominated the discussion.

Assessing the process and outcome of the development of practice guidelines and recommendations: PANELVIEW instrument development

BACKGROUND

Guideline recommendations may be affected by flaws in the process, inappropriate panel member selection or conduct, conflicts of interest and other factors. To our knowledge, no validated tool exists to evaluate guideline development from the perspective of those directly involved in the process. Our objective was to develop and validate a universal tool, the PANELVIEW instrument, to assess guideline processes, methods and outcomes from the perspective of the participating guideline panellists and group members.

METHODS

We performed a systematic literature search and surveys of guideline groups (identified through contacting international organizations and convenience sampling of working panels) to inform item generation. Subsequent groups of guideline methodologists and panellists reviewed items for face validity and missing items. We used surveys, interviews and expert review for item reduction and phrasing. For reliability assessment and feedback, we tested the PANELVIEW tool in 8 international guideline groups.

RESULTS

We surveyed 62 members from 13 guideline panels, contacted 19 organizations and reviewed 20 source documents to generate items. Fifty-three additional key informants provided feedback about phrasing of the items and response options. We reduced the number of items from 95 to 34 across domains that included administration, training, conflict of interest, group dynamics, chairing, evidence synthesis, formulating recommendations and publication. The tool takes about 10 minutes to complete and showed acceptable measurement properties.

INTERPRETATION

The PANELVIEW instrument fills a gap by enabling guideline organizations to involve clinicians, patients and other participants in evaluating their guideline processes. The tool can inform quality improvement of existing or new guideline programs, focusing on insight into and transparency of the guideline development process, methods and outcomes.

Decision-making and evidence use during the process of prenatal record review in Canada: a multiphase qualitative study

BACKGROUND

Prenatal records are potentially powerful tools for the translation of best-practice evidence into routine prenatal care. Although all jurisdictions in Canada use standardized prenatal records to guide care and provide data for health surveillance, their content related to risk factors such as maternal smoking and alcohol use varies widely. Literature is lacking on how prenatal records are developed or updated to integrate research evidence. This multiphase project aimed to identify key contextual factors influencing decision-making and evidence use among Canadian prenatal record committees (PRCs), and formulate recommendations for the prenatal record review process in Canada.

METHODS

Phase 1 comprised key informant interviews with PRC leaders across 10 Canadian jurisdictions. Phase 2, was a qualitative comparative case study of PRC factors influencing evidence-use and decision-making in five selected jurisdictions. Interview data were analysed using qualitative content analysis. Phase 3 involved a dissemination workshop with key stakeholders to review and refine recommendations derived from Phases 1 and 2.

RESULTS

Prenatal record review processes differed considerably across Canadian jurisdictions. PRC decision-making was complex, revealing the competing functions of the prenatal record as a clinical guide, documentation tool and data source. Internal contextual factors influencing evidence use included PRC resources to conduct evidence reviews; group composition and dynamics; perceived function of the prenatal record; and expert opinions. External contextual factors included concerns about user buy-in; health system capacities; and pressures from public health stakeholders. Our recommendations highlight the need for: broader stakeholder involvement and explicit use of decision-support strategies to support the revision process; a national template of evidence-informed changes that can be used across jurisdictions; consideration of both clinical and surveillance functions of the prenatal record; and dissemination plans to communicate prenatal record modifications.

CONCLUSIONS

Decision-making related to prenatal record content involves a negotiated effort to balance research evidence with the needs and preferences of prenatal care providers, health system capacities as well as population health priorities. The development of a national template for prenatal records would reduce unnecessary duplication of PRC work and enhance the consistency of prenatal care delivery and perinatal surveillance data across Canada.

Appraisal tools for clinical practice guidelines: a systematic review

INTRODUCTION

Clinical practice guidelines can improve healthcare processes and patient outcomes, but are often of low quality. Guideline appraisal tools aim to help potential guideline users in assessing guideline quality. We conducted a systematic review of publications describing guideline appraisal tools in order to identify and compare existing tools.

METHODS

Among others we searched MEDLINE, EMBASE and the Cochrane Database of Systematic Reviews from 1995 to May 2011 for relevant primary and secondary publications. We also handsearched the reference lists of relevant publications. On the basis of the available literature we firstly generated 34 items to be used in the comparison of appraisal tools and grouped them into thirteen quality dimensions. We then extracted formal characteristics as well as questions and statements of the appraisal tools and assigned them to the items.

RESULTS

We identified 40 different appraisal tools. They covered between three and thirteen of the thirteen possible quality dimensions and between three and 29 of the possible 34 items. The main focus of the appraisal tools were the quality dimensions "evaluation of evidence" (mentioned in 35 tools; 88%), "presentation of guideline content" (34 tools; 85%), "transferability" (33 tools; 83%), "independence" (32 tools; 80%), "scope" (30 tools; 75%), and "information retrieval" (29 tools; 73%). The quality dimensions "consideration of different perspectives" and "dissemination, implementation and evaluation of the guideline" were covered by only twenty (50%) and eighteen tools (45%) respectively.

CONCLUSIONS

Most guideline appraisal tools assess whether the literature search and the evaluation, synthesis and presentation of the evidence in guidelines follow the principles of evidence-based medicine. Although conflicts of interest and norms and values of guideline developers, as well as patient involvement, affect the trustworthiness of guidelines, they are currently insufficiently considered. Greater focus should be placed on these issues in the further development of guideline appraisal tools.

The process of developing evidence-based guidance in medicine and public health: a qualitative study of views from the inside

Background

There has been significant investment in developing guidelines to improve clinical and public health practice. Though much is known about the processes of evidence synthesis and evidence-based guidelines implementation, we know little about how evidence presented to advisory groups is interpreted and used to form practice recommendations or what happens where evidence is lacking. This study investigates how members of advisory groups of NICE (National Institute of Health and Clinical Excellence) conceptualize evidence and experience the process.

Methods

Members of three advisory groups for acute physical, mental and public health were interviewed at the beginning and end of the life of the group. Seventeen were interviewed at both time points; five were interviewed just once at time one; and 17 were interviewed only once after guidance completion. Using thematic and content analysis, interview transcripts were analysed to identify the main themes.

Results

Three themes were identified:

1. What is the task? Different members conceptualized the task differently; some emphasized the importance of evidence at the top of the quality hierarchy while others emphasized the importance of personal experience.

2. Who gets heard? Managing the diversity of opinion and vested interests was a challenge for the groups; service users were valued and as was the importance of fostering good working relationships between group members.

3. What is the process? Group members valued debate and recognized the need to marshal discussion; most members were satisfied with the process and output.

Conclusions

Evidence doesn’t form recommendations on its own, but requires human judgement. Diversity of opinion within advisory groups was seen as key to making well-informed judgments relevant to forming recommendations. However, that diversity can bring tensions in the evaluation of evidence and its translation into practice recommendations.

Human computation as a new method for evidence-based knowledge transfer in Web-based guideline development groups: proof of concept randomized controlled trial

BACKGROUND

Guideline developers use different consensus methods to develop evidence-based clinical practice guidelines. Previous research suggests that existing guideline development techniques are subject to methodological problems and are logistically demanding. Guideline developers welcome new methods that facilitate a methodologically sound decision-making process. Systems that aggregate knowledge while participants play a game are one class of human computation applications. Researchers have already proven that these games with a purpose are effective in building common sense knowledge databases.

OBJECTIVE

We aimed to evaluate the feasibility of a new consensus method based on human computation techniques compared to an informal face-to-face consensus method.

METHODS

We set up a randomized design to study 2 different methods for guideline development within a group of advanced students completing a master of nursing and obstetrics. Students who participated in the trial were enrolled in an evidence-based health care course. We compared the Web-based method of human-based computation (HC) with an informal face-to-face consensus method (IC). We used 4 clinical scenarios of lower back pain as the subject of the consensus process. These scenarios concerned the following topics: (1) medical imaging, (2) therapeutic options, (3) drugs use, and (4) sick leave. Outcomes were expressed as the amount of group (dis)agreement and the concordance of answers with clinical evidence. We estimated within-group and between-group effect sizes by calculating Cohen's d. We calculated within-group effect sizes as the absolute difference between the outcome value at round 3 and the baseline outcome value, divided by the pooled standard deviation. We calculated between-group effect sizes as the absolute difference between the mean change in outcome value across rounds in HC and the mean change in outcome value across rounds in IC, divided by the pooled standard deviation. We analyzed statistical significance of within-group changes between round 1 and round 3 using the Wilcoxon signed rank test. We assessed the differences between the HC and IC groups using Mann-Whitney U tests. We used a Bonferroni adjusted alpha level of .025 in all statistical tests. We performed a thematic analysis to explore participants' arguments during group discussion. Participants completed a satisfaction survey at the end of the consensus process.

RESULTS

Of the 135 students completing a master of nursing and obstetrics, 120 participated in the experiment. We formed 8 HC groups (n=64) and 7 IC groups (n=56). The between-group comparison demonstrated that the human computation groups obtained a greater improvement in evidence scores compared to the IC groups, although the difference was not statistically significant. The between-group effect size was 0.56 (P=.30) for the medical imaging scenario, 0.07 (P=.97) for the therapeutic options scenario, and 0.89 (P=.11) for the drug use scenario. We found no significant differences in improvement in the degree of agreement between HC and IC groups. Between-group comparisons revealed that the HC groups showed greater improvement in degree of agreement for the medical imaging scenario (d=0.46, P=.37) and the drug use scenario (d=0.31, P=.59). Very few evidence arguments (6%) were quoted during informal group discussions.

CONCLUSIONS

Overall, the use of the IC method was appropriate as long as the evidence supported participants' beliefs or usual practice, or when the availability of the evidence was sparse. However, when some controversy about the evidence existed, the HC method outperformed the IC method. The findings of our study illustrate the importance of the choice of the consensus method in guideline development. Human computation could be an acceptable methodology for guideline development specifically for scenarios in which the evidence shows no resonance with participants' beliefs. Future research is needed to confirm the results of this study and to establish practical significance in a controlled setting of multidisciplinary guideline panels during real-life guideline development.

Developing clinical practice guidelines: target audiences, identifying topics for guidelines, guideline group composition and functioning and conflicts of interest

Clinical practice guidelines are one of the foundations of efforts to improve health care. In 1999, we authored a paper about methods to develop guidelines. Since it was published, the methods of guideline development have progressed both in terms of methods and necessary procedures and the context for guideline development has changed with the emergence of guideline clearing houses and large scale guideline production organisations (such as the UK National Institute for Health and Clinical Excellence). It therefore seems timely to, in a series of three articles, update and extend our earlier paper. In this first paper we discuss: the target audience(s) for guidelines and their use of guidelines; identifying topics for guidelines; guideline group composition (including consumer involvement) and the processes by which guideline groups function and the important procedural issue of managing conflicts of interest in guideline development.

The contribution of advisory committees and public involvement to large studies: case study

BACKGROUND

Many large studies have complex advisory committee structures, yet there is no empirical evidence regarding their optimal composition, scope and contribution. The aim of this study was to inform the committee and advice infrastructure for future research studies.

METHODS

In the context of a five-year study funded by the UK National Institute for Health Research, three advisory committees were formed. In addition, advice was obtained from individual experts. All recommendations received in the start-up phase (first seven months) of the study were recorded, along with the decision about implementation of the recommendation. A particular focus was on the impact of public involvement.

RESULTS

A total of 172 recommendations were made, including 70 from 20 individual experts. The recommendations were grouped into five emergent themes: Scientific, Pragmatic, Resources, Committee and Collaboration. Most recommendations related to strengthening existing components or adding new components to the study protocol. Very few recommendations either proposed removing study components or contradicted other recommendations. Three 'implementation criteria' were identified: scientific value, pragmatic feasibility, and paradigmatic consistency. 103 (60%) of recommendations were implemented and 25 (15%) were not implemented. The benefits identified by the research team were improved quality and confidence, and the costs were increased cognitive demands, protocol revision time, and slower progress.

CONCLUSIONS

The findings are discussed in the context of the wider literature on public involvement in research. Six recommendations are identified. First, have a clear rationale for each advisory committee expressed as terms of reference, and consider the best balance between committees and individual consultation with experts. Second, an early concern of committees is inter-committee communication, so consider cross-representation and copying minutes between committees. Third, match the scope of advisory committees to the study, with a less complex advisory structure for studies with more finalised designs. Fourth, public involvement has a mixed impact, and relies on relationships of trust, which take time to develop. Fifth, carefully consider the match between the scientific paradigm applied in the study and the contribution of different types of knowledge and expertise, and how this will impact on possibilities for taking on advice. Finally, responding to recommendations uses up research team resources, and the costs can be reduced by using the three implementation criteria.