Small-group judgment methods for determining resource-based relative values

Human computation as a new method for evidence-based knowledge transfer in Web-based guideline development groups: proof of concept randomized controlled trial

BACKGROUND

Guideline developers use different consensus methods to develop evidence-based clinical practice guidelines. Previous research suggests that existing guideline development techniques are subject to methodological problems and are logistically demanding. Guideline developers welcome new methods that facilitate a methodologically sound decision-making process. Systems that aggregate knowledge while participants play a game are one class of human computation applications. Researchers have already proven that these games with a purpose are effective in building common sense knowledge databases.

OBJECTIVE

We aimed to evaluate the feasibility of a new consensus method based on human computation techniques compared to an informal face-to-face consensus method.

METHODS

We set up a randomized design to study 2 different methods for guideline development within a group of advanced students completing a master of nursing and obstetrics. Students who participated in the trial were enrolled in an evidence-based health care course. We compared the Web-based method of human-based computation (HC) with an informal face-to-face consensus method (IC). We used 4 clinical scenarios of lower back pain as the subject of the consensus process. These scenarios concerned the following topics: (1) medical imaging, (2) therapeutic options, (3) drugs use, and (4) sick leave. Outcomes were expressed as the amount of group (dis)agreement and the concordance of answers with clinical evidence. We estimated within-group and between-group effect sizes by calculating Cohen's d. We calculated within-group effect sizes as the absolute difference between the outcome value at round 3 and the baseline outcome value, divided by the pooled standard deviation. We calculated between-group effect sizes as the absolute difference between the mean change in outcome value across rounds in HC and the mean change in outcome value across rounds in IC, divided by the pooled standard deviation. We analyzed statistical significance of within-group changes between round 1 and round 3 using the Wilcoxon signed rank test. We assessed the differences between the HC and IC groups using Mann-Whitney U tests. We used a Bonferroni adjusted alpha level of .025 in all statistical tests. We performed a thematic analysis to explore participants' arguments during group discussion. Participants completed a satisfaction survey at the end of the consensus process.

RESULTS

Of the 135 students completing a master of nursing and obstetrics, 120 participated in the experiment. We formed 8 HC groups (n=64) and 7 IC groups (n=56). The between-group comparison demonstrated that the human computation groups obtained a greater improvement in evidence scores compared to the IC groups, although the difference was not statistically significant. The between-group effect size was 0.56 (P=.30) for the medical imaging scenario, 0.07 (P=.97) for the therapeutic options scenario, and 0.89 (P=.11) for the drug use scenario. We found no significant differences in improvement in the degree of agreement between HC and IC groups. Between-group comparisons revealed that the HC groups showed greater improvement in degree of agreement for the medical imaging scenario (d=0.46, P=.37) and the drug use scenario (d=0.31, P=.59). Very few evidence arguments (6%) were quoted during informal group discussions.

CONCLUSIONS

Overall, the use of the IC method was appropriate as long as the evidence supported participants' beliefs or usual practice, or when the availability of the evidence was sparse. However, when some controversy about the evidence existed, the HC method outperformed the IC method. The findings of our study illustrate the importance of the choice of the consensus method in guideline development. Human computation could be an acceptable methodology for guideline development specifically for scenarios in which the evidence shows no resonance with participants' beliefs. Future research is needed to confirm the results of this study and to establish practical significance in a controlled setting of multidisciplinary guideline panels during real-life guideline development.

Surgical outcomes in American Indian veterans: a closer look

BACKGROUND

American Indian/Alaska Native (AI/ANs) male veterans have considerably higher postoperative mortality rates than their Caucasian counterparts, but similar postoperative morbidity rates even after adjusting for major preoperative risk factors. This study seeks to explain the discrepancy in morbidity and mortality.

STUDY DESIGN

We obtained data from the Veterans Affairs National Surgical Quality Improvement Program on major, noncardiac, surgical procedures performed from 1991 to 2002 for all AI/AN men (n = 2,155), and a random sample of Caucasian men (n = 2,264), matched by site. We compared the number and types of postoperative complications and mortality rates for those patients in whom complications developed. We also examined complication and mortality rates by whether they occurred after hospital discharge, or by specific type of surgical procedure. Preoperative risk factors were assessed in patients who died. Chi-square or Fisher's exact tests were used for all comparisons.

RESULTS

AI/ANs and Caucasians did not differ by number of complications but Caucasian patients had considerably higher rates for three specific complications. There was no difference in deaths after discharge or in mortality rates after specific surgical procedures. The groups differed considerably in the types of procedures performed. Among patients who died, three preoperative risk factors, ie, hemiplegia, diabetes, and wound infection, occurred more frequently among AI/AN than Caucasian veterans.

CONCLUSIONS

We cannot fully explain higher postoperative mortality rates experienced by AI/AN relative to Caucasian veterans after examining complications, types of procedures, and other relevant factors. AI/ANs with certain preoperative risk factors can be vulnerable to 30-day postoperative mortality and benefit from closer postoperative surveillance.

Lessons learned from advanced practice nursing payment

For more than 25 years, advanced practice nurses have been incrementally included as a part of the health care financing structure. Following physician payment revisions at the federal level, advanced practice nurses were overtly recognized as Medicare providers and have participated in the establishment of current procedural terminology codes and the subsequent relative work values associated with payment. Success in this regard has been the result of business, political, and policy savvy that has important lessons for moving forward in any health care restructuring for both nurses and advanced practice nurses. Principles of valuing nurse work, time, and intensity in the Resource-Based Relative Value Scale are discussed with implications for future opportunities of measuring nursing work and any potential relationship to quality outcomes of care.

Inequity in the price of physician activity across surgical procedures

OBJECTIVES

A rational payment system is being sought in Japanese health care-one that accurately reflects the required time and the level of technical difficulty when valuing physician activity. The objective of this study is to examine the current surgical payment system in Japan by clarifying the hourly values allocated to physician activity.

METHODS

This study focused on the 22 surgical procedures most frequently registered in our study database of administrative data gathered from 11 teaching hospitals in Japan. The current fee-for-service reimbursement system does not formally define which cost components surgical fees cover. It was therefore necessary for us to examine directly each reimbursement item to determine which component it represented. Next we examined the current system from the following viewpoints: (1) variation in the hourly values allocated to physician activity, for an individual surgeon or a surgical team, among types of surgery by using the actual data; (2) the association between the hourly values and the operation time or the level of technical difficulty.

RESULTS

The hourly values allocated to physician activity were low (US dollars 61.0 and 121.5 per surgeon: means of case 1 and case 2 estimations). The hourly values varied inequitably among types of surgery (from US dollars -28 to 237 and from US dollars 6 to 328: ranges in the case 1 and case 2 estimations). When long surgeries were excluded, shorter surgeries tended to have higher hourly values. The association between the hourly values and the difficulty level was less clear and their variation was large even at the same difficulty level.

CONCLUSION

In the current payment system, the surgical fee is deemed to include fee for physician activity as well as materials, equipment and so on. To develop a rational payment system, first, the scope of the surgical fee and that of the physician activity fee should be separated and clearly defined. Second, the latter should be modeled to reflect the manpower volume and the level of technical difficulty needed for each surgical procedure. Third, fees should be set by utilizing the cost estimates with empirical data.

Surgical Morbidity and Mortality among American Indian and Alaska Native Veterans: A Comparative Analysis

BACKGROUND

Few studies have examined surgical risk factors and outcomes in American Indians and Alaska Natives (AI/ANs). My colleagues and I sought to determine if prevalence of preoperative risk factors for morbidity and mortality differed between male AI/AN and Caucasian surgical patients, and to determine if AI/ANs had an increased risk of surgical morbidity or mortality.

STUDY DESIGN

We obtained data from the Veterans Affairs National Surgical Quality Improvement Program on major, noncardiac, surgical procedures performed between 1991 and 2002 for all AI/AN men (n = 2,155) and a random sample of Caucasian men (n = 2,264), matched by facility. Chi-square and t-test analyses were used to assess differences in preoperative risk factors between the two groups. Logistic regression was used to determine whether AI/AN race was independently associated with 30-day morbidity (defined as 1 or more of 21 postoperative complications) or 30-day all cause mortality after adjustment for major risk factors.

RESULTS

Prevalence of major preoperative risk factors for morbidity and mortality often differed between the groups. Compared with Caucasians, AI/AN race did not predict morbidity (adjusted odds ratio, 0.92; 95% CI, 0.75-1.13), but AI/ANs were at higher risk for 30-day all cause postoperative mortality (adjusted odds ratio, 1.56; 95% CI, 1.04-2.35).

CONCLUSIONS

Our results add postoperative mortality to health disparities experienced by AI/ANs. Future research should be conducted to identify other factors that contribute to this disparity.

The development of evidence-based guidelines for over-the-counter treatment of vulvovaginal candidiasis

OBJECTIVE

The purpose of this study was to develop evidence-based guidelines for over-the-counter (OTC) treatment of vulvovaginal candidiasis with non-prescription anti-fungal medicines purchased from community pharmacies.

METHOD

A multidisciplinary guideline development group was recruited from the locality where the guidelines were to be tested. A Nominal Group Technique (NGT) was used to achieve formal consensus within the group regarding the issues that the guidelines would address. Guideline recommendations were developed from the results of two systematic literature reviews that assessed which symptoms were predictive of vulvovaginal candidiasis (using data from epidemiological studies) and estimated the relative effectiveness of oral and intra-vaginal anti-fungals using data from randomised controlled trials.

MAIN OUTCOME MEASURES

Evidence-based guideline recommendations. The guideline statements were linked to the evidence using a standard hierarchy.

RESULTS

The guideline development group met four times. The use of NGT was an effective way of achieving consensus on guideline content. Two systematic reviews carried out as part of the guideline development process identified evidence for the guidelines on the efficacy of OTC treatments and symptoms suggestive of vulvovaginal candidiasis. The guideline recommendations were presented as a booklet and a laminated algorithm. In summary, the guidelines highlighted symptoms suggestive of vulvovaginal candidiasis, and symptoms associated with other vaginal conditions that should be referred to the GP. The guidelines stated that oral treatment and intra-vaginal treatment are equally effective, and that selection of an anti-fungal should be based upon safety, cost and patient preference. Many of the recommendations were influenced by OTC license restrictions of each antifungal product. Contra-indications to, and special precautions with, antifungals were also listed. In addition, the guidelines stated that the male sexual partner does not require treatment unless symptomatic.

CONCLUSION

There is sufficient evidence available to develop evidence-based guidelines for the treatment of vulvovaginal candidiasis in the community pharmacy setting. The NGT is a useful component in the guideline development process.

The implications of plan design options for Medicare beneficiaries

To make an informed selection between traditional Medicare and a Medicare managed care plan, a consumer needs to understand the implications of choosing one over the other. What are the implications of plan design for care, cost, and patient autonomy? Consumers need information about these questions. However, a barrier to developing this consumer information is the lack of a consistent body of evidence. An intermediate step is to tap expert knowledge. The purpose of this study is to use expert consensus (across a spectrum of health care experts) to identify the implications of plan design. Experts were surveyed and the degree to which there is consensus provides an initial picture of what experts judge to be important to the consumer. The findings show that experts agree on several implications associated with choosing managed care over the traditional Medicare plan. They also agree that many of these attributes vary considerably across health plans.

Nurse practitioner services: content and relative work value

The resource-based relative value scale is used to quantify work for reimbursement of services in the Medicare Fee Schedule. This pilot study explored use of the resource-based relative value scale for services provided by nurse practitioners. Estimation of relative work values for office visits by nurse practitioners was consistent with the Medicare Fee Schedule. Content analysis revealed that nurse practitioners provide additional services including comprehensive patient evaluation and education and attendance to social factors. Future research is needed to examine systems that identify and reimburse nurse practitioners for their services.

Consensus development methods: a review of best practice in creating clinical guidelines

BACKGROUND

Although there is debate about the appropriate place of guidelines in clinical practice, guidelines can be seen as one way of assisting clinicians in decision-making. Given the likely diversity of opinion that any group of people may display when considering a topic, methods are needed for organising subjective judgements. Three principal methods (Delphi, nominal group technique, consensus development conference) exist which share the common objective of synthesising judgements when a state of uncertainty exists.

OBJECTIVES

To identify the factors that shape and influence the clinical guidelines that emerge from consensus development methods and to make recommendations about best practice in the use of such methods.

METHODS

Five electronic databases were searched: Medline (1966-1996), PsychLIT (1974-1996), Social Science Citation Index (1990-1996), ABI Inform and Sociofile. From the searches and reference lists of articles a total of 177 empirical and review articles were selected for review.

RESULTS

The output from consensus development methods may be affected by: the way the task is set (choice of cues, recognition of contextual cues, the focus of the task, the comprehensiveness of the scenarios); the selection of participants (choice of individuals, degree of homogeneity of the group, their background, their number); the selection and presentation of scientific information (format, extent to which its quality and content is assessed); the way any interaction is structured (number of rating rounds, ensuring equitable participation, physical environment for meetings); and the method of synthesising individual judgements (definition of agreement, rules governing outliers, method of mathematical aggregation).

CONCLUSIONS

Although a considerable amount of research has been carried out, many aspects have not been investigated sufficiently. For the time being at least, advice on those aspects has, therefore, to be based on the user's own commonsense and the experience of those who have used or participated in these methods. Even in the long term, some aspects will not be amenable to scientific study. Meanwhile, adherence to best practice will enhance the validity, reliability and impact of the clinical guidelines produced.

Minority populations and advance directives: insights from a focus group methodology

Numerous studies have shown almost uniformly positive opinions among patients and physicians regarding the concept of advance directives (either a healthcare proxy or living will). Several of these studies have also shown that the actual use of advance directives is significantly lower than this enthusiasm would suggest, but they have not explained the apparent discordance. Nor have researchers explained why members of minority groups are much less likely to complete advance directives than are white patients. In this study, we used a focus group methodology to examine the ways in which diverse populations of patients view the medical, philosophical, and practical issues surrounding advance directives. We were motivated by the significantly lower prevalence of advance directives among African-American and Hispanic patients at one urban teaching hospital (18% for Caucasians, 4% for African-Americans, and 2% for Hispanics). Our premise was that African-American and Hispanic populations, who have had higher rates of morbidity and mortality across numerous disease categories, and historically have had limited access to care and opportunities to discuss health concerns, may be more suspicious about the right of autonomy that an advance directive is designed to ensure.

Panellist consistency in the assessment of medical appropriateness

Where information about the appropriateness of a surgical procedure is lacking, expert panels have been used to establish guidelines for medical practitioners. Such a panel was convened to assess the appropriateness of percutaneous transluminal coronary angioplasty and coronary artery bypass graft surgery in the Netherlands. The panel, consisting of interventional cardiologists and cardiothoracic surgeons, used a modified Delphi process to rate 1126 clinical indications over two rounds. This article describes the degree of change in both agreement amongst members and in the appropriateness ratings over the two rounds, and examines the internal consistency of the ratings of individual panellists. Over the two rounds, agreement increased. Although most appropriateness ratings remained unchanged, there was significant movement from equivocal ratings to determinate ratings. While individual members showed some degree of inconsistency in their scoring, the panel as a whole scored very consistently. The observed changes in appropriateness were consistent with expectations, showing that the appropriateness method is used logically and consistently by panellists.

The resource-based relative value scale: methods, results and impacts on urology

Increased concern for rising health care costs in the United States has led to the passage of legislation to reform physician payment for Medicare services based on resource inputs. In January 1992 the Health Care Financing Administration began implementing the new law, which replaces the existing Medicare system of physician payment with a fee schedule based on the resource-based relative value scale (RBRVS). We summarize the methods and data used to derive the RBRVS for urology. A national random sample of 115 practicing urologists completed structured telephone surveys to provide ratings of physician time and work required before, during and after most frequently performed urological services. Subsequent survey cycles with urologists provided further refinement. Urologists then participated in a cross-specialty physician panel to link services from all specialties onto a common scale. This common scale was adjusted for geographic differences in practice overhead costs and malpractice insurance premiums. A monetary conversion factor, determined by the Health Care Financing Administration, was then applied to convert the RBRVS into a Medicare fee schedule. The merits and demerits of the scientific process used to develop and maintain the relative value scale are extensive. While statistically valid and reproducible, the study results have been altered in the political arena. The results and impacts of the new Medicare payment system on urology will be significant, although it is not yet clear how urological practice will be affected. Although faring better than most surgical specialties, urologists stand to lose approximately 8% of their Medicare income when the new fee schedule is fully implemented. There will be relative gains for evaluation and management services and losses for most invasive procedures.