Utility of RAND/UCLA appropriateness method in validating multiple-choice questions on ECG

OBJECTIVES

This study aimed to investigate the utility of the RAND/UCLA appropriateness method (RAM) in validating expert consensus-based multiple-choice questions (MCQs) on electrocardiogram (ECG).

METHODS

According to the RAM user's manual, nine panelists comprising various experts who routinely handle ECGs were asked to reach a consensus in three phases: a preparatory phase (round 0), an online test phase (round 1), and a face-to-face expert panel meeting (round 2). In round 0, the objectives and future timeline of the study were elucidated to the nine expert panelists with a summary of relevant literature. In round 1, 100 ECG questions prepared by two skilled cardiologists were answered, and the success rate was calculated by dividing the number of correct answers by 9. Furthermore, the questions were stratified into "Appropriate," "Discussion," or "Inappropriate" according to the median score and interquartile range (IQR) of appropriateness rating by nine panelists. In round 2, the validity of the 100 ECG questions was discussed in an expert panel meeting according to the results of round 1 and finally reassessed as "Appropriate," "Candidate," "Revision," and "Defer."

RESULTS

In round 1 results, the average success rate of the nine experts was 0.89. Using the median score and IQR, 54 questions were classified as " Discussion." In the expert panel meeting in round 2, 23% of the original 100 questions was ultimately deemed inappropriate, although they had been prepared by two skilled cardiologists. Most of the 46 questions categorized as "Appropriate" using the median score and IQR in round 1 were considered "Appropriate" even after round 2 (44/46, 95.7%).

CONCLUSIONS

The use of the median score and IQR allowed for a more objective determination of question validity. The RAM may help select appropriate questions, contributing to the preparation of higher-quality tests.

Comprehensive Spinal Tuberculosis Score: A Clinical Guide for the Management of Thoracolumbar Spinal Tuberculosis

STUDY DESIGN

A newly proposed scoring tool was designed to assist in the clinical management of adult thoracolumbar spinal tuberculosis (TB).

PURPOSE

To formulate a comprehensive yet simple scoring tool to guide decision-making in the management of adult thoracolumbar spinal TB.

OVERVIEW OF LITERATURE

Spine surgeons have differing consensus in defining the threshold grade for clinico-radiological parameters when deciding between operative or conservative treatment for adult thoracolumbar spinal TB. Currently, the void in decision-making from the lack of well-defined guidelines is compensated by the surgeon's experience in treating these patients. To the best of our knowledge, no scoring system holistically integrates multiple facets of spinal TB to guide clinical decision-making.

METHODS

The RAND/University of California, Los Angeles appropriateness method was employed among an expert panel of 10 spine surgeons from four apex tertiary care centers. Vital characteristics that independently influenced treatment decisions in spinal TB were identified, and a scoring tool was formulated. Points were assigned for each component based on their severity. The cutoff scores to guide clinical management were determined from the receiver operating characteristic curve based on the retrospective records of 151 patients treated operatively or non-operatively with improved functional outcomes at the 1-year follow-up.

RESULTS

The components of the comprehensive spinal TB score (CSTS) are pain, kyphosis angle, vertebral destruction, and neurological status. A score classification of <5.5, 5.5-6.5, and >6.5 was established to guide the patient toward conservative, conservative/ operative, and operative management, respectively.

CONCLUSIONS

The CSTS was designed to reflect the essential indicators of mechanical stability, neurological stability, and disease process stabilization in spinal TB. The scoring tool is devised to be practical and serve as a common language in the spine community to facilitate discussions and decision-making in thoracolumbar spinal TB. The validity, reliability, and reproducibility of this tool must be assessed through multicenter long-term studies.

Planning and Reporting Effective Web-Based RAND/UCLA Appropriateness Method Panels: Literature Review and Preliminary Recommendations

BACKGROUND

The RAND/UCLA Appropriateness Method (RAM), a variant of the Delphi Method, was developed to synthesize existing evidence and elicit the clinical judgement of medical experts on the appropriate treatment of specific clinical presentations. Technological advances now allow researchers to conduct expert panels on the internet, offering a cost-effective and convenient alternative to the traditional RAM. For example, the Department of Veterans Affairs recently used a web-based RAM to validate clinical recommendations for de-intensifying routine primary care services. A substantial literature describes and tests various aspects of the traditional RAM in health research; yet we know comparatively less about how researchers implement web-based expert panels.

OBJECTIVE

The objectives of this study are twofold: (1) to understand how the web-based RAM process is currently used and reported in health research and (2) to provide preliminary reporting guidance for researchers to improve the transparency and reproducibility of reporting practices.

METHODS

The PubMed database was searched to identify studies published between 2009 and 2019 that used a web-based RAM to measure the appropriateness of medical care. Methodological data from each article were abstracted. The following categories were assessed: composition and characteristics of the web-based expert panels, characteristics of panel procedures, results, and panel satisfaction and engagement.

RESULTS

Of the 12 studies meeting the eligibility criteria and reviewed, only 42% (5/12) implemented the full RAM process with the remaining studies opting for a partial approach. Among those studies reporting, the median number of participants at first rating was 42. While 92% (11/12) of studies involved clinicians, 50% (6/12) involved multiple stakeholder types. Our review revealed that the studies failed to report on critical aspects of the RAM process. For example, no studies reported response rates with the denominator of previous rounds, 42% (5/12) did not provide panelists with feedback between rating periods, 50% (6/12) either did not have or did not report on the panel discussion period, and 25% (3/12) did not report on quality measures to assess aspects of the panel process (eg, satisfaction with the process).

CONCLUSIONS

Conducting web-based RAM panels will continue to be an appealing option for researchers seeking a safe, efficient, and democratic process of expert agreement. Our literature review uncovered inconsistent reporting frameworks and insufficient detail to evaluate study outcomes. We provide preliminary recommendations for reporting that are both timely and important for producing replicable, high-quality findings. The need for reporting standards is especially critical given that more people may prefer to participate in web-based rather than in-person panels due to the ongoing COVID-19 pandemic.

Appropriate use criteria for ancillary diagnostic testing in dermatopathology: New recommendations for 11 tests and 220 clinical scenarios from the American Society of Dermatopathology Appropriate Use Criteria Committee

BACKGROUND

Appropriate use criteria (AUC) provide patient-centered physician guidance in test selection. An initial set of AUC was reported by the American Society of Dermatopathology (ASDP) in 2018. AUC reflect evidence collected at single timepoints and may be affected by evolving evidence and experience. The objective of this study was to update and expand AUC for selected tests.

METHODS

RAND/UCLA (RAND Corporation [Santa Monica, CA]/University of California Los Angeles) methodology used includes the following: (a) literature review; (b) review of previously rated tests and previously employed clinical scenarios; (c) selection of previously rated tests for new ratings; (d) development of new clinical scenarios; (e) selection of additional tests; (f) three rating rounds with feedback and group discussion after rounds 1 and 2.

RESULTS

For 220 clinical scenarios comprising lymphoproliferative (light chain clonality), melanocytic (comparative genomic hybridization, fluorescence in situ hybridization, reverse transcription polymerase chain reaction, telomerase reverse transcriptase promoter), vascular disorders (MYC), and inflammatory dermatoses (periodic acid-Schiff, Gömöri methenamine silver), consensus by panel raters was reached in 172 of 220 (78%) scenarios, with 103 of 148 (70%) rated "usually appropriate" or "rarely appropriate" and 45 of 148 (30%), "appropriateness uncertain."

LIMITATIONS

The study design only measures appropriateness. Cost, availability, test comparison, and additional clinical considerations are not measured. The possibility that the findings of this study may be influenced by the inherent biases of the dermatopathologists involved in the study cannot be excluded.

CONCLUSIONS

AUC are reported for selected diagnostic tests in clinical scenarios that occur in dermatopathology practice. Adhering to AUC may reduce inappropriate test utilization and improve healthcare delivery.

Management of Infants with Brief Resolved Unexplained Events (BRUE) and Apparent Life-Threatening Events (ALTE): A RAND/UCLA Appropriateness Approach

Unexpected events of breath, tone, and skin color change in infants are a cause of considerable distress to the caregiver and there is still debate on their appropriate management. The aim of this study is to survey the trend in prevention, decision-making, and management of brief resolved unexplained events (BRUE)/apparent life-threatening events (ALTE) and to develop a shared protocol among hospitals and primary care pediatricians regarding hospital admission criteria, work-up and post-discharge monitoring of patients with BRUE/ALTE. For the study purpose, a panel of 54 experts was selected to achieve consensus using the RAND/UCLA appropriateness method. Twelve scenarios were developed: one addressed to primary prevention of ALTE and BRUE, and 11 focused on hospital management of BRUE and ALTE. For each scenario, participants were asked to rank each option from ‘1’ (extremely inappropriate) to ‘9’ (extremely appropriate). Results derived from panel meeting and discussion showed several points of agreement but also disagreement with different opinion emerged and the need of focused education on some areas. However, by combining previous recommendations with expert opinion, the application of the RAND/UCLA appropriateness permitted us to drive pediatricians to reasoned and informed decisions in term of evaluation, treatment and follow-up of infants with BRUE/ALTE, reducing inappropriate exams and hospitalisation and highlighting priorities for educational interventions.

A Spanish Consensus on the Use of Safinamide for Parkinson's Disease in Clinical Practice

Safinamide is an approved drug for the treatment of fluctuations in Parkinson's disease (PD). Scarce data are available on its use in clinical practice. A group of Spanish movement disorders specialists was convened to review the use of safinamide across different clinical scenarios that may guide neurologists in clinical practice. Eight specialists with recognized expertise in PD management elaborated the statements based on available evidence in the literature and on their clinical experience. The RAND/UCLA method was carried, with final conclusions accepted after a 2-round modified Delphi process. Higher level of agreement between panellists was reached for the following statements. Safinamide significantly improves mean daily ON time without troublesome dyskinesias [corrected]. Adjunctive treatment with safinamide is associated with motor improvements in patients with mid-to-late PD. The efficacy of safinamide on motor fluctuations is maintained at long-term, with no increase over time in dyskinesias severity. The clinical benefits of safinamide on pain and depression remain unclear. Safinamide presents a similar incidence of adverse events compared with placebo. The efficacy and safety of safinamide shown in the pivotal clinical trials are reproduced in clinical practice, with improvement of parkinsonian symptoms, decrease of daily OFF time, control of dyskinesias at the long term, and good tolerability and safety.

Perioperative dental screening and treatment in patients undergoing cardiothoracic surgery and interventional cardiovascular procedures. A consensus report based on RAND/UCLA methodology

AIM

To reach a consensus on a consistent strategy to adopt when screening patients for dental/periodontal infections and on the feasibility of providing dental treatment before cardiothoracic surgery, cardiovascular surgery or other cardiovascular invasive procedures.

METHODOLOGY

A panel of experts from six Italian scientific societies was created. The deliberations of the panel were based on the RAND method. From an initial systematic literature review, it became clear that a consensually validated protocol for the reproducible dental screening of patients awaiting cardiac interventions was considered mandatory by professionals with expertise in the dental, cardiologic and cardiac surgery areas. However, a systematic review also concluded that the treatment options to be provided, their prognosis and timing in relation to the physical condition of patients, had never been defined. Following the systematic review, several fundamental questions were generated. The panel was divided into two working groups each of which produced documents that addressed the topic and which were subsequently used to generate a questionnaire. Each member of the panel completed the questionnaire independently, and then, a panel discussion was held to reach a consensus on how best to manage patients with dental/periodontal infections who were awaiting invasive cardiac procedures.

RESULTS

A high level of agreement was reached regarding all the items on the questionnaire, and each of the clinical questions formulated were answered. Three tables were created which can be used to generate a useful tool to provide standardized dental/periodontal screening of patients undergoing elective cardiovascular interventions and to summarize both the possible oral and cardiovascular conditions of the patient and the timing available for the procedures considered.

CONCLUSIONS

Upon publication of this consensus document, the dissemination of the information to a wide dental and cardiac audience should commence. The authors hope that this consensus will become a model for the development of a dedicated protocol, ideally usable by heart and dental teams in the pre-interventional preparation phase.

Development of a core set of quality indicators for paediatric primary care practices in Europe, COSI-PPC-EU

Paediatric ambulatory healthcare systems in Europe are, because of historical reasons, diverse and show strikingly different outcomes. All across Europe, the benchmarking of structures, processes and outcomes could reveal opportunities for improving Paediatric Primary Care (PPC). The aim of this study was to develop a set of Quality Indicators (QIs) to assess and monitor PPC in Europe. In a three-step process, we used the available external evidence and European expert consensus in a modified RAND/UCLA Appropriateness Method (RAM) to develop an indicator set. (1) A broad literature and online research of published QI and guidelines yielded an inventory of 1516 QI. (2) A collaborative panel of paediatric senior experts from the European Academy of Paediatrics (EAP) and the European Confederation of Primary Care Paediatricians (ECPCP) from 15 European countries participated in a first consensus process to reduce the initial indicator inventory by eliminating not PPC-focused indicators and duplicates. (3) In a second consensus process, the panel rated the QI regarding validity and feasibility. The final QI set "COSI-PPC-EU" consists of 42 indicators in five categories of PPC: (A) health promotion/prevention/screening (13 QI), (B) acute care (9 QI), (C) chronic care (8 QI), (D) practice management (3 QI) and (E) patient safety (9 QI).

CONCLUSION

COSI-PPC-EU represents a consented set of a limited number of valid quality indicators for the application in paediatric primary care in different healthcare systems throughout Europe. What is Known: • Paediatric ambulatory healthcare systems in Europe are diverse and show strikingly different outcomes. • There are known gaps in quality performance measures of paediatric primary care in Europe. Pre-existing sets of quality indicators are predominantly limited to national populations, specific diseases and hospital care. What is New: • A set of 42 quality indicators for primary paediatric care in Europe was developed in a multi-country collaborative effort. The method combined a systematic literature review and a consensus process among European paediatric experts. • The quality indicator set can facilitate quality improvement of PPC. After studying the feasibility, providers can use COSI-PPC-EU to monitor, compare and improve performance of practices, regions and countries.

One consensual depression diagnosis tool to serve many countries: a challenge! A RAND/UCLA methodology

OBJECTIVE

From a systematic literature review (SLR), it became clear that a consensually validated tool was needed by European General Practitioner (GP) researchers in order to allow multi-centred collaborative research, in daily practice, throughout Europe. Which diagnostic tool for depression, validated against psychiatric examination according to the DSM, would GPs select as the best for use in clinical research, taking into account the combination of effectiveness, reliability and ergonomics? A RAND/UCLA, which combines the qualities of the Delphi process and of the nominal group, was used. GP researchers from different European countries were selected. The SLR extracted tools were validated against the DSM. The Youden index was used as an effectiveness criterion and Cronbach's alpha as a reliability criterion. Ergonomics data were extracted from the literature. Ergonomics were tested face-to-face.

RESULTS

The SLR extracted 7 tools. Two instruments were considered sufficiently effective and reliable for use: the Hospital Anxiety and Depression Scale and the Hopkins Symptoms Checklist-25 (HSCL-25). After testing face-to-face, HSCL-25 was selected. A multicultural consensus on one diagnostic tool for depression was obtained for the HSCL-25. This tool will provide the opportunity to select homogeneous populations for European collaborative research in daily practice.

Referring Parkinson's disease patients for deep brain stimulation: a RAND/UCLA appropriateness study

In 2005, a European expert panel developed and validated an electronic tool to support the appropriate referral of patients with Parkinson’s disease (PD) for the consideration of deep brain stimulation (DBS). Since new evidence has become available over the last decade an update of the tool is necessary. A world-wide expert panel (71 neurologists and 11 neurosurgeons) used the RAND/UCLA Appropriateness Method to assess the appropriateness of referral for 1296 scenarios (9-point scale). Scenarios were permutations of 8 clinical variables relevant to the decision of referral. Appropriateness of referral was calculated on the basis of the median score and the extent of agreement. Compared to 2005, the impact of clinical variables on the appropriateness of referral was similar for severity of on–off fluctuations, dyskinesias and refractory tremor (positive association, p < 0.001), and cognitive impairment (negative association, p < 0.001). A relatively stronger negative impact was seen for levodopa-unresponsive gait and balance disturbances as well as older age, the latter most likely due to a higher cut-off value (75 versus 70 years in the previous study). The impact of PD duration on the appropriateness of referral was less pronounced than in 2005. The contribution of the newly included variable ‘non-motor side effects of anti-PD medication’ was very modest. Based on these results the panel produced new recommendations on the appropriateness of referral for the evaluation of DBS in PD patients. Differences from the previous study reflect the new clinical evidence, particularly related to the use of DBS in an earlier stage of PD. The validation of the updated recommendations is in progress.