Use of Persuasive Language in Communication of Risk during Prostate Cancer Treatment Consultations

BACKGROUND

Physician treatment preference may influence how risks are communicated in prostate cancer consultations. We identified persuasive language used when describing cancer prognosis, life expectancy, and side effects in relation to a physician's recommendation for aggressive (surgery/radiation) or nonaggressive (active surveillance/watchful waiting) treatment.

METHODS

A qualitative analysis was performed on transcribed treatment consultations of 40 men with low- and intermediate-risk prostate cancer across 10 multidisciplinary providers. Quotes pertaining to cancer prognosis, life expectancy, and side effects were randomized. Coders predicted physician treatment recommendations from isolated blinded quotes. Testing characteristics of consensus predictions against the physician's treatment recommendation were reported. Coders then identified persuasive strategies favoring aggressive/nonaggressive treatment for each quote. Frequencies of persuasive strategies favoring aggressive/nonaggressive treatment were reported. Logistic regression quantified associations between persuasive strategies and physician treatment recommendations.

RESULTS

A total of 496 quotes about cancer prognosis (n = 127), life expectancy (n = 51), and side effects (n = 318) were identified. The accuracy of predicting treatment recommendation based on individual quotes containing persuasive language (n = 256/496, 52%) was 91%. When favoring aggressive treatment, persuasive language downplayed side effect risks and amplified cancer risk (recurrence, progression, or mortality). Significant predictors (P < 0.05) of aggressive treatment recommendation included favorable side effect interpretation, downplaying side effects, and long time horizon for cancer risk due to longevity. When favoring nonaggressive treatment, persuasive language amplified side effect risks and downplayed cancer risk. Significant predictors of nonaggressive treatment recommendation included unfavorable side effect interpretation, favorable interpretation of cancer risk, and short time horizon for cancer risk due to longevity.

CONCLUSIONS

Physicians use persuasive language favoring their preferred treatment, regardless of whether their recommendation is appropriate.

IMPLICATIONS

Clinicians should quantify risk so patients can judge potential harm without solely relying on persuasive language.

HIGHLIGHTS

Physicians use persuasive language favoring their treatment recommendation when communicating risks of prostate cancer treatment, which may influence a patient's treatment choice.Coders predicted physician treatment recommendations based on isolated, randomized quotes about cancer prognosis, life expectancy, and side effects with 91% accuracy.Qualitative analysis revealed that when favoring nonaggressive treatment, physicians used persuasive language that amplified side effect risks and downplayed cancer risk. When favoring aggressive treatment, physicians did the opposite.Providers should be cognizant of using persuasive strategies and aim to provide quantified assessments of risk that are jointly interpreted with the patient so that patients can make evidence-based conclusions regarding risks without solely relying on persuasive language.

Variation in communication of side effects in prostate cancer treatment consultations

BACKGROUND

Effective communication of treatment side effects (SE) is critical for shared decision-making (SDM) in localized prostate cancer. We sought to qualitatively characterize how physicians communicate SE in consultations.

METHODS

We transcribed 50 initial prostate cancer treatment consultations across nine multidisciplinary providers (Urologists, Radiation Oncologists, Medical Oncologists) at our tertiary referral, academic center. Coders identified quotes describing SE and used an inductive approach to establish a hierarchy for granularity of communication: (1) not mentioned, (2) name only, (3) generalization("high"), (4) average incidence without timepoint, (5) average incidence with timepoint, and (6) precision estimate. We reported the most granular mode of communication for each SE throughout the consultation overall and across specialty and tumor risk.

RESULTS

Among consultations discussing surgery (n = 40), erectile dysfunction (ED) and urinary incontinence (UI) were omitted in 15% and 12%, not quantified (name only or generalization) in 47% and 30%, and noted as average incidence without timeline in 8% and 8%, respectively. In only 30% and 49% were ED and UI quantified with timeline (average incidence with timeline or precision estimate), respectively. Among consultations discussing radiation (n = 36), irritative urinary symptoms, ED, and other post-radiotherapy SE were omitted in 22%, 42%, and 64-67%, not quantified in 61%, 33%, and 23-28%, and noted as average incidence without timeline in 8%, 22%, and 6-8%, respectively. In only 3-8% were post-radiotherapy SE quantified with timeline. Specialty concordance (but not tumor risk) was associated with higher granularity of communication, though physicians frequently failed to quantify specialty-concordant SE.

CONCLUSIONS

SE was often omitted, not quantified, and/or lacked a timeline in treatment consultations in our sample. Physicians should articulate, quantify, and assign a timeline for SE to optimize SDM.

Do current pregnancy weight gain guidelines balance risks of adverse maternal and child health in a United States cohort?

BACKGROUND

The Institute of Medicine pregnancy weight gain guidelines were developed without evidence linking high weight gain to maternal cardiometabolic disease and child obesity. The upper limit of current recommendations may be too high for the health of the pregnant individual and child.

OBJECTIVES

The aim of this study was to identify the range of pregnancy weight gain for pregnancies within a normal body mass index (BMI) range that balances the risks of high and low weight gain by simultaneously considering 10 different health conditions.

METHODS

We used data from an United States prospective cohort study of nulliparae followed until 2 to 7 y postpartum (N = 2344 participants with a normal BMI). Pregnancy weight gain z-score was the main exposure. The outcome was a composite consisting of the occurrence of ≥1 of 10 adverse health conditions that were weighted for their seriousness. We used multivariable Poisson regression to relate weight gain z-scores with the weighted composite outcome.

RESULTS

The lowest risk of the composite outcome was at a pregnancy weight gain z-score of -0.6 SD (standard deviation) (equivalent to 13.1 kg at 40 wk). The weight gain ranges associated with no more than 5%, 10%, and 20% increase in risks were -1.0 to -0.2 SD (11.2-15.3 kg), -1.4 to 0 SD (9.4-16.4 kg), and -2.0 to 0.4 SD (7.0-18.9 kg). When we used a lower threshold to define postpartum weight increase in the composite outcome (>5 kg compared with >10 kg), the ranges were 1.6 to -0.7 SD (8.9-12.6 kg), -2.2 to -0.3 SD (6.3-14.7 kg), and ≤0.2 SD (≤17.6 kg). Compared with the ranges of the current weight gain guidelines (-0.9 to -0.1 SD, 11.5-16 kg), the lower limits from our data tended to be lower while upper limits were similar or lower.

CONCLUSIONS

If replicated, our results suggest that policy makers should revisit the recommended pregnancy weight gain range for individuals within a normal BMI range.

Expert consensus-based guidance on approaches to opioid management in individuals with advanced cancer-related pain and nonmedical stimulant use

BACKGROUND

Clinicians treating cancer-related pain with opioids regularly encounter nonmedical stimulant use (i.e., methamphetamine, cocaine), yet there is little evidence-based management guidance. The aim of the study is to identify expert consensus on opioid management strategies for an individual with advanced cancer and cancer-related pain with nonmedical stimulant use according to prognosis.

METHODS

The authors conducted two modified Delphi panels with palliative care and addiction experts. In Panel A, the patient's prognosis was weeks to months and in Panel B the prognosis was months to years. Experts reviewed, rated, and commented on the case using a 9-point Likert scale from 1 (very inappropriate) to 9 (very appropriate) and explained their responses. The authors applied the three-step analytical approach outlined in the RAND/UCLA to determine consensus and level of clinical appropriateness of management strategies. To better conceptualize the quantitative results, they thematically analyzed and coded participant comments.

RESULTS

Consensus was achieved for all management strategies. The 120 Experts were mostly women (47 [62%]), White (94 [78%]), and physicians (115 [96%]). For a patient with cancer-related and nonmedical stimulant use, regardless of prognosis, it was deemed appropriate to continue opioids, increase monitoring, and avoid opioid tapering. Buprenorphine/naloxone transition was inappropriate for a patient with a short prognosis and of uncertain appropriateness for a patient with a longer prognosis.

CONCLUSION

Study findings provide urgently needed consensus-based guidance for clinicians managing cancer-related pain in the context of stimulant use and highlight a critical need to develop management strategies to address stimulant use disorder in people with cancer.

PLAIN LANGUAGE SUMMARY

Among palliative care and addiction experts, regardless of prognosis, it was deemed appropriate to continue opioids, increase monitoring, and avoid opioid tapering in the context of cancer-related pain and nonmedical stimulant use. Buprenorphine/naloxone transition as a harm reduction measure was inappropriate for a patient with a short prognosis and of uncertain appropriateness for a patient with a longer prognosis.

Disciplinary trends in the use of the Delphi method: A bibliometric analysis

The Delphi method is an iterative, anonymous, group-based process for eliciting and aggregating opinion on a topic to explore the existence of consensus among experts. The year 2023 marks the 60th anniversary of the first peer-reviewed journal article on the Delphi method. Originally developed for operations research, this method is now applied extensively by researchers representing diverse scientific fields. We used a bibliometric analysis to describe general trends in the expansion of its use across disciplines over time. We conducted a systematic literature search for all English-language, peer-reviewed journal articles on the Delphi method through its first 60 years. We found 19,831 articles: 96.8% (n = 19,204) on the actual use of the Delphi method in an empirical study and 3.2% (n = 627) describing, examining, or providing some guidance on how to use the Delphi method. Almost half (49.9%) of all articles were published in the 2010s and an additional third (32.5%) in the first few years of the 2020s. Nearly two-thirds (65%, n = 12,883) of all published articles have appeared in medical journals, compared to 15% in science and technology (n = 3,053) or social science (n = 3,016) journals. We conclude that the expanded use of the Delphi method has been driven largely by the medical field, though social scientists and technologists continue to be at the forefront of methodological work on the Delphi method. Therefore, we call for greater transdisciplinary collaboration on methodological guidance and standards for the Delphi method.

Variation in Communication of Competing Risks of Mortality in Prostate Cancer Treatment Consultations

PURPOSE

Men with prostate cancer prefer patient-specific, quantitative assessments of longevity in shared decision making. We sought to characterize how physicians communicate the 3 components of competing risks-life expectancy (LE), cancer prognosis and treatment-related survival benefit-in treatment consultations.

MATERIALS AND METHODS

Conversation related to LE, cancer prognosis and treatment-related survival benefit was identified in transcripts from treatment consultations of 42 men with low- and intermediate-risk disease across 10 multidisciplinary providers. Consensus of qualitative coding by multiple reviewers noted the most detailed mode of communication used to describe each throughout the consultation.

RESULTS

Physicians frequently failed to provide patient-specific, quantitative estimates of LE and cancer mortality. LE was omitted in 17% of consultations, expressed as a generalization (eg "long"/"short") in 17%, rough number of years in 31%, probability of mortality/survival at an arbitrary timepoint in 17% and in only 19% as a specific number of years. Cancer mortality was omitted in 24% of consultations, expressed as a generalization in 7%, years of expected life in 2%, probability at no/arbitrary timepoint in 40% and in only 26% as the probability at LE. Treatment-related survival benefit was often omitted; cancer mortality was reported without treatment in 38%, with treatment in 10% and in only 29% both with and without treatment. Physicians achieved "trifecta"-1) quantifying probability of cancer mortality 2) with and without treatment 3) at the patient's LE-in only 14% of consultations.

CONCLUSIONS

Physicians often fail to adequately quantify competing risks. We recommend the "trifecta" approach, reporting 1) probability of cancer mortality 2) with and without treatment 3) at the patient's LE.

Consensus-Based Guidance on Opioid Management in Individuals With Advanced Cancer-Related Pain and Opioid Misuse or Use Disorder

Importance

Opioid misuse and opioid use disorder (OUD) are important comorbidities in people with advanced cancer and cancer-related pain, but there is a lack of consensus on treatment.

Objective

To develop consensus among palliative care and addiction specialists on the appropriateness of various opioid management strategies in individuals with advanced cancer-related pain and opioid misuse or OUD.

Design, Setting, and Participants

For this qualitative study, using ExpertLens, an online platform and methodology for conducting modified Delphi panels, between August and October 2020, we conducted 2 modified Delphi panels to understand the perspectives of palliative and addiction clinicians on 3 common clinical scenarios varying by prognosis (weeks to months vs months to years). Of the 129 invited palliative or addiction medicine specialists, 120 participated in at least 1 round. A total of 84 participated in all 3 rounds.

Main Outcomes and Measures

Consensus was investigated for 3 clinical scenarios: (1) a patient with a history of an untreated opioid use disorder, (2) a patient taking more opioid than prescribed, and (3) a patient using nonprescribed benzodiazepines.

Results

Participants were mostly women (47 [62%]), White (94 (78 [65%]), and held MD/DO degrees (115 [96%]). For a patient with untreated OUD, regardless of prognosis, it was deemed appropriate to begin treatment with buprenorphine/naloxone and inappropriate to refer to a methadone clinic. Beginning split-dose methadone was deemed appropriate for patients with shorter prognoses and of uncertain appropriateness for those with longer prognoses. Beginning a full opioid agonist was deemed of uncertain appropriateness for those with a short prognosis and inappropriate for those with a longer prognosis. Regardless of prognosis, for a patient with no medical history of OUD taking more opioids than prescribed, it was deemed appropriate to increase monitoring, inappropriate to taper opioids, and of uncertain appropriateness to increase the patient's opioids or transition to buprenorphine/naloxone. For a patient with a urine drug test positive for non-prescribed benzodiazepines, regardless of prognosis, it was deemed appropriate to increase monitoring, inappropriate to taper opioids and prescribe buprenorphine/naloxone.

Conclusions and Relevance

The findings of this qualitative study provide urgently needed consensus-based guidance for clinicians and highlight critical research and policy gaps.

Improving Mental Health Guardianship: From Prevention to Treatment

OBJECTIVE

The authors sought to identify the most promising strategies for improving the mental health guardianship process in Los Angeles County for adults with mental illness who are gravely disabled.

METHODS

In May and June 2019, 56 experts, working in hospitals or outpatient facilities or representing legal, advocacy, policy, or forensic organizations, participated in an online modified-Delphi panel, rating the ethical appropriateness, impact on care quality, efficiency, and feasibility of nine strategies for improvement of mental health guardianship. Agreement was determined with the RAND/UCLA appropriateness method, and comments were thematically analyzed.

RESULTS

The strategy ranked highest by the participating experts was improving the administrative functioning and judicial processes of entities involved in mental health guardianship proceedings-it was the only strategy that achieved agreement among panelists and was rated highly on all four criteria. Other preferred strategies were enhancing the ability of assertive outpatient mental health teams to serve individuals before they experience a crisis and expanding the continuum of unlocked residential treatment settings.

CONCLUSIONS

Opportunities exist to improve all stages of the mental health guardianship process. Experts favored strategies that streamline administrative processes, facilitate community integration into treatment, and ensure fidelity to best practices. Improving the mental health guardianship process has the potential to speed up delivery of services, better manage resources, and increase access to treatment for individuals with mental illness who are gravely disabled.

The Effect of the Medicare Advantage Value-Based Insurance Design Model Test on Utilization in 2017

In 2015, the Centers for Medicare and Medicaid Services announced the Medicare Advantage (MA) Value-Based Insurance Design (VBID) model test, which allows MA insurers to use flexible benefit design strategies, such as reduced cost-sharing, to encourage beneficiaries with chronic disease to use high-value care. During the first year of implementation (2017), nine MA insurers offered VBID in 45 health plans to a total of 96 053 eligible beneficiaries. We used MA encounter data to estimate the impact of VBID on health services utilization in 2017 using a difference-in-differences research design. We found that VBID increased use of 10 out of 18 targeted services, and led to general increases in primary care visits, specialty care visits, and drug fills across eligible beneficiaries. The model was also associated with increases in ambulatory care sensitive inpatient and emergency department visits, an unanticipated effect that may be temporary. Overall, our findings suggest that VBID successfully increased the use of high-value services among eligible MA beneficiaries, an important first step along the pathway to better chronic disease management, lower spending, and improved beneficiary health.

Standardizing the Clinical Definition of Opioid Withdrawal in the Neonate

OBJECTIVE

To standardize the clinical definition of opioid withdrawal in neonates to address challenges in clinical care, quality improvement, research, and public policy for this patient population.

STUDY DESIGN

Between October and December 2020, we conducted 2 modified-Delphi panels using ExpertLens, a virtual platform for performing iterative expert engagement panels. Twenty clinical experts specializing in care for the substance-exposed mother-neonate dyad explored the necessity of key evidence-based clinical elements in defining opioid withdrawal in the neonate leading to a diagnosis of neonatal abstinence syndrome (NAS)/neonatal opioid withdrawal syndrome (NOWS). Expert consensus was assessed using descriptive statistics, the RAND/UCLA Appropriateness Method, and thematic analysis of participants' comments.

RESULTS

Expert panels concluded the following were required for diagnosis: in utero exposure (known by history, not necessarily by toxicology testing) to opioids with or without the presence of other psychotropic substances, and the presence of at least two of the most common clinical signs characteristic of withdrawal (excessive crying, fragmented sleep, tremors, increased muscle tone, gastrointestinal dysfunction).

CONCLUSIONS

Results indicate that both a known history of in utero opioid exposure and a distinct set of withdrawal signs are necessary to standardize a definition of neonatal withdrawal. Implementation of a standardized definition requires both patient engagement and a mother-neonate dyadic approach mindful of program and policy implications.

Development of a Metric to Detect and Decrease Low-Value Prescribing in Older Adults

IMPORTANCE

Metrics that detect low-value care in common forms of health care data, such as administrative claims or electronic health records, primarily focus on tests and procedures but not on medications, representing a major gap in the ability to systematically measure low-value prescribing.

OBJECTIVE

To develop a scalable and broadly applicable metric that contains a set of quality indicators (EVOLV-Rx) for use in health care data to detect and reduce low-value prescribing among older adults and that is informed by diverse stakeholders' perspectives.

DESIGN, SETTING, AND PARTICIPANTS

This qualitative study used an online modified-Delphi method to convene an expert panel of 15 physicians and pharmacists. This panel, comprising clinicians, health system leaders, and researchers, was tasked with rating and discussing candidate low-value prescribing practices that were derived from medication safety criteria; peer-reviewed literature; and qualitative studies of patient, caregiver, and physician perspectives. The RAND ExpertLens online platform was used to conduct the activities of the panel. The panelists were engaged for 3 rounds between January 1 and March 31, 2021.

MAIN OUTCOMES AND MEASURES

Panelists used a 9-point Likert scale to rate and then discuss the scientific validity and clinical usefulness of the criteria to detect low-value prescribing practices. Candidate low-value prescribing practices were rated as follows: 1 to 3, indicating low validity or usefulness; 3.5 to 6, uncertain validity or usefulness; and 6.5 to 9, high validity or usefulness. Agreement among panelists and the degree of scientific validity and clinical usefulness were assessed using the RAND/UCLA (University of California, Los Angeles) Appropriateness Method.

RESULTS

Of the 527 low-value prescribing recommendations identified, 27 discrete candidate low-value prescribing practices were considered for inclusion in EVOLV-Rx. After round 1, 18 candidate practices were rated by the panel as having high scientific validity and clinical usefulness (scores of ≥6.5). After round 2 panel deliberations, the criteria to detect 19 candidate practices were revised. After round 3, 18 candidate practices met the inclusion criteria, receiving final median scores of 6.5 or higher for both scientific validity and clinical usefulness. Of those practices that were not included in the final version of EVOLV-Rx, 3 received high scientific validity (scores ≥6.5) but uncertain clinical usefulness (scores <6.5) ratings, whereas 6 received uncertain scientific validity rating (scores <6.5).

CONCLUSIONS AND RELEVANCE

This study culminated in the development of EVOLV-Rx and involved a panel of experts who identified the 18 most salient low-value prescribing practices in the care of older adults. Applying EVOLV-Rx may enhance the detection of low-value prescribing practices, reduce polypharmacy, and enable older adults to receive high-value care across the full spectrum of health services.

The impact of panel composition and topic on stakeholder perspectives: Generating hypotheses from online maternal and child health modified-Delphi panels

Introduction

Multistakeholder engagement is crucial for conducting health services research. Delphi‐based methodologies combining iterative rounds of questions with feedback on and discussion of group results are a well‐documented approach to multistakeholder engagement. This study develops hypotheses about the impact of panel composition and topic on the propensity and meaningfulness of response changes in multistakeholder modified‐Delphi panels.

Methods

We conducted three online modified‐Delphi (OMD) multistakeholder panels using the same protocol. We assigned 60 maternal and child health professionals to a homogeneous (professionals only) panel, 60 pregnant or postpartum women (patients) to a homogeneous panel, and 30 professionals and 30 patients to a mixed panel. In Round 1, participants rated the seriousness of 11 maternal and child health outcomes using a 0–100 scale and explained their ratings. In Round 2, participants saw their own and their panel's Round 1 results and discussed them using asynchronous, anonymous discussion boards moderated by the study investigators. In Round 3, participants revised their original ratings. Our outcome measures included binary indicators of response changes to ratings of the low, medium and high severity maternal and child health outcomes and their meaningfulness, measured by a change of 10 or more points.

Results

Participants changed 818 of 1491 (55%) of responses; the majority of response changes were meaningful. Patterns of response changes were different for patients and professionals and for different levels of outcome seriousness. Using study results and the literature, we developed three hypotheses. First, OMD participants, regardless of their stakeholder group, are more likely to change their responses on preference‐sensitive topics where there is a range of viable alternatives or perspectives. Second, patients are more likely to change their responses and to do so meaningfully in mixed panels, whereas professionals are more likely to do so in homogeneous panels. Third, the association between panel composition and response change varies according to the topic (e.g., the level of outcome seriousness).

Conclusions

Results of our work not only helped generate empirically derived hypotheses to be tested in future research but also offer practical recommendations for designing multistakeholder OMD panels.

Patient or Public Contribution

Pregnant or postpartum women were involved in this study.

Expert Panel Consensus on Management of Advanced Cancer-Related Pain in Individuals With Opioid Use Disorder

IMPORTANCE

Opioid use disorder (OUD) is an important comorbidity in individuals with advanced cancer, in whom pain is common. Full-agonist opioid medications are the cornerstone of cancer pain management, but the existing literature does not address how to manage cancer pain in patients with OUD.

OBJECTIVE

To conduct an expert panel to develop consensus on the appropriateness of management of cancer pain in individuals with co-occurring advanced cancer and OUD.

EVIDENCE REVIEW

A 3-round modified Delphi process was completed from August to October 2020 with 2 cases: patient with advanced cancer, pain, and OUD treated with buprenorphine-naloxone or methadone. Participants rated management strategies in round 1, discussed results in round 2, and provided final responses in round 3. ExpertLens, an online approach to conducting modified Delphi panels, was used. Participants were experts in palliative care, addiction, or both, recruited by email from palliative care and addiction-focused professional groups, lists from prior studies, and snowball sampling. Data analysis was performed from November 2020 to July 2021.

FINDINGS

Of 120 experts (median age, 40-49 years), most were White (78 participants [94%]), female (74 participants [62%]), and held MD or DO degrees (115 participants [96%]); 84 (70%) participated in all rounds. For a patient with OUD taking buprenorphine-naloxone, it was deemed appropriate to continue buprenorphine-naloxone with thrice-daily dosing. Continuing buprenorphine-naloxone and adding a full-agonist opioid was deemed to be appropriate for patients with a prognosis of weeks to months and of uncertain appropriateness for patients with a prognosis of months to years. For a patient with OUD taking methadone dispensed at a methadone clinic, it was deemed appropriate to take over prescribing and dose twice or thrice daily. Continuing methadone daily while adding another full-agonist opioid was deemed appropriate for patients with a prognosis of weeks to months and of uncertain appropriateness for those with a prognosis of months to years.

CONCLUSIONS AND RELEVANCE

The findings of this qualitative study provide urgently needed, consensus-based guidance for clinicians and highlight critical research and policy gaps needed to facilitate implementation.

Alignment Between Objective and Subjective Assessments of Health System Performance: Findings From a Mixed-Methods Study

EXECUTIVE SUMMARY

The article examines whether subjective performance assessments from health system executives match objective performance assessments and qualitatively explores ways to achieve high performance. We interviewed 138 C-suite executives of 24 health systems in California, Minnesota, Washington, and Wisconsin between 2017 and 2019. We used maximum variation sampling to select health systems to achieve diversity in performance on objective measures of clinical performance. Our interviews focused on executives' perceptions of their own health system's performance and factors they thought generally contributed to high performance. In our analysis, we grouped health systems based on objective performance levels (high, medium, and low) used in sampling, compared objective performance ratings with executives' subjective performance assessments, and used thematic analysis to identify reasons for subjective assessment of health system performance and levers of high performance in general. There was poor agreement between objective and subjective performance assessments (kappa = 0.082). Subjective assessments were higher than objective assessments and captured more factors than are typically considered in performance accountability and value-based payment initiatives. Executives whose views were inconsistent with objective performance assessments did not cite clinical care quality per se as the basis for their assessment, focusing instead on market competition, financial performance, and high customer satisfaction and loyalty. Executives who cited clinical quality metrics as the basis of their assessment offered subjective ratings consistent with objective ratings. Executives identified organizational culture, organizational governance, and staff engagement as levers for achieving high performance. Future research should explore the benefits and drawbacks of considering subjective performance assessments in value-based payment initiatives.

Informing the Development of a Standardized Clinical Definition of Neonatal Abstinence Syndrome: Protocol for a Modified-Delphi Expert Panel

Background

Neonatal abstinence syndrome (NAS) is a postnatal withdrawal syndrome that most commonly results from prenatal opioid exposure. Every 15 minutes, an infant is born in the United States with signs of NAS. The field lacks a standardized clinical definition of NAS, complicating discussions on programmatic and policy development to support opioid-exposed mothers and infants.

Objective

The goal of this paper is to describe a protocol for a systematic expert panel process to inform the development of a clinical definition of NAS.

Methods

We will conduct two three-round online modified-Delphi panels using the ExpertLens system and will follow the recommendations for Conducting and REporting of DElphi Studies (CREDES). One panel will focus on developing key components of a clinical definition of NAS, and the second panel will focus on neonatal opioid withdrawal syndrome (NOWS), which is a term that has come into use to differentiate opioid-exposed infants from infants exposed to other substances in utero. However, there is lack of agreement on the precise clinical definition of NOWS and how it is distinct from or overlaps with NAS. Each panel will complete two rating rounds and a discussion round using a similar protocol. We will analyze all rating data descriptively and determine the presence of agreement within and between the two panels. We will also perform thematic analysis of the qualitative comments to contextualize the panel findings.

Results

The panels were convened between October 29 and December 17, 2020. Their results were disseminated and discussed at a national conference on NAS that took place on March 17-18, 2021.

Conclusions

A standardized clinical definition of NAS will help to better characterize NAS incidence and to design effective clinical, public health, and policy interventions to support opioid-exposed mother-infant dyads.

International Registered Report Identifier (IRRID)

DERR1-10.2196/25387

Rating the seriousness of maternal and child health outcomes linked with pregnancy weight gain

BACKGROUND

Current pregnancy weight gain guidelines were developed based on implicit assumptions of a small group of experts about the relative seriousness of adverse health outcomes. Therefore, they will not necessarily reflect the values of women.

OBJECTIVE

To estimate the seriousness of 11 maternal and child health outcomes that have been consistently associated with pregnancy weight gain by engaging patients and health professionals.

METHODS

We collected data using an online panel approach with a modified Delphi structure. We selected a purposeful sample of maternal and child health professionals (n = 84) and women who were pregnant or recently postpartum (patients) (n = 82) in the United States as panellists. We conducted three concurrent panels: professionals only, patients only, and patients and professionals. During a 3-round online modified Delphi process, participants rated the seriousness of health outcomes (Round 1), reviewed and discussed the initial results (Round 2), and revised their original ratings (Round 3). Panellists assigned seriousness ratings (0, [not serious] to 100 [most serious]) for infant death, stillbirth, preterm birth, gestational diabetes, preeclampsia, small-for-gestational-age (SGA) birth, large-for-gestational-age (LGA) birth, unplanned caesarean delivery, maternal obesity, childhood obesity, and maternal metabolic syndrome.

RESULTS

Each panel individually came to a consensus on all seriousness ratings. The final median seriousness ratings combined across all panels were highest for infant death (100), stillbirth (95), preterm birth (80), and preeclampsia (80). Obesity in children, metabolic syndrome in women, obesity in women, and gestational diabetes had median seriousness ratings ranging from 55 to 65. The lowest seriousness ratings were for SGA birth, LGA birth, and unplanned caesarean delivery (30-40).

CONCLUSION

Professionals and women rate some adverse outcomes as being more serious than others. These ratings can be used to establish the range of pregnancy weight gain associated with the lowest risk of a broad range of maternal and child health outcomes.

Consensus-based approach to managing opioids, including opioid misuse and opioid use disorder, in patients with serious illness: protocol for a modified Delphi process

INTRODUCTION

Management of opioid misuse and opioid use disorder (OUD) among individuals with serious illness is an important yet understudied issue. Palliative care clinicians caring for individuals with serious illness, many of whom may live for months or years, describe a complex tension between weighing the benefits of opioids, which are considered a cornerstone of pain management in serious illness, and serious opioid-related harms like opioid misuse and OUD. And yet, little literature exists to inform the management of opioid misuse and OUDs among individuals with serious illness. Our objective is to provide evidence-based management guidance to clinicians caring for individuals with serious illness who develop opioid misuse or OUD.

METHODS AND ANALYSIS

We chose a modified Delphi approach, which is appropriate when empirical evidence is lacking and expert input must be used to shape clinical guidance. We sought to recruit 60 clinicians with expertise in palliative care, addiction or both to participate in this study. We created seven patient cases that capture important management challenges in individuals with serious illness prescribed opioid therapy. We used ExpertLens, an online platform for conducting modified Delphi panels. Participants completed three rounds of data collection. In round 1, they rated and commented on the appropriateness of management choices for cases. In round 2, participants reviewed and discussed their own and other participants' round 1 numerical responses and comments. In round 3 (currently ongoing), participants again reviewed rounds 1 and 2, and are allowed to change their final numerical responses. We used ExpertLens to automatically identify whether there is consensus, or disagreement, among responses in panels. Only round 3 responses will be used to assess final consensus and disagreement.

ETHICS AND DISSEMINATION

This project received ethical approval from the University of Pittsburgh's Institutional Review Board (study 19110301) and the RAND Institutional Research Board (study 2020-0142). Guidance from this work will be disseminated through national stakeholder networks to gain buy-in and endorsement. This study will also form the basis of an implementation toolkit for clinicians caring for individuals with serious illness who are at risk of opioid misuse or OUD.

Priority-Setting to Address the Geriatric Pharmacoparadox for Pain Management: A Nursing Home Stakeholder Delphi Study

BACKGROUND

Evidence to guide clinical decision making for pain management in nursing home residents is scant.

OBJECTIVE

Our objective was to explore the extent of consensus among expert stakeholders regarding what analgesic issues should be prioritized for comparative-effectiveness studies of beneficial and adverse effects of analgesic regimens in nursing home residents.

METHODS

Two stakeholder panels (nurses only and a mix of clinicians/researchers) were engaged (n = 83). During a three-round online modified Delphi process, participants rated and commented on the need for new evidence on nonopioid analgesic regimens and opioid regimens, short-term adverse effects, long-term adverse effects, comorbid conditions, and other factors in the nursing home setting (9-point scale; 1 = not essential to 9 = very essential to obtain new evidence). The quantitative data were analyzed to determine the existence of consensus using an approach from the RAND/UCLA Appropriateness Method User's Manual. The qualitative data, consisting of participant explanations of their numeric ratings, were thematically analyzed by an experienced qualitative researcher.

RESULTS

For nursing home residents, evidence generation was deemed essential for opioids, gabapentin (alone or with serotonin norepinephrine reuptake inhibitors [SNRIs]), and nonsteroid anti-inflammatory drugs with SNRIs. Experts prioritized the following outcomes as essential: long-term adverse effects, including delirium, cognitive decline, and decline in activities of daily living (ADLs). Kidney disease and depression were deemed essential conditions to consider in studies of pain medications. Coprescribing analgesic regimens with benzodiazepines, sedating medications, serotonergic medications, and non-SNRI antidepressants were considered essential areas of study. Experts noted that additional study was essential in residents with moderate/severe cognitive impairment and limitations in ADLs.

CONCLUSIONS

Stakeholder priorities for more evidence reflect concerns related to treating medically complex residents with complex drug regimens and included long-term adverse effects, coprescribing, and sedating medications. Carefully conducted observational studies are needed to address the vast evidence gap for nursing home residents.

Nature and Predictors of Response Changes in Modified-Delphi Panels

OBJECTIVES

To describe the extent and nature of changes in participants' responses after exposure to group feedback and discussion in modified-Delphi panels and to identify factors affecting those changes.

METHODS

We analyzed data from 2 online modified-Delphi panels, each consisting of 2 rating rounds and an online discussion round. We included responses from 55 participants who answered 38 questions in both rating rounds. Because not all participants answered each question twice, our sample consisted of 1846 cases (response changes). We used mixed-effect logistic and multinomial logistic regression to identify factors predicting response changes and their direction relative to group median-our consensus measure.

RESULTS

Participants changed, on average, 49% of their responses. A response was changed in 47% of the 1846 cases: 28% of responses were changed toward consensus and 19% away from it. Although some measures of subjective participation experiences had a marginally significant impact on the propensity and direction of response changes, several objective measures of discussion engagement were statistically significant predictors of both the presence and direction of response changes.

CONCLUSION

Our results illustrate the nature of response changes and highlight the importance of exposing participants to alternative perspectives and encouraging them to explain their perspectives.

Optimizing health IT to improve health system performance: A work in progress

BACKGROUND

Despite significant investments in health information technology (IT), the technology has not yielded the intended performance effects or transformational change. We describe activities that health systems are pursuing to better leverage health IT to improve performance.

METHODS

We conducted semi-structured telephone interviews with C-suite executives from 24 U.S. health systems in four states during 2017-2019 and analyzed the data using a qualitative thematic approach.

RESULTS

Health systems reported two broad categories of activities: laying the foundation to improve performance with IT and using IT to improve performance. Within these categories, health systems were engaged in similar activities but varied greatly in their progress. The most substantial effort was devoted to the first category, which enabled rather than directly improved performance, and included consolidating to a single electronic health record (EHR) platform and common data across the health system, standardizing data elements, and standardizing care processes before using the EHR to implement them. Only after accomplishing such foundational activities were health systems able to focus on using the technology to improve performance through activities such as using data and analytics to monitor and provide feedback, improving uptake of evidence-based medicine, addressing variation and overuse, improving system-wide prevention and population health management, and making care more convenient.

CONCLUSIONS AND IMPLICATIONS

Leveraging IT to improve performance requires significant and sustained effort by health systems, in addition to significant investments in hardware and software. To accelerate change, better mechanisms for creating and disseminating best practices and providing advanced technical assistance are needed.

Standardizing care to manage bleeding disorders in adolescents with heavy menses-A joint project from the ISTH pediatric/neonatal and women's health SSCs

BACKGROUND

Bleeding disorders (BD) are under-recognized in adolescents with heavy menstrual bleeding (HMB).

OBJECTIVES

The lack of clinical guidelines and variable symptomatic management of HMB created the imperative to standardize HMB care to identify and manage BD in adolescents.

METHODS

We convened an international working group (WG), utilized the results of a literature review to define knowledge gaps in HMB care, and used the collective clinical experience of the WG to develop care considerations for adolescents with BD and HMB. We then solicited input on the appropriateness of HMB care considerations from expert stakeholders representing hematology, adolescent medicine, and obstetrics-gynecology. We conducted an expert panel online, using the ExpertLens platform. During a three-round online modified-Delphi process, the expert panel rated the appropriateness of 21 care considerations using a 9-point scale to designate care as appropriate (7-9), uncertain (4-6), or inappropriate (1-3) covering screening for BD, the laboratory work-up, and management of adolescents with BD that present with HMB. We used the RAND/UCLA appropriateness method to determine the existence of consensus among the interdisciplinary panel of experts.

RESULTS

Thirty-nine experts participated in the panel. The experts rated fifteen HMB care considerations as appropriate, six as uncertain, and none as inappropriate.

CONCLUSIONS

The HMB care statements represent the first set of HMB care considerations in adolescents with BD, developed with broad expert input on appropriateness. Although likely to be of interest to a range of clinicians who routinely manage adolescents with HMB, additional research is required in many key areas.

ADAPT study: adaptation of evidence-informed complex population health interventions for implementation and/or re-evaluation in new contexts: protocol for a Delphi consensus exercise to develop guidance

INTRODUCTION

Complex population health interventions that are effective in one context may not be effective elsewhere, and may even be harmful. As such, an intervention may require adaptation to ensure it fits with a new context. To date, there is no overarching guidance to help researchers to adapt and evaluate interventions in new contexts, and no criteria to support research funders or journals assess proposed or reported adaptations or evaluation. There is limited assistance for policy-makers and practitioners to decide if evidence-informed interventions are appropriate to their context, or if adaptation and further evaluation is needed. This Delphi exercise will contribute to the development of guidance for these communities to support the adaptation, implementation and/or re-evaluation of complex population health interventions in new contexts.

METHODS

We will conduct a Delphi consensus exercise to gather expert opinion from researchers, research funders, journal editors and policy-makers. Expert opinion will be sought on: appropriate definitions and concepts, identifying key methodological considerations and establishing adaptations and processes to be undertaken during adaptation of complex population health interventions in new contexts.

ETHICS AND DISSEMINATION

Ethics approval for the Delphi exercise has been obtained from the University of Glasgow and and the RAND institutional research board. Dissemination of the results of this study will be through peer-reviewed publications, workshops at national and international conferences, and a summary of the guidance developed for key organisations and stakeholders.

Engaging Patients and Professionals to Evaluate the Seriousness of Maternal and Child Health Outcomes: Protocol for a Modified Delphi Study

BACKGROUND

Maternal weight gain during pregnancy is one of the few potentially modifiable risk factors for many adverse maternal and child health outcomes. Defining the optimal pregnancy weight gain range is difficult because, while lower weight gain may prevent some outcomes, such as maternal and child obesity, it may increase the risk of others such as fetal growth restriction and infant death. These health outcomes vary in their seriousness to mothers and their health care providers, and these differences in seriousness should be taken into account when determining optimal weight gain ranges. However, the relative seriousness that women and their care providers place on different health outcomes is unknown.

OBJECTIVE

We will determine the seriousness of 11 maternal and child health outcomes that have been consistently associated with pregnancy weight gain. We will achieve this by engaging patients and maternal and child health professionals using an online modified Delphi panel process.

METHODS

We aim to recruit a racially/ethnically and geographically diverse group of 90 US maternal and child health professionals and 90 women who are pregnant or less than 2 years postpartum. We will conduct 3 concurrent panels using the ExpertLens system, a previously evaluated online modified Delphi system that combines 2 rounds of rating with 1 round of feedback and moderated online discussion. In Round 1, panelists are asked to rate the seriousness of each health outcome on a scale of 0-100 and to provide a rationale for their scores. In Round 2, panelists will review their responses relative to those of other panelists. They will discuss their seriousness ratings anonymously using a moderated online discussion board. In Round 3, participants will revise their Round 1 responses based on group feedback and discussion. Each round will be open for 1-2 weeks.

RESULTS

The study protocol was reviewed by our ethics boards and did not require approval as human research. A pilot study of 6 professionals and 7 patients was completed in December 2019.

CONCLUSIONS

Our numeric estimates of the seriousness of maternal and child health outcomes will enable future studies to determine pregnancy weight gain ranges that balance the risks of low and high weight gain for mothers and children.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/16478.

From "Informed" to "Engaged" Consent: Risks and Obligations in Consent for Participation in a Health Data Repository

The development and use of large and dynamic health data repositories designed to support research pose challenges to traditional informed consent models. We used semi-structured interviewing (n=44) to elicit diverse research stakeholders' views of a model of consent appropriate to participation in initiatives that entail collection, long-term storage, and undetermined future research use of multiple types of health data. We demonstrate that, when considering health data repositories, research stakeholders replace a concept of consent as informed with one in which consent is engaged. In engaged consent, a participant's ongoing relationship with a repository serves as a substitute or adjunct to information exchange at enrollment. We detail research stakeholders' views of the risks of engaged consent and suggest questions for further study about engagement and consent procedures in initiatives that aim to store data for future unspecified research purposes.

Practical Considerations in Using Online Modified-Delphi Approaches to Engage Patients and Other Stakeholders in Clinical Practice Guideline Development

Patients and caregivers are increasingly recognized as key stakeholders in developing clinical practice guidelines (CPGs). Online engagement approaches offer the promise of a rigorous, scalable, and convenient engagement method. This paper illustrates how an online modified-Delphi approach could be used to engage patients, caregivers, and other stakeholder in CPG development. It provides practical guidance for conducting online modified-Delphi panels that covers (1) joint development of the engagement approach with relevant stakeholders, (2) adaptation of methods used by experts in guideline development, (3) pilot testing, (4) participant recruitment, (5) determining panel size and composition, (6) building participant capacity, (7) facilitation of two-way interaction, (8) continuous engagement and retention of participants, (9) rigorous data analysis, (10) evaluation of engagement activities, and (11) result dissemination. The paper is based on a recently completed study about engaging individuals with Duchenne muscular dystrophy (DMD) and their caregivers in determining the patient-centeredness of DMD care guidelines.

Using an Online, Modified Delphi Approach to Engage Patients and Caregivers in Determining the Patient-Centeredness of Duchenne Muscular Dystrophy Care Considerations

Purpose. To determine the patient-centeredness of endocrine and bone health Duchenne muscular dystrophy (DMD) care considerations using the RAND/PPMD Patient-Centeredness Method (RPM), which is a novel, online, modified-Delphi approach to engaging patients and caregivers in clinical guideline development. Methods. We solicited input on the patient-centeredness of care considerations from 28 individuals with DMD and 94 caregivers, randomly assigned to 1 of 2 mixed panels. During a 3-round online modified-Delphi process, participants rated the importance and acceptability of 19 DMD care considerations (round 1), reviewed and discussed the initial results (round 2), and revised their original ratings (round 3). Patient-centeredness was operationalized as importance and acceptability of recommendations. We considered a care consideration to be patient-centered if both panels deemed it important and acceptable. Results. Ninety-five panelists (78%) participated in this study. Of these, 88 (93%) participated in round 1, 74 (78%) in round 2, and 56 (59%) in round 3. Panelists deemed 12 care considerations to be patient-centered: 3 weight management, 3 bone health, 4 vertical growth, and 2 puberty recommendations. Seven care considerations did not meet patient-centeredness criteria. Common reasons were lack of evidence specific to DMD and concerns about insurance coverage, access to treatment, and patient safety. Conclusions. Using the RPM, Duchenne families considered most care considerations to be patient-centered. Besides being clinically appropriate, these considerations are likely to be consistent with the preferences, needs, and values of Duchenne families. While all relevant care considerations should be discussed during patient-provider encounters, those that did not meet patient-centeredness criteria in particular should be carefully considered as part of joint decision making between Duchenne families and their providers. Study Registration: HSRProj 20163126.

Return of Value in the New Era of Biomedical Research-One Size Will Not Fit All

Background: There is a growing interest in creating large-scale repositories that store genetic, behavioral, and environmental data for future, unspecified uses. The All of Us Research Program is one example of such a repository. Its participants will get access to their personal data and the results of the studies that used them. However, little is known about what researchers should return to participants and how they should do it in a way that is valuable and meaningful to participants. Methods: To better understand the concept of "return of value" and the practice of returning valuable study information, we conducted semi-structured telephone interviews with 44 stakeholders with diverse perspectives on this topic. All interviews have been transcribed and coded thematically to identify the most salient themes, to explore differences between returning different types of study results, and to describe differences and similarities in perspectives of different stakeholder groups. Results: We found that one size does not fit all when it comes to returning value to participants: the decisions about return of results are affected by participant preferences, researchers' concerns about feasibility, the types of data collected, their level of granularity, and available options for supporting result interpretation. Conclusions: Our findings suggest that the key to operationalizing return of value and to identifying ways to return valuable information to study participants may be to find a point of equilibrium between criteria that may affect usefulness and feasibility. The point of equilibrium may vary by study, by participants' backgrounds and preferences, by their health literacy and access to regular healthcare, and by the resources available to professionals controlling the data. Future studies should explore the factors that determine the point of equilibrium between feasibility and usefulness.

Patient and caregiver perspectives on guideline adherence: the case of endocrine and bone health recommendations for Duchenne muscular dystrophy

BACKGROUND

Clinical care guidelines are typically developed by clinicians and researchers. Including patient and caregiver voices in guideline development may help create guidelines that are more useful for patients and consequently improve their guideline adherence. Although there is substantial research on the factors the affect providers' adherence to guidelines, there is less research on the factors that affect patients' compliance with guideline recommendations, especially among those with rare disorders. The purpose of this study is to explore factors that are likely to affect patient/caregiver adherence to endocrine and bone health recommendations for Duchenne Muscular Dystrophy (DMD). To do so, we used qualitative data collected as part of the study designed to develop, implement, and evaluate a new online, modified-Delphi approach to engaging patients with rare diseases and their caregivers in guideline development, using care guidelines for DMD as a case study.

METHODS

We thematically analyzed qualitative data collected from 95 adults with DMD and their caregivers who participated in at least one round of our online Modified-Delphi panel process. Participants rated and commented on the patient-centeredness of 19 recommendations about vertical growth, weight management, bone health, and delayed puberty included in the 2018 DMD care considerations. Patient-centeredness was operationalized as the importance and acceptability of care recommendations.

RESULTS

Thematic analyses revealed six factors that affect guideline adherence from the patient/caregiver perspective: content and format of recommendations, patient and provider characteristics, and social and financial factors.

CONCLUSIONS

This study used a novel approach to exploring patient and caregiver perspectives on factors that may affect guideline adherence. The six factors identified by DMD patients and caregivers are similar to the factors affecting provider adherence and are not limited to DMD. Understanding consistency between provider- and patient/caregiver-identified barriers to following guideline recommendations can lead to developing more successful interventions for increasing guideline adherence.

Insurers' perspectives on MA value-based insurance design model

OBJECTIVES

Value-based insurance design (VBID) lowers cost sharing for high-value healthcare services that are clinically beneficial to patients with certain conditions. In 2017, the Center for Medicare and Medicaid Innovation began a voluntary VBID model test in Medicare Advantage (MA). This article describes insurers' perspectives on the MA VBID model, explores perceived barriers to joining this model, and describes ways to address participation barriers.

STUDY DESIGN

A descriptive, qualitative study.

METHODS

In spring/summer 2017, we conducted semistructured interviews with 24 representatives of 10 nonparticipating MA insurers to learn why they did not join the model test. We interviewed 73 representatives of 8 VBID-participating insurers about their participation decisions and implementation experiences. All interview data were analyzed thematically.

RESULTS

Fewer than 30% of eligible insurers participated in the first 2 years of the model test. The main barriers to entry were a perceived lack of information on VBID in MA, an expectation of low return on investment, concerns over administrative and information technology (IT) hurdles, and model design parameters. Most VBID participants encountered administrative and IT hurdles but overcame them. CMS made changes to the model parameters to increase the uptake.

CONCLUSIONS

The model uptake was low, and implementation challenges and concerns over VBID effectiveness in the Medicare population were important factors in participation decisions. To increase uptake, CMS could consider providing in-kind implementation assistance to model participants. Nonparticipants may want to incorporate lessons learned from current participants, and insurers should engage their IT departments/vendors early on.

Participant experiences with a new online modified-Delphi approach for engaging patients and caregivers in developing clinical guidelines

BACKGROUND

Patient engagement in clinical practice guideline (CPG) development has increased significantly in recent years. However, only a few patients and caregivers join CPG development groups.

OBJECTIVE

To describe participant experiences with a novel online, scalable approach for patient and caregiver engagement in CPG development.

DESIGN

We developed and tested the RAND/PPMD Patient-Centeredness Method (RPM), a novel online modified-Delphi approach to patient engagement in CPG development that consists of an optional idea generation round and two rating rounds interspersed with an online discussion round.

SETTING AND PARTICIPANTS

Using the online ExpertLens™ system, we ran 2 concurrent panels of patients and caregivers of individuals with Duchenne Muscular Dystrophy (DMD).

MAIN OUTCOME MEASURES

We surveyed all 95 panel participants about their participation and satisfaction with the process. We also conducted telephone interviews with 25 participants.

RESULTS

Participants expressed satisfaction with various ExpertLens™ features, noting that the system fostered lively interaction among them. Panelists also appreciated participating in an educational, interactive and convenient discussion forum that allowed them to share their opinions on the importance and acceptability of different recommendations. The RPM was viewed as empowering by patients and their caregivers who felt it would be useful for CPG developers.

DISCUSSION AND CONCLUSION

The results of our study show the overall participant satisfaction with a novel, scalable, online approach to engaging patients and caregivers in CPG development, which allows them to share their perspectives and lived experiences using a rigorous, systematic and iterative way that is similar to how clinicians provide their input.

The RAND/PPMD Patient-Centeredness Method: a novel online approach to engaging patients and their representatives in guideline development

Although clinical practice guidelines (CPGs) provide recommendations for how best to treat a typical patient with a given condition, patients and their representatives are not always engaged in CPG development. Despite the agreement that patient participation may improve the quality and utility of CPGs, there is no systematic, scalable method for engaging patients and their representatives, as well as no consensus on what exactly patients and their representatives should be asked to do during CPG development. To address these gaps, an interdisciplinary team of researchers, patient representatives, and clinicians developed the RAND/PPMD Patient-Centeredness Method (RPM) - a novel online approach to engaging patients and their representatives in CPG development. The RPM is an iterative approach that allows patients and their representatives to provide input by (1) generating ideas; (2) rating draft recommendations on two criteria (importance and acceptability); (3) explaining and discussing their ratings with other participants using online, asynchronous, anonymous, moderated discussion boards, and (4) revising their responses if needed. The RPM was designed to be consistent with the RAND/UCLA Appropriateness Method used by clinicians and researchers to develop CPG, while helping patients and their representative rate outcome importance and recommendation acceptability - two key components of the GRADE Evidence to Decision (EtD) framework. With slight modifications, the RPM has the potential to explore consensus among key stakeholders on other dimensions of the EtD, including feasibility, equity, and resource use.

Practical Considerations for Using Online Methods to Engage Patients in Guideline Development

Clinical practice guidelines (CPGs) have been widely used in healthcare policy, practice, and for suggesting future research. As patients increasingly become involved in CPG development to produce patient-centered recommendations, more research is needed on methods to engage patients, particularly methods allowing for scalable engagement of large, diverse, and geographically distributed groups of patients. In this article, we discuss practical considerations for using online methods to engage patients in CPG development. To inform this discussion, we conducted a rapid, systematic review of literature on patient involvement in CPG development and used qualitative evidence synthesis techniques to make inferences about potential advantages and challenges of using online methods to engage patients in this context. We identified 79 articles containing information about involving patients in CPG development. Potential advantages include the ability of online methods to facilitate greater openness and honesty by patients, as well as to reflect the diversity of patient views, which in turn further improve the utility of CPGs. Potential challenges of using online methods may include the extra skill, time, and certain types of resources that may be needed for patient engagement, as well as the difficulty engaging specific patient populations. However, these challenges are mitigated by growing calls for patient engagement as normative for CPG development in addition to patients' increasing familiarity with online technologies. These practical considerations should be examined empirically as guideline development groups further explore the appropriateness of using online methods to engage patients across different stages of CPG development.

Whole Person Care in Under-resourced Communities: Stakeholder Priorities at Long-Term Follow-Up in Community Partners in Care

Objective

Depressed individuals may require help from different agencies to address health and social needs, but how such coordination occurs in under-resourced communities is poorly understood. This study sought to identify priorities of Latino and African American depressed clients, explore whether service providers understand client priorities, and describe how providers address them.

Methods

Between October 2014 and February 2015, we interviewed 104 clients stratified by depression history and 50 representatives of different programs in health and social community agencies who participated in Community Partners in Care, a cluster-randomized trial of coalition-building approaches to delivering depression quality improvement programs. Clients were queried about their most pressing needs; program representatives identified their clients' needs and explained how they addressed them.

Results

Physical and mental health were clients' top priorities, followed by housing, caring for and building relationships with others, and employment. While persistently depressed clients prioritized mental health, those with improved depression prioritized relationships with others. Program representatives identified housing, employment, mental health, and improving relationships with others as clients' top priorities. Needs assessment, client-centered services, and linkages to other agencies were main strategies used to address client needs.

Conclusion

Depressed clients have multiple health and social needs, and program representatives in under-resourced communities understand the complexity of clients' needs. Agencies rely on needs assessment and referrals to meet their clients' needs, which enhances the importance of agency partnership in "whole person" initiatives. Our results illustrate agency capacity to adopt integrated care models that will address clients' multiple needs through multi-sector collaboration and describe potential strategies to help reach the goal of whole person care.

Lack of preregistered analysis plans allows unacceptable data mining for and selective reporting of consensus in Delphi studies

OBJECTIVES

To empirically demonstrate how undisclosed analytic flexibility provides substantial latitude for data mining and selective reporting of consensus in Delphi processes.

STUDY DESIGN AND SETTING

Pooling data across eight online modified-Delphi panels, we first calculated the percentage of items reaching consensus according to descriptive analysis procedures commonly used in health research but selected post hoc in this article. We then examined the variability of items reaching consensus across panels.

RESULTS

Pooling all panel data, the percentage of items reaching consensus ranged from 0% to 84%, depending on the analysis procedure. Comparing data across panels, variability in the percentage of items reaching consensus for each analysis procedure ranged from 0 (i.e., all panels had the same percentage of items reaching consensus for a given analysis procedure) to 83 (i.e., panels had a range of 11% to 94% of items reaching consensus for a given analysis procedure). Of 200 total panel-by-analysis-procedure configurations, four configurations (2%) had all items and 64 (32%) had no items reaching consensus.

CONCLUSION

Undisclosed analytic flexibility makes it unacceptably easy to data mine for and selectively report consensus in Delphi processes. As a solution, we recommend prospective, complete registration of preanalysis plans for consensus-oriented Delphi processes in health research.

Importance and Feasibility of Transitional Care for Children With Medical Complexity: Results of a Multistakeholder Delphi Process

BACKGROUND

Children with medical complexity (CMC) account for disproportionate hospital utilization and adverse outcomes after discharge, and several gaps exist regarding the quality of hospital to home transitional care for this population. We conducted an expert elicitation process to identify important and feasible hospital to home transitional care interventions for CMC from the perspectives of parents and health care professionals.

METHODS

We conducted a 2-round electronic Delphi process to identify important and feasible transitional care interventions. Panelists included parents of CMC and multidisciplinary health care professionals. In the first round, panelists rated the importance and feasibility of 39 transitional care interventions on a 9-point Likert scale; agreement between panelists was defined according to RAND/UCLA Appropriateness Methods. The second round of data collection evaluated 16 interventions that panelists did not agree on in the first round and 8 new or revised interventions, accompanied by quantitative and qualitative data summaries.

RESULTS

A total of 29 parents of CMC and 37 health care professionals participated in the Delphi process (response rate 75%). Both stakeholder panels endorsed most interventions as important; health care professionals were less likely to rate several interventions as feasible compared with the parent panel. Over 2 rounds of data collection, the 2 stakeholder panels endorsed 25 interventions as important as well as feasible. These interventions related to family engagement during the hospitalization, care coordination and social support assessment, predischarge education, and written materials.

CONCLUSIONS

Parents and health care professionals considered several transitional care interventions important as well as feasible. This research might inform hospitals' transitional care programs and policies.

A Community-Partnered, Participatory, Cluster-Randomized Study of Depression Care Quality Improvement: Three-Year Outcomes

OBJECTIVE

Community Partners in Care, a community-partnered, cluster-randomized trial with depressed clients from 93 Los Angeles health and community programs, examined the added value of a community coalition approach (Community Engagement and Planning [CEP]) versus individual program technical assistance (Resources for Services [RS]) for implementing depression quality improvement in underserved communities. CEP was more effective than RS in improving mental health-related quality of life, reducing behavioral health hospitalizations, and shifting services toward community-based programs at six months. At 12 months, continued evidence of improvement was found. This study examined three-year outcomes.

METHODS

Among 1,004 participants with depression who were eligible for three-year follow-up, 600 participants from 89 programs completed surveys. Multiple regression analyses estimated intervention effects on poor mental health-related quality of life and depression, physical health-related quality of life, behavioral health hospital nights, and use of services.

RESULTS

At three years, no differences were found in the effects of CEP versus RS on depression or mental health-related quality of life, but CEP had modest effects in improving physical health-related quality of life and reducing behavioral health hospital nights, and CEP participants had more social- and community-sector depression visits and greater use of mood stabilizers. Sensitivity analyses with longitudinal modeling reproduced these findings but found no significant differences between groups in change from baseline to three years.

CONCLUSIONS

At three years, CEP and RS did not have differential effects on primary mental health outcomes, but CEP participants had modest improvements in physical health and fewer behavioral health hospital nights.

Comparative Effectiveness of Two Models of Depression Services Quality Improvement in Health and Community Sectors

OBJECTIVE

The effectiveness of community coalition building and program technical assistance was compared in implementation of collaborative care for depression among health care and community sector clients.

METHODS

In under-resourced communities, within 93 programs randomly assigned to coalition building (Community Engagement and Planning) or program technical assistance (Resources for Services) models, 1,018 clients completed surveys at baseline and at six, 12, or 36 months. Regression analysis was used to estimate intervention effects and intervention-by-sector interaction effects on depression, mental health-related quality of life, and community-prioritized outcomes and on services use.

RESULTS

For outcomes, there were few significant intervention-by-sector interactions, and stratified findings suggested benefits of coalition building in both sectors. For services use, at 36 months, increases were found for coalition building in primary care visits, self-help visits, and appropriate treatment for community clients and in community-based services use for health care clients.

CONCLUSIONS

Relative to program technical assistance, community coalition building benefited clients across sectors and shifted long-term utilization across sectors.

Development of a Canadian Core Clinical Dataset to Support High-quality Care for Canadian Patients with Rheumatoid Arthritis

OBJECTIVE

To develop a Canadian Rheumatoid Arthritis Core Clinical Dataset (CAN-RACCD) to standardize documentation encouraging high-quality care.

METHODS

A set of candidate elements was drafted through meetings with 27 rheumatologists, researchers, and patients, and supplemented with focused literature reviews. A 3-round online-modified Delphi consensus process was held with rheumatologists (n = 26), allied health professionals (n = 7), and patients (n = 4); for the remainder there was no demographic information. Participants rated both the importance and feasibility of documenting candidate elements on a Likert scale of 1-9, contributed to an online moderated discussion, and re-rated the elements for inclusion in the CAN-RACCD. Elements were included in the final set if importance and feasibility ratings had a median score of ≥ 6.5 and there was no disagreement among participants.

RESULTS

Fifty-five individual elements in 10 subgroups were proposed to the Delphi participants: measures of RA disease activity; dates to calculate waiting times, disease duration, and disease-modifying antirheumatic drug start; comorbidities; smoking status; patient-reported pain and fatigue; physical function; laboratory and radiographic investigations; medications; clinical characteristics; and vaccines. All groups were included in the final set, with the exception of vaccination status. Additionally, 3 individual elements from the smoking subgroup were eliminated with a recommendation to record smoking status as never/ever/current, and 2 elements relating to coping and effect of fatigue were eliminated due to low feasibility and importance ratings.

CONCLUSION

The CAN-RACCD stands as a national recommendation on which data elements should be routinely collected in clinical practice to monitor and support high-quality RA care.

Ethics and Science in the Participatory Era: A Vignette-Based Delphi Study

Participatory researchers rely on relationship-based ethical considerations to address ethical challenges, but little is known about how these considerations shape day-to-day decision-making. We presented vignettes describing ethical dilemmas to health researchers with varying degrees of experience in participatory research in an online modified-Delphi panel. Panelists rated potential solutions to dilemmas and provided rationales for choosing solutions in text boxes and discussion boards. Panelists agreed on a preferred solution for three of five vignettes. Panelists indicated that ethical research decision-making should be inclusive of diverse perspectives. Panelists prioritized scientific goals above participatory ones and viewed tenets of scientific validity as value-free. Panelists indicated that participatory research ethics would be difficult to codify, but that vignette-based learning could support didactic objectives.

Using Stakeholder Input to Inform an Innovative Research and Policy Initiative to Improve Depression in Safety Net Communities

THE PROBLEM

Depression quality improvement programs based on chronic disease management models have been shown to improve depression outcomes. Nonetheless, access to and the use of such programs is limited in minority, under-resourced communities.

PURPOSE OF ARTICLE

We report on the outcomes of a Delphi-based consensus exercise conducted by our partnership at a community-wide conference in Los Angeles. Participants identified and prioritized the needs of depressed individuals that should be addressed in a county-wide Health Neighborhood Initiative designed to increase existing mental health, substance use, healthcare, and social services for individuals with low socioeconomic position.

KEY POINTS

Participants agreed that housing is the number one priority. Delphi results also illustrate the importance of addressing social, spiritual, and healthcare access needs of depressed individuals.

CONCLUSIONS

Our study shows how to systematically engage community-based organizations, patients, families, and community members in the process of improving the design of community-wide health policy initiatives.

Comparative analysis of stakeholder experiences with an online approach to prioritizing patient-centered research topics

OBJECTIVE

Little evidence exists about effective and scalable methods for meaningful stakeholder engagement in research. We explored patient/caregiver experiences with a high-tech online engagement approach for patient-centered research prioritization, compared their experiences with those of professional stakeholders, and identified factors associated with favorable participant experiences.

METHODS

We conducted 8 online modified-Delphi (OMD) panels. Panelists participated in 2 rating rounds with a statistical feedback/online discussion round in between. Panels focused on weight management/obesity, heart failure, and Kawasaki disease. We recruited a convenience sample of adults with any of the 3 conditions (or parents/guardians of Kawasaki disease patients), clinicians, and researchers. Measures included self-reported willingness to use OMD again, the panelists' study participation and online discussion experiences, the system's perceived ease of use, and active engagement metrics.

RESULTS

Out of 349 panelists, 292 (84%) completed the study. Of those, 46% were patients, 36% were clinicians, and 19% were researchers. In multivariate models, patients were not significantly more actively engaged (Odds ratio (OR) = 1.69, 95% confidence interval (CI), 0.94-3.05) but had more favorable study participation (β = 0.49; P  ≤ .05) and online discussion (β = 0.18; P  ≤ .05) experiences and were more willing to use OMD again (β = 0.36; P  ≤ .05), compared to professional stakeholders. Positive perceptions of the OMD system's ease of use (β = 0.16; P  ≤ .05) and favorable study participation (β = 0.26; P  ≤ .05) and online discussion (β = 0.57; P  ≤ .05) experiences were also associated with increased willingness to use OMD in the future. Active engagement was not associated with online experience indices or willingness to use OMD again.

CONCLUSION

Online approaches to engaging large numbers of stakeholders are a promising and efficient adjunct to in-person meetings.

Engaging Patients and Caregivers Managing Rare Diseases to Improve the Methods of Clinical Guideline Development: A Research Protocol

BACKGROUND

Clinical guidelines provide systematically developed recommendations for deciding on appropriate health care options for specific conditions and clinical circumstances. Up until recently, patients and caregivers have rarely been included in the process of developing care guidelines.

OBJECTIVE

This project will develop and test a new online method for including patients and their caregivers in this process using Duchenne muscular dystrophy (DMD) care guidelines as an example. The new method will mirror and complement the RAND/UCLA Appropriateness Method (RAM)-the gold standard approach for conducting clinical expert panels that uses a modified Delphi format. RAM is often used in clinical guideline development to determine care appropriateness and necessity in situations where existing clinical evidence is uncertain, weak, or unavailable.

METHODS

To develop the new method for engaging patients and their caregivers in guideline development, we will first conduct interviews with experts on RAM, guideline development, patient engagement, and patient-centeredness and engage with Duchenne patients and caregivers to identify how RAM should be modified for the purposes of patient engagement and what rating criteria should patients and caregivers use to provide their input during the process of guideline development. Once the new method is piloted, we will test it by conducting two concurrently run patient/caregiver panels that will rate patient-centeredness of a subset of DMD care management recommendations already deemed clinically appropriate and necessary. The ExpertLens™ system-a previously evaluated online modified Delphi system that combines two rounds of rating with a round of feedback and moderated online discussions-will be used to conduct these panels. In addition to developing and testing the new engagement method, we will work with the members of our project's Advisory Board to generate a list of best practices for enhancing the level of patient and caregiver involvement in the guideline development process. We will solicit input on these best practice from Duchenne patients, caregivers, and clinicians by conducting a series of round-table discussions and making a presentation at an annual conference on Duchenne.

RESULTS

The study protocol was reviewed by RAND's Human Subjects Protection Committee, which determined it to be exempt from review. Interviews with RAM experts have been completed. The projected study completion date is May 2020.

CONCLUSIONS

We expect that the new method will make it easier to engage large numbers of patients and caregivers in the process of guideline development in a rigorous and culturally appropriate manner that is consistent with the way clinicians participate in guideline development. Moreover, this project will develop best practices that could help involve patients and caregivers in the clinical guideline development process in other clinical areas, thereby facilitating the work of guideline developers.

Does a quality improvement campaign accelerate take-up of new evidence? A ten-state cluster-randomized controlled trial of the IHI's Project JOINTS

Background

A decade ago, the Institute for Healthcare Improvement pioneered a quality improvement (QI) campaign, leveraging organizational and personal social networks to disseminate new practices. There have been few rigorous studies of the QI campaign approach.

Methods

Project JOINTS (Joining Organizations IN Tackling SSIs) engaged a network of state-based organizations and professionals in a 6-month QI campaign promoting adherence to three new evidence-based practices known to reduce the risk of infection after joint replacement. We conducted a cluster-randomized trial including ten states (five campaign states and five non-campaign states) with 188 hospitals providing joint replacement to Medicare. We measured adherence to the evidence-based practices before and after the campaign using a survey of surgical staff and a difference-in-difference design with multivariable adjustment to compare adherence to each of the relevant practices and an all-or-none composite measure of the three new practices.

Results

In the campaign states, there were statistically significant increases in adherence to the three new evidence-based practices promoted by the campaign. Compared to the non-campaign states, the relative increase in adherence to the three new practices in the campaign states ranged between 1.9 and 15.9 percentage points, but only one of these changes (pre-operative nasal screening for Staphylococcus aureus carriage and decolonization prior to surgery) was statistically significant (p < 0.05). On the all-or-none composite measure, adherence to all three evidence-based practices increased from 19.6 to 37.9% in the campaign states, but declined slightly in the comparison states, yielding a relative increase of 23 percentage points (p = 0.004). In the non-campaign states, changes in adherence were not statistically significant.

Conclusions

Within 6 months, in a cluster-randomized trial, a multi-state campaign targeting hospitals and professionals involved in surgical care and infection control was associated with an increase in adherence to evidence-based practices that can reduce surgical site infection.

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-017-0579-7) contains supplementary material, which is available to authorized users.

Acceptability of an online modified Delphi panel approach for developing health services performance measures: results from 3 panels on arthritis research

RATIONALE, AIMS, AND OBJECTIVES

Online modified Delphi (OMD) panel approaches can be used to engage large and diverse groups of clinical experts and stakeholders in developing health services performance measures. Such approaches are increasing in popularity among health researchers. However, information about their acceptability to participating experts and stakeholders is lacking but important to determine before recommending widespread use of online approaches. Therefore, the objective of this paper is to explore acceptability of the OMD panel approach from the participants' perspective.

METHOD

We use data from participants in three OMD panels designed to develop performance measures for use in arthritis research and quality improvement efforts. At the end of each online panel, we surveyed clinical experts and stakeholders who shared their experiences with the OMD process by answering 13 close-ended questions using 7-point Likert-type scales. A mean of 5 or higher on a given question was treated as an indication of acceptability.

RESULTS

Ninety-eight clinical experts and stakeholders (92% participation rate) answered survey questions about the online process. They considered the OMD panel approach to be acceptable, particularly the ease of using the online system (mean = 5.3, standard deviation = 1.3) and the understanding gained from online discussions (mean = 5.2, standard deviation = 1.0). Participants also felt that participation in the Delphi study was interesting (mean = 5.6, standard deviation =1.1).

CONCLUSION

These findings illustrate likely acceptability and a potential for a more widespread use of OMD panel approaches by stakeholders in developing health services performance measures.