Quantitative Benefit-Risk Assessment in Medical Product Decision Making: A Good Practices Report of an ISPOR Task Force

Patient Engagement and Patient Experience Data in Regulatory Review and Health Technology Assessment: Where Are We Today?

BACKGROUND

As healthcare stakeholders aim to support patient-centered care, patients play an increasingly important role in pharmaceutical and medical technology development and healthcare decision-making. Patient engagement (PE), patient experience data (PED), and meaningful integration of PE to enrich PED have been evolving rapidly. This landscape review focuses on emerging PE/PED practices and guidelines in 2023.

METHODS

References published between January-December 2023 on the use of PE and PED from health technology assessment (HTA) and regulatory bodies in different countries, three peer-reviewed journals, and referred resources from collaborators were analyzed. These references were compared with those in our previous publication (August 2021-January 2023, 17-month period).

RESULTS

Overall, 28 references from HTA/regulatory bodies, 26 from peer-reviewed articles, and 17 referred resources were identified. Eight references on PE and PED integration (PE + PED) were identified in 2023 from HTA/regulatory bodies, compared with none in the previous 17-month analysis. Emerging trends on the role of PE, PED, and real-world evidence in HTA/regulatory deliberations, transparency and geographic variations in the use of such evidence and practices, and gaps thereof have been highlighted.

CONCLUSIONS

The increase in PE, PED, and PE + PED references worldwide in 2023 versus the prior 17-month analysis suggests accelerated adoption of PE + PED practices. However, a need remains for comprehensive, actionable guidance on best practices for use of PE and PED for harmonization and incorporation into HTA/regulatory processes. Patient input-essential for evidence-based decision-making-provides valuable insights that enhance care quality, treatment relevance and effectiveness, and builds trust and sustainability.

Quantifying Patient Preferences for Risk Tolerance With Novel Dual Biologic Therapies for Inflammatory Bowel Disease

INTRODUCTION

Many patients with inflammatory bowel disease (IBD) experience treatment failures despite availability of effective advanced biologic and small-molecule therapies with differing mechanisms of action. Dual biologic therapy (DBT) is being explored to improve efficacy outcomes and address unmet needs in this difficult-to-treat population. This study aimed to understand patient preferences for DBT efficacy and risk tolerance.

METHODS

Built on evidence from existing treatment-preference studies, a focused discrete-choice experiment (DCE) was developed to measure preferences for treatment type, chance of remission, and risk of serious infection. Individuals with a physician-verified diagnosis of Crohn's disease or ulcerative colitis who had failed or were taking advanced therapy were recruited through an IBD registry. DCE responses were analyzed using a fully correlated random parameters logit model.

RESULTS

The DCE survey was completed by 280 respondents. The majority of respondents were White, female; had been previously hospitalized for IBD; and were receiving biologic monotherapy for their IBD disease. There was no meaningful difference in preference for DBT or monotherapy (P = 0.25), while there was strong preference to avoid corticosteroids (P < 0.001). To improve from a 50% to 70% in chance of remission, respondents would accept up to a 17.5% (95% confidence interval 17.0%-18.0%) risk of serious infection.

DISCUSSION

The findings suggest that patients prefer safe and efficacious treatments and DBT may be an acceptable option for those who have failed an advanced therapy for IBD. We demonstrate the value of building on existing evidence and designing efficient DCE studies to address knowledge gaps to improve IBD care.

What Next for the Science of Patient Preference? Interoperability, Standardization, and Transferability

Using patient preference information (PPI) to incorporate patient voices into the drug development lifecycle can help align therapies with the needs and values of patients. However, several barriers have limited the use of PPI, including a lack of clarity on its use by decision-makers, a need for greater decision-maker trust in PPI, and a lack of time, budgets, and access to specialist expertise. The value proposition for PPI could be enhanced by making it FAIR: Findable, Accessible, Interoperable, and Reusable. To support the development of a research agenda to deliver FAIR PPI, we reviewed related endeavors in the development of repositories of existing studies, disease models, benefit transfer, and common data standards. We concluded that developing FAIR PPI would require advances in the science of PPI, including the establishment of a consortium, mirroring the Clinical Data Interchange Standards Consortium (CDISC) or Observational Medical Outcomes Partnership (OPOM), to develop PPI data standards, and research into the sources of variation in patient preferences. This will require the science of PPI to graduate from being a body of empirical observations to developing theories that explain variations in patient preferences, simultaneously driving both efficiency in the generation of PPI and trust in PPI.

Patient Preferences for First-Line Treatment of Locally Advanced or Metastatic Urothelial Carcinoma: An Application of Multidimensional Thresholding

OBJECTIVES

Patient preferences have the potential to influence the development of new treatments for locally advanced/metastatic urothelial carcinoma (la/mUC), and therefore we explored how patients with la/mUC value different attributes of first-line treatments.

METHODS

An online preference survey and multidimensional thresholding (MDT) exercise were developed following a targeted literature review and qualitative interviews with physicians, patients with la/mUC, and their caregivers. Treatment attributes included two benefits (overall response rate [ORR], pain related to bladder cancer [scored 0-100; 100 being the worst pain possible]) and four treatment-related risks (peripheral neuropathy, severe side effects, mild to moderate nausea, mild to moderate skin reactions). A Dirichlet regression was used to estimate average preference weights. Marginal utility and the reduction in ORR that patients would accept in exchange for a 10-point decrease or a 10% decrease in other attributes were calculated.

RESULTS

A total of 100 patients were recruited and self-completed the survey and MDT. Mean patient age was 64.9 years (standard deviation, 7.6), 54% were female, and 38% identified as white. All included treatment attributes had a statistically significant impact on preferences. Changes in ORR had the largest impact, followed by cancer-related pain and treatment-related risks. Patients were willing to accept an 8.4% decrease in ORR to reduce their pain level by 10 points or a 7.8% decrease in ORR to reduce the risk of peripheral neuropathy by 10%. For a 10% decrease in severe side effects, mild to moderate nausea, or skin reaction, patients would accept decreases in ORR of 5.5%, 3.7%, or 3.4%, respectively.

CONCLUSIONS

Of the attributes tested, changes in ORR were most important to patients. Patients made tradeoffs between treatment attributes indicating that a lower ORR may be acceptable for an improvement in other attributes such as reduced cancer-related pain or the risk of treatment-related adverse events.

Consideration of Cybersecurity Risks in the Benefit-Risk Analysis of Medical Devices: Scoping Review

BACKGROUND

The integration of connected medical devices (MDs) into health care brings benefits but also introduces new, often challenging-to-assess risks related to cybersecurity, which have the potential to harm patients. Current regulations in the European Union and the United States mandate the consideration of these risks in the benefit-risk analysis (BRA) required for MD approval. This important step in the approval process weighs all the defined benefits of a device with its anticipated risks to ensure that the product provides a positive argument for use. However, there is limited guidance on how cybersecurity risks should be systematically evaluated and incorporated into the BRA.

OBJECTIVE

This scoping review aimed to identify current legal frameworks, guidelines, and standards in the United States, Canada, South Korea, Singapore, Australia, the United Kingdom, and the European Union on how cybersecurity risks should be considered in the BRA of MDs.

METHODS

This scoping review followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) framework. A systematic literature search of 10 databases was conducted in two phases on July 3, 2024 and September 30, 2024, including the guidance databases of the Food and Drug Administration, the Medical Device Coordination Group, and other International Medical Device Regulators Forum members; the International Medical Device Regulators Forum database; PubMed; and Scopus. Search terms included "cybersecurity," "security," "benefit/risk," "benefit-risk," and "risk-benefit." Additional references were identified via citation searching and expert interviews. Inclusion criteria were met if a document was a guideline or standard in force that provided guidance on the BRA or cybersecurity risks of MDs. Documents were excluded when they were not relevant to MDs, they were limited to a subclass of devices, they were about in vitro diagnostic MDs or investigational devices, and the content of the source was insufficient to undertake a scientific analysis. Data were extracted and analyzed using MAXQDA 2022, and the findings were narratively summarized and visualized in figures and tables.

RESULTS

The search identified 150 documents, with 34 (22.7%) meeting the inclusion criteria. These 34 documents included 4 (12%) regulations, 5 (15%) standards, 6 (18%) technical reports, and 19 (56%) guidance documents. While cybersecurity risks were acknowledged in most documents, detailed methods for their integration into the BRA were lacking. Some standards and guidelines provided examples of how to consider cybersecurity risks in the BRA, but a comprehensive and standardized approach was lacking.

CONCLUSIONS

This review highlights a substantial gap between the recognition of cybersecurity risks in MDs and the guidance on their incorporation into the BRA. Standardized frameworks are needed to provide clear methods for evaluating cybersecurity risks and their impact on the safety and security of MDs.

Participant Engagement and Preference Study for Clinical Outcomes Associated With Atrial Fibrillation: The PEARL-AF Study

Background

Quantifying patients' preferences for health outcomes associated with atrial fibrillation (AF) and its treatments offers a replicable approach to considering the patient perspective in regulatory decision-making.

Objective

The authors conducted a preference survey to estimate the relative importance of AF-related events for use in clinical trial analyses to estimate net health benefits with anticoagulants.

Methods

The survey included nontechnical descriptions of three severities of stroke, systemic embolism, myocardial infarction (MI) with or without subsequent heart failure (HF), major bleeding, clinically relevant nonmajor bleeding, and death. A best-worst scaling question format was used in which patients were shown 10 sets of four events and asked to select what they considered to be most and least serious.

Results

One thousand twenty-eight patients, mean age 69.2 years, 40.4% female, completed the survey. Best-worst scaling importance weights were significantly different across all events except between major bleeding and MI with HF. Death was considered the most serious (reweighted to 1), followed by severe disabling stroke (0.83), then major bleeding (0.53) or MI with HF (0.50), moderate-severity stroke (0.28) and systemic embolism (0.13). Clinically relevant nonmajor bleeding, MI without HF, and minor stroke (0.10, 0.06, and 0.04, respectively) were considered least serious. Events ordered by importance were consistent across age, sex, and race, but relative weights across events varied by sex and race.

Conclusions

Patients expressed relatively high levels of concern about major bleeding compared to moderate-severity stroke or systemic embolism, endpoints frequently used in AF trials. Estimated weights could be used in patient-centered net-benefit determinations for AF therapies.

Key Opinion Leaders' Interviews to Inform the Future of Benefit-Risk Planning in the Medical Total Product Life Cycle of Global Pharmaceutical and Medical Device Organizations

BACKGROUND AND OBJECTIVES

Key opinion leader (KOL) interviews were conducted by the Benefit-Risk Assessment Planning (BRAP) Taskforce to seek expert opinion mainly from industry and regulatory bodies, about the current status and future direction of benefit-risk assessment (BRA) planning in the lifecycle of medical product development. The findings from these interviews are intended to help communication concerning planning for BRA between industry and regulators and shape future guidance.

METHODS

Key opinion leader interviews consisted of 5 questions related to BRA planning, which were administered to volunteers (mainly clinicians and statisticians) within a pool of experienced pharmaceutical and medical device professionals representing academia, industry, regulatory agencies and a patient group. The interviewees' responses to the 5 questions were summarized. To analyze the qualitative data, a Coding System was developed to label themes arising from the interviews. The key findings from the interviews were summarized into a Master Template. A quantitative analysis based on descriptive statistics was also conducted.

RESULTS

Of the 27 interviewees, there were 11 professionals from regulatory agencies, 11 from industry, 4 from academia and 1 from a patient advocacy group. Key findings based on the comments provided by 48% of the interviewees indicated the need of incorporating BRA into other (e.g., existing) processes with the importance of alignment between processes being stressed in the comments provided by 59% of the interviewees. Commencing BRA early in the product lifecycle was emphasized in comments provided by 44% of the interviewees. Among other needs identified were an appropriate contextualization of benefits and risks (based on comments provided by 41% of interviewees) through adoption of an integrated approach with structured support by regulatory agencies and a need for understanding the audience with better communication of benefit-risk (BR) among all stakeholders (based on comments provided by 44% of the interviewees). Almost all comments provided by interviewees (96%) highlighted the importance of utilizing patient experience/preference to guide new product development and BRA. Comments provided by 74% of the interviewees expressed the need to understand patient tolerance for risk and trade-offs, with a majority (78%) of interviewees highlighting how to gather information, and 59% stressing the need for the selection and development of appropriate methodologies as important considerations for enhancing the quality and relevance of the data collected from patients.

CONCLUSIONS

Interviewees indicated that BRA should commence early in the medical product development and inform decision-making throughout the product lifecycle. Better planning and integration of BRA into existing processes within industry would be valuable. The importance of incorporating the patient voice into BRA and medical product development was emphasized. Other key findings from the KOL interviews included a need for improved communication of BR information, and establishment of methodologies for performing BRA and soliciting patient input.

A Targeted Review of Worldwide Indirect Treatment Comparison Guidelines and Best Practices

OBJECTIVES

Controls and governance over the methodology and reporting of indirect treatment comparisons (ITCs) have been introduced to minimize bias and ensure scientific credibility and transparency in healthcare decision making. The objective of this study was to highlight ITC techniques that are key to conducting objective and analytically sound analyses and to ascertain circumstantial suitability of ITCs as a source of comparative evidence for healthcare interventions.

METHODS

Ovid MEDLINE was searched from January 2010 through August 2023 to identify publicly available ITC-related documents (ie, guidelines and best practices) in the English language. This was supplemented with hand searches of websites of various international organizations, regulatory agencies, and reimbursement agencies of Europe, North America, and Asia-Pacific. The jurisdiction-specific ITC methodology and reporting recommendations were reviewed.

RESULTS

Sixty-eight guidelines from 10 authorities worldwide were included for synthesis. Many of the included guidelines were updated within the last 5 years and commonly cited the absence of direct comparative studies as primary justification for using ITCs. Most jurisdictions favored population-adjusted or anchored ITC techniques opposed to naive comparisons. Recommendations on the reporting and presentation of these ITCs varied across authorities; however, there was some overlap among the key elements.

CONCLUSIONS

Given the challenges of conducting head-to-head randomized controlled trials, comparative data from ITCs offer valuable insights into clinical-effectiveness. As such, multiple ITC guidelines have emerged worldwide. According to the most recent versions of the guidelines, the suitability and subsequent acceptability of the ITC technique used depends on the data sources, available evidence, and magnitude of benefit/uncertainty.

A Survey to Assess the Current Status of Structured Benefit-Risk Assessment in the Global Drug and Medical Device Industry

BACKGROUND

This industry survey was conducted to gain insight into the ways structured Benefit-Risk assessment (sBRA) of medical products is approached across drug or medical device developing companies, including frameworks and methods that are currently used and areas where future work is being planned.

METHODS

A survey containing 28 questions covering five key areas of sBRA was set-up and shared with representatives from the participating companies. Each company was asked to complete a single survey response including inputs across the company's multidisciplinary key representatives involved in benefit-risk assessment.

RESULTS

Of the 26 participating companies, 21 (81%) are conducting sBRA. Considering these 21 qualitative frameworks were used by almost every company (19, 90%), while only 12 (57%) have used a quantitative method. Many companies have sBRA training (17, 81%), document templates (16,76%), Standard Operating Procedures (SOPs)/checklists (13, 62%), and /or best practice manuals/examples (12,57%) available. Considering all 26 companies Software tools (15, 58%) and BR planning documents (11,42%) were identified as areas into which many companies intend to put effort.

CONCLUSIONS

The industry survey confirmed a wide usage of sBRA by many companies involved in research and development. Nevertheless, sBRA is evolving and several future opportunities like the implementation of visualization tools were identified by the representatives of the pharmaceutical companies. Finally, challenges like the cross-functional comprehension of the added value of sBRA are still seen.

Methodological guidelines and publications of benefit-risk assessment for health technology assessment: a scoping review

OBJECTIVES

To map the available methodological guidelines and documents for conducting and reporting benefit-risk assessment (BRA) during health technologies' life cycle; and to identify methodological guidelines for BRA that could serve as the basis for the development of a BRA guideline for the context of health technology assessment (HTA) in Brazil.

DESIGN

Scoping review.

METHODS

Searches were conducted in three main sources up to March 2023: (1) electronic databases; (2) grey literature (48 HTA and regulatory organisations) and (3) manual search and contacting experts. We included methodological guidelines or publications presenting methods for conducting or reporting BRA of any type of health technologies in any context of the technology's life cycle. Selection process and data charting were conducted by independent reviewers. We provided a structured narrative synthesis of the findings.

RESULTS

From the 83 eligible documents, six were produced in the HTA context, 30 in the regulatory and 35 involved guidance for BRA throughout the technology's life cycle. We identified 129 methodological approaches for BRA in the documents. The most commonly referred to descriptive frameworks were the Problem, Objectives, Alternatives, Consequences, Trade-offs, Uncertainty, Risk and Linked decisions and the Benefit-Risk Action Team. Multicriteria decision analysis was the most commonly cited quantitative framework. We also identified the most cited metric indices, estimation and utility survey techniques that could be used for BRA.

CONCLUSIONS

Methods for BRA in HTA are less established. The findings of this review, however, will support and inform the elaboration of the Brazilian methodological guideline on BRA for HTA.

TRIAL REGISTRATION NUMBER

https://doi.org/10.17605/OSF.IO/69T3V.

Multidimensional Thresholding for Individual-Level Preference Elicitation

OBJECTIVES

Multiple methods are available for collecting health preference information. However, information on the design and analysis of novel methods is limited. This article aims to provide the first introduction into the design and analysis of multidimensional thresholding (MDT).

METHODS

We introduce MDT as a 2-step approach: First, participants rank the largest possible improvements in all considered attributes by their importance. Second, participants complete a series of systematically combined trade-off questions. Hit-and-Run sampling is used for obtaining preference weights. We also use a computational experiment to compare different MDT designs.

RESULTS

The outlined MDT can generate preference information suitable for specifying a multiattribute utility function at the individual level. The computational experiment demonstrates the method's ability to recover preference weights at a high level of precision. While all designs in the computation experiment perform comparably well on average, the design outlined in the paper stands out with a high level of precision even if differences in relative attribute importance are large.

CONCLUSION

MDT is suitable for preference elicitation, in particular if sample sizes are small. Future research should help improve the methods (e.g., remove the need for an initial ranking) to increase the potential reach of MDT.

Two Methods, One Story? Comparing Results of a Choice Experiment and Multidimensional Thresholding From a Clinician Preference Study in Aneurysmal Subarachnoid Hemorrhage

OBJECTIVES

An increasing number of methods are used to elicit health preference information. It is unclear whether different elicitation methods produce similar results and policy advice. Here, we compared the results from a discrete choice experiment (DCE) and multidimensional thresholding (MDT) that were conducted in the same sample.

METHODS

Clinicians (N = 350) completed a DCE and MDT to elicit their preferences for 4 attributes related to the medical management of subarachnoid hemorrhage after aneurysm repair. Preference weights were compared between the DCE and MDT using a complete combinatorial convolution test. Additionally, data from the DCE and MDT were used to compute preference-based net treatment values for 16 hypothetical treatment profiles versus 1000 simulated comparators. The implied treatment recommendations were compared between the DCE and MDT.

RESULTS

Preference weight distributions and median weights did not differ significantly between the DCE and MDT for any attribute: likelihood of delayed cerebral ischemia (medians 0.48 vs 0.40; P = .41), risk of lung complications (medians 0.27 vs 0.30; P = .52), risk of hypotension (medians 0.10 vs 0.11; P = .55), and risk of anemia (medians 0.07 vs 0.07; P = .50). The DCE and MDT produced similar treatment net value distributions (P > .05) and implied the same treatment recommendations in 82.3% of cases.

CONCLUSIONS

The DCE and MDT elicited similar preference distributions and produced the same treatment recommendations for most tested cases. However, the share of people supporting the average treatment recommendation differed. More research is needed to determine how these findings would compare with those in other populations (in particular, patients) and applications.

Decision tool of medical endoscope maintenance service in Chinese hospitals: a conjoint analysis

BACKGROUND

Medical devices are instruments, apparatus, appliances, software, implants, reagents, materials or other articles that are intended for use in the treatment or diagnosis of disease or injury in humans. Concerning medical endoscope devices, which enable doctors to observe and manipulate the area under examination through a puncture hole in the body cavity or organ, hospitals predominantly consider the quality and cost of maintenance services when making their selection. The effective and efficient provision of maintenance services plays a crucial role in ensuring cost-effective and high-quality management of medical devices. In this study, we have developed an innovative decision tool that analyzed key factors impacting the choice of medical devices' maintenance service. This tool assists hospitals in evaluating and selecting appropriate maintenance services for medical device, specifically endoscopy devices. Moreover, it also serves as a valuable resource for manufacturers and suppliers to enhance their after-sales service offerings.

METHODS

A cross-sectional survey was undertaken in 50 Chinese hospitals, including primary and tertiary hospitals. Moreover, 56 medical staff and 65 medical engineers were recruited from 50 Chinese hospitals to participate the survey. A comprehensive set of factors were defined and investigated. Conjoint analysis and orthogonal design were used for survey design and statistical analysis.

RESULTS

Factors importance and utility values of decision-making factors were analyzed at the aggregate, occupation, and medical institution levels. (1) At the aggregate level, the most critical factor is "maintenance response" and the least important one is "maintenance efficiency". (2) At the occupation level, medical staff paid more attention to "maintenance response" while medical engineers paid more attention to "maintenance quality". (3) At the medical institution level, Primary hospitals paid more attention to "maintenance price", while tertiary hospitals paid more attention to "maintenance quality".

CONCLUSIONS

In general, this study provides a more scientific decision-making tool to both hospitals in choosing maintenance service for medical device such as endoscopy, and it also helps manufacturers and suppliers improve the after-sales service.

Illustrating Emerging Good Practices for Quantitative Benefit-Risk Assessment: A Hypothetical Case Study of Systemic Biologic Treatments for Plaque Psoriasis

OBJECTIVES

Quantitative benefit-risk assessment (qBRA) is a structured process to evaluate the benefit-risk balance of treatment options to support decision making. The ISPOR qBRA Task Force was recently established to provide recommendations for the design, conduct, and reporting of qBRA. This report presents a hypothetical case study illustrating how to apply the Task Force's recommendations toward a qBRA to inform the benefit-risk assessment of brodalumab at the time of initial marketing approval. The qBRA evaluated 2 dosing regimens of brodalumab (210 mg or 140 mg twice weekly) compared with weight-based dosing of ustekinumab and placebo.

METHODS

We followed the 5 steps recommended by the Task Force. Attributes included treatment response (≥75% improvement in Psoriasis Area and Severity Index), suicidal ideation and behavior, and infections. Performance data were drawn from pivotal clinical trials of brodalumab. The qBRA used multicriteria decision analysis and preference weights from a hypothetical discrete choice experiment. Sensitivity analyses examined the robustness of benefit-risk ranking to uncertainty in clinical effect and preference estimates, consideration of a subgroup (nail psoriasis), and the maintenance phase of treatment (52 weeks instead of 12).

RESULTS

Results from this hypothetical qBRA suggest that brodalumab 210 mg had a more favorable benefit-risk profile compared with ustekinumab and placebo. Ranking of brodalumab compared with ustekinumab was dependent on brodalumab's dose. Sensitivity analyses demonstrated robustness of benefit-risk ranking to uncertainty in clinical effect and preference estimates, as well as choice of attributes and length of follow-up.

CONCLUSION

This case study demonstrates how to implement the ISPOR Task Force's good practice recommendations on qBRA.