Evaluating genomic tests from bench to bedside: a practical framework

Acceptability of magnetic resonance imaging for prostate cancer diagnosis with patients and GPs: a qualitative interview study

BACKGROUND

Magnetic resonance imaging (MRI) of the prostate is a new, more accurate, non-invasive test for prostate cancer diagnosis.

AIM

To understand the acceptability of MRI for patients and GPs for prostate cancer diagnosis.

DESIGN AND SETTING

Qualitative study of men who had undergone a prostate MRI for possible prostate cancer, and GPs who had referred at least one man for possible prostate cancer in the previous 12 months in West London and Devon.

METHOD

Semi-structured interviews, conducted in person or via telephone, were audio-recorded and transcribed verbatim. Deductive thematic analysis was undertaken using Sekhon's Theoretical Framework of Acceptability, retrospectively for patients and prospectively for GPs.

RESULTS

Twenty-two men (12 from Devon, age range 47-80 years), two patients' partners, and 10 GPs (6 female, age range 36-55 years) were interviewed. Prostate MRI was broadly acceptable for most patient participants, and they reported that it was not a significant undertaking to complete the scan. GPs were more varied in their views on prostate MRI, with a broad spectrum of knowledge and understanding of prostate MRI. Some GPs expressed concerns about additional clinical responsibility and local availability of MRI if direct access to prostate MRI in primary care were to be introduced.

CONCLUSION

Prostate MRI appears to be acceptable to patients. Some differences were found between patients in London and Devon, mainly around burden of testing and opportunity costs. Further exploration of GPs' knowledge and understanding of prostate MRI could inform future initiatives to widen access to diagnostic testing in primary care.

Patient-centred outcomes of imaging tests: recommendations for patients, clinicians and researchers

BACKGROUND

Imaging tests are one of the most frequently used diagnostic modalities in healthcare, but the benefits of their direct impacts on clinical decision-making have been countered by concerns that they can be overused. Assessing the relative value of imaging tests has largely focused on measures of test accuracy, which overlooks more comprehensive benefits and risks of imaging tests, particularly their impact on patient-centred outcomes (PCOs). We present the findings of the Patient Reported Outcomes of Diagnostics (PROD) research study in response to a methodological gap in the area of diagnostic test comparative effectiveness research.

METHODS

Over a 3-year period, the PROD Study engaged with multiple stakeholders to identify existing conceptual models related to PCOs for imaging testing, conducted primary research and evidence synthesis, and developed consensus recommendations to describe and categorise PCOs related to imaging testing.

RESULTS

The PROD framework categorises PCOs from imaging studies within four main domains: information or knowledge yielded, physical impact, emotional outcomes and test burden. PCOs interact with each other and influence effects across domains, and can be modified by factors related to the patient, clinical situation, healthcare team and the testing environment.

CONCLUSIONS

Using PCOs to inform healthcare decision-making will require ways of collating and presenting information on PCOs in ways that can inform patient-provider decision-making, and developing methods to determine the relative importance of outcomes (including test accuracy) to one another.

Development and evaluation of safety and effectiveness of novel cancer screening tests for routine clinical use with applications to multicancer detection technologies

Multicancer screening is a promising approach to improving the detection of preclinical disease, but current technologies have limited ability to identify precursor or early stage lesions, and approaches for developing the evidentiary chain are unclear. Frameworks to enable development and evaluation from discovery through evidence of clinical effectiveness are discussed.

A Systematic Review and Recommendations Around Frameworks for Evaluating Scientific Validity in Nutritional Genomics

Background: There is a significant lack of consistency used to determine the scientific validity of nutrigenetic research. The aims of this study were to examine existing frameworks used for determining scientific validity in nutrition and/or genetics and to determine which framework would be most appropriate to evaluate scientific validity in nutrigenetics in the future.

Methods: A systematic review (PROSPERO registration: CRD42021261948) was conducted up until July 2021 using Medline, Embase, and Web of Science, with articles screened in duplicate. Gray literature searches were also conducted (June-July 2021), and reference lists of two relevant review articles were screened. Included articles provided the complete methods for a framework that has been used to evaluate scientific validity in nutrition and/or genetics. Articles were excluded if they provided a framework for evaluating health services/systems more broadly. Citing articles of the included articles were then screened in Google Scholar to determine if the framework had been used in nutrition or genetics, or both; frameworks that had not were excluded. Summary tables were piloted in duplicate and revised accordingly prior to synthesizing all included articles. Frameworks were critically appraised for their applicability to nutrigenetic scientific validity assessment using a predetermined categorization matrix, which included key factors deemed important by an expert panel for assessing scientific validity in nutrigenetics.

Results: Upon screening 3,931 articles, a total of 49 articles representing 41 total frameworks, were included in the final analysis (19 used in genetics, 9 used in nutrition, and 13 used in both). Factors deemed important for evaluating nutrigenetic evidence related to study design and quality, generalizability, directness, consistency, precision, confounding, effect size, biological plausibility, publication/funding bias, allele and nutrient dose-response, and summary levels of evidence. Frameworks varied in the components of their scientific validity assessment, with most assessing study quality. Consideration of biological plausibility was more common in frameworks used in genetics. Dose-response effects were rarely considered. Two included frameworks incorporated all but one predetermined key factor important for nutrigenetic scientific validity assessment.

Discussion/Conclusions: A single existing framework was highlighted as optimal for the rigorous evaluation of scientific validity in nutritional genomics, and minor modifications are proposed to strengthen it further.

Systematic Review Registration:https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=261948, PROSPERO [CRD42021261948].

A Systematic Review of the Value Assessment Frameworks Used within Health Technology Assessment of Omics Technologies and Their Actual Adoption from HTA Agencies

BACKGROUND

Omics technologies, enabling the measurements of genes (genomics), mRNA (transcriptomics), proteins (proteomics) and metabolites (metabolomics), are valuable tools for personalized decision-making. We aimed to identify the existing value assessment frameworks used by health technology assessment (HTA) doers for the evaluation of omics technologies through a systematic review.

METHODS

PubMed, Scopus, Embase and Web of Science databases were searched to retrieve potential eligible articles published until 31 May 2020 in English. Additionally, through a desk research in HTA agencies' repositories, we retrieved the published reports on the practical use of these frameworks.

RESULTS

Twenty-three articles were included in the systematic review. Twenty-two frameworks, which addressed genetic and/or genomic technologies, were described. Most of them derived from the ACCE framework and evaluated the domains of analytical validity, clinical validity and clinical utility. We retrieved forty-five reports, which mainly addressed the commercial transcriptomic prognostics and next generation sequencing, and evaluated clinical effectiveness, economic aspects, and description and technical characteristics.

CONCLUSIONS

A value assessment framework for the HTA evaluation of omics technologies is not standardized and accepted, yet. Our work reports that the most evaluated domains are analytical validity, clinical validity and clinical utility and economic aspects.

The development of the Clinician-reported Genetic testing Utility InDEx (C-GUIDE): a novel strategy for measuring the clinical utility of genetic testing

Purpose

Clinical utility describes a genetic test’s value to patients, families, health-care providers, systems, or society. This study aims to define clinical utility from the perspective of clinicians and develop a novel outcome measure that operationalizes this concept.

Methods

Item selection for the Clinician-reported Genetic testing Utility InDEx (C-GUIDE) was informed by a scoping review of the literature. Item reduction was guided by qualitative and quantitative feedback from semistructured interviews and a cross-sectional survey of genetics and nongenetics specialists. Final item selection, index scoring, and structure were guided by feedback from an expert panel of genetics professionals.

Results

A review of 194 publications informed the selection of a preliminary set of 25 items. Feedback from 35 semistructured interviews, 113 surveys, and 11 expert panelists informed the content and wording of C-GUIDE’s final set of 18 items that reflect on the utility of testing related to diagnosis, management, and familial/psychosocial impact. C-GUIDE achieves content and face validity for use in a range of diagnostic genetic testing settings.

Conclusion

Work to establish reliability and construct validity is underway. C-GUIDE will be useful in comparative studies to generate policy-relevant evidence pertaining to the clinical utility of genetic testing across a range of settings.

Evaluating diagnostic strategies for early detection of cancer: the CanTest framework

BACKGROUND

Novel diagnostic triage and testing strategies to support early detection of cancer could improve clinical outcomes. Most apparently promising diagnostic tests ultimately fail because of inadequate performance in real-world, low prevalence populations such as primary care or general community populations. They should therefore be systematically evaluated before implementation to determine whether they lead to earlier detection, are cost-effective, and improve patient safety and quality of care, while minimising over-investigation and over-diagnosis.

METHODS

We performed a systematic scoping review of frameworks for the evaluation of tests and diagnostic approaches.

RESULTS

We identified 16 frameworks: none addressed the entire continuum from test development to impact on diagnosis and patient outcomes in the intended population, nor the way in which tests may be used for triage purposes as part of a wider diagnostic strategy. Informed by these findings, we developed a new framework, the 'CanTest Framework', which proposes five iterative research phases forming a clear translational pathway from new test development to health system implementation and evaluation.

CONCLUSION

This framework is suitable for testing in low prevalence populations, where tests are often applied for triage testing and incorporated into a wider diagnostic strategy. It has relevance for a wide range of stakeholders including patients, policymakers, purchasers, healthcare providers and industry.

Methods for the evaluation of biomarkers in patients with kidney and liver diseases: multicentre research programme including ELUCIDATE RCT

Background

Protein biomarkers with associations with the activity and outcomes of diseases are being identified by modern proteomic technologies. They may be simple, accessible, cheap and safe tests that can inform diagnosis, prognosis, treatment selection, monitoring of disease activity and therapy and may substitute for complex, invasive and expensive tests. However, their potential is not yet being realised.

Design and methods

The study consisted of three workstreams to create a framework for research: workstream 1, methodology – to define current practice and explore methodology innovations for biomarkers for monitoring disease; workstream 2, clinical translation – to create a framework of research practice, high-quality samples and related clinical data to evaluate the validity and clinical utility of protein biomarkers; and workstream 3, the ELF to Uncover Cirrhosis as an Indication for Diagnosis and Action for Treatable Event (ELUCIDATE) randomised controlled trial (RCT) – an exemplar RCT of an established test, the ADVIA Centaur® Enhanced Liver Fibrosis (ELF) test (Siemens Healthcare Diagnostics Ltd, Camberley, UK) [consisting of a panel of three markers – (1) serum hyaluronic acid, (2) amino-terminal propeptide of type III procollagen and (3) tissue inhibitor of metalloproteinase 1], for liver cirrhosis to determine its impact on diagnostic timing and the management of cirrhosis and the process of care and improving outcomes.

Results

The methodology workstream evaluated the quality of recommendations for using prostate-specific antigen to monitor patients, systematically reviewed RCTs of monitoring strategies and reviewed the monitoring biomarker literature and how monitoring can have an impact on outcomes. Simulation studies were conducted to evaluate monitoring and improve the merits of health care. The monitoring biomarker literature is modest and robust conclusions are infrequent. We recommend improvements in research practice. Patients strongly endorsed the need for robust and conclusive research in this area. The clinical translation workstream focused on analytical and clinical validity. Cohorts were established for renal cell carcinoma (RCC) and renal transplantation (RT), with samples and patient data from multiple centres, as a rapid-access resource to evaluate the validity of biomarkers. Candidate biomarkers for RCC and RT were identified from the literature and their quality was evaluated and selected biomarkers were prioritised. The duration of follow-up was a limitation but biomarkers were identified that may be taken forward for clinical utility. In the third workstream, the ELUCIDATE trial registered 1303 patients and randomised 878 patients out of a target of 1000. The trial started late and recruited slowly initially but ultimately recruited with good statistical power to answer the key questions. ELF monitoring altered the patient process of care and may show benefits from the early introduction of interventions with further follow-up. The ELUCIDATE trial was an ‘exemplar’ trial that has demonstrated the challenges of evaluating biomarker strategies in ‘end-to-end’ RCTs and will inform future study designs.

Conclusions

The limitations in the programme were principally that, during the collection and curation of the cohorts of patients with RCC and RT, the pace of discovery of new biomarkers in commercial and non-commercial research was slower than anticipated and so conclusive evaluations using the cohorts are few; however, access to the cohorts will be sustained for future new biomarkers. The ELUCIDATE trial was slow to start and recruit to, with a late surge of recruitment, and so final conclusions about the impact of the ELF test on long-term outcomes await further follow-up. The findings from the three workstreams were used to synthesise a strategy and framework for future biomarker evaluations incorporating innovations in study design, health economics and health informatics.

Trial registration

Current Controlled Trials ISRCTN74815110, UKCRN ID 9954 and UKCRN ID 11930.

Funding

This project was funded by the NIHR Programme Grants for Applied Research programme and will be published in full inProgramme Grants for Applied Research; Vol. 6, No. 3. See the NIHR Journals Library website for further project information.

How is genetic testing evaluated? A systematic review of the literature

Given the rapid development of genetic tests, an assessment of their benefits, risks, and limitations is crucial for public health practice. We performed a systematic review aimed at identifying and comparing the existing evaluation frameworks for genetic tests. We searched PUBMED, SCOPUS, ISI Web of Knowledge, Google Scholar, Google, and gray literature sources for any documents describing such frameworks. We identified 29 evaluation frameworks published between 2000 and 2017, mostly based on the ACCE Framework (n = 13 models), or on the HTA process (n = 6), or both (n = 2). Others refer to the Wilson and Jungner screening criteria (n = 3) or to a mixture of different criteria (n = 5). Due to the widespread use of the ACCE Framework, the most frequently used evaluation criteria are analytic and clinical validity, clinical utility and ethical, legal and social implications. Less attention is given to the context of implementation. An economic dimension is always considered, but not in great detail. Consideration of delivery models, organizational aspects, and consumer viewpoint is often lacking. A deeper analysis of such context-related evaluation dimensions may strengthen a comprehensive evaluation of genetic tests and support the decision-making process.

Mapping the Ethics of Translational Genomics: Situating Return of Results and Navigating the Research-Clinical Divide

Both bioethics and law have governed human genomics by distinguishing research from clinical practice. Yet the rise of translational genomics now makes this traditional dichotomy inadequate. This paper pioneers a new approach to the ethics of translational genomics. It maps the full range of ethical approaches needed, proposes a "layered" approach to determining the ethics framework for projects combining research and clinical care, and clarifies the key role that return of results can play in advancing translation.

Evidence of clinical utility: an unmet need in molecular diagnostics for patients with cancer

This article defines and describes best practices for the academic and business community to generate evidence of clinical utility for cancer molecular diagnostic assays. Beyond analytical and clinical validation, successful demonstration of clinical utility involves developing sufficient evidence to demonstrate that a diagnostic test results in an improvement in patient outcomes. This discussion is complementary to theoretical frameworks described in previously published guidance and literature reports by the U.S. Food and Drug Administration, Centers for Disease Control and Prevention, Institute of Medicine, and Center for Medical Technology Policy, among others. These reports are comprehensive and specifically clarify appropriate clinical use, adoption, and payer reimbursement for assay manufacturers, as well as Clinical Laboratory Improvement Amendments-certified laboratories, including those that develop assays (laboratory developed tests). Practical criteria and steps for establishing clinical utility are crucial to subsequent decisions for reimbursement without which high-performing molecular diagnostics will have limited availability to patients with cancer and fail to translate scientific advances into high-quality and cost-effective cancer care. See all articles in this CCR Focus section, "The Precision Medicine Conundrum: Approaches to Companion Diagnostic Co-development."

Reflections on the cost of "low-cost" whole genome sequencing: framing the health policy debate

The future clinical applications of whole genome sequencing come with speculation and enthusiasm but require careful consideration of the true system costs and health benefits of the clinical uses of this exciting technology.

The cost of whole genome sequencing is dropping rapidly. There has been a great deal of enthusiasm about the potential for this technological advance to transform clinical care. Given the interest and significant investment in genomics, this seems an ideal time to consider what the evidence tells us about potential benefits and harms, particularly in the context of health care policy. The scale and pace of adoption of this powerful new technology should be driven by clinical need, clinical evidence, and a commitment to put patients at the centre of health care policy.