Efficiency of study designs in diagnostic randomized clinical trials

A review of methods for the analysis of diagnostic tests performed in sequence

BACKGROUND

Many clinical pathways for the diagnosis of disease are based on diagnostic tests that are performed in sequence. The performance of the full diagnostic sequence is dictated by the diagnostic performance of each test in the sequence as well as the conditional dependence between them, given true disease status. Resulting estimates of performance, such as the sensitivity and specificity of the test sequence, are key parameters in health-economic evaluations. We conducted a methodological review of statistical methods for assessing the performance of diagnostic tests performed in sequence, with the aim of guiding data analysts towards classes of methods that may be suitable given the design and objectives of the testing sequence.

METHODS

We searched PubMed, Scopus and Web of Science for relevant papers describing methodology for analysing sequences of diagnostic tests. Papers were classified by the characteristics of the method used, and these were used to group methods into themes. We illustrate some of the methods using data from a cohort study of repeat faecal immunochemical testing for colorectal cancer in symptomatic patients, to highlight the importance of allowing for conditional dependence in test sequences and adjustment for an imperfect reference standard.

RESULTS

Five overall themes were identified, detailing methods for combining multiple tests in sequence, estimating conditional dependence, analysing sequences of diagnostic tests used for risk assessment, analysing test sequences in conjunction with an imperfect or incomplete reference standard, and meta-analysis of test sequences.

CONCLUSIONS

This methodological review can be used to help researchers identify suitable analytic methods for studies that use diagnostic tests performed in sequence.

Regulatory Agencies and PET/CT Imaging in the Clinic

PURPOSE OF REVIEW

The regulatory steps necessary to bring new PET radiopharmaceuticals to the clinic will be reviewed. The US Food and Drug Administration (FDA) provides approval to manufacture and use diagnostic radiopharmaceuticals, including those for cardiovascular PET/CT. Medicare not only provides insurance reimbursement for imaging procedures for its beneficiaries but also sets an example for third-party insurers to cover these procedures.

RECENT FINDINGS

FDA provides extensive guidance for performing studies to obtain the safety and efficacy data needed to approve PET radiopharmaceuticals, and the pace of approval has recently increased. There also has been considerable progress in insurance coverage for PET by Medicare. Several promising agents for cardiovascular PET imaging are in the development pipeline. Challenges remain, however, including low levels of reimbursement and the application of appropriate use criteria for imaging procedures. It is important for cardiologists to understand the regulatory steps involved in translating PET radiopharmaceuticals to the clinic. Recent progress in both FDA approvals and Medicare coverage should facilitate the clinical use of new PET agents for molecular imaging of the heart.

Foundational Considerations for Artificial Intelligence Using Ophthalmic Images

IMPORTANCE

The development of artificial intelligence (AI) and other machine diagnostic systems, also known as software as a medical device, and its recent introduction into clinical practice requires a deeply rooted foundation in bioethics for consideration by regulatory agencies and other stakeholders around the globe.

OBJECTIVES

To initiate a dialogue on the issues to consider when developing a bioethically sound foundation for AI in medicine, based on images of eye structures, for discussion with all stakeholders.

EVIDENCE REVIEW

The scope of the issues and summaries of the discussions under consideration by the Foundational Principles of Ophthalmic Imaging and Algorithmic Interpretation Working Group, as first presented during the Collaborative Community on Ophthalmic Imaging inaugural meeting on September 7, 2020, and afterward in the working group.

FINDINGS

Artificial intelligence has the potential to improve health care access and patient outcome fundamentally while decreasing disparities, lowering cost, and enhancing the care team. Nevertheless, substantial concerns exist. Bioethicists, AI algorithm experts, as well as the Food and Drug Administration and other regulatory agencies, industry, patient advocacy groups, clinicians and their professional societies, other provider groups, and payors (i.e., stakeholders) working together in collaborative communities to resolve the fundamental ethical issues of nonmaleficence, autonomy, and equity are essential to attain this potential. Resolution impacts all levels of the design, validation, and implementation of AI in medicine. Design, validation, and implementation of AI warrant meticulous attention.

CONCLUSIONS AND RELEVANCE

The development of a bioethically sound foundation may be possible if it is based in the fundamental ethical principles of nonmaleficence, autonomy, and equity for considerations for the design, validation, and implementation for AI systems. Achieving such a foundation will be helpful for continuing successful introduction into medicine before consideration by regulatory agencies. Important improvements in accessibility and quality of health care, decrease in health disparities, and lower cost thereby can be achieved. These considerations should be discussed with all stakeholders and expanded on as a useful initiation of this dialogue.

OUP accepted manuscript

STUDY QUESTION

Does hysterosalpingo-foam sonography (HyFoSy) lead to similar pregnancy outcomes, compared with hysterosalpingography (HSG), as first-choice tubal patency test in infertile couples?

SUMMARY ANSWER

HyFoSy and HSG produce similar findings in a majority of patients and clinical management based on the results of either HyFoSy or HSG, leads to comparable pregnancy outcomes. HyFoSy is experienced as significantly less painful.

WHAT IS KNOWN ALREADY

Traditionally, tubal patency testing during fertility work-up is performed by HSG. HyFoSy is an alternative imaging technique lacking ionizing radiation and iodinated contrast medium exposure which is less expensive than HSG. Globally, there is a shift towards the use of office-based diagnostic methods, such as HyFoSy.

STUDY DESIGN, SIZE, DURATION

This multicentre, prospective, comparative study with a randomized design was conducted in 26 hospitals in The Netherlands. Participating women underwent both HyFoSy and HSG in randomized order. In case of discordant results, women were randomly allocated to either a management strategy based on HyFoSy or one based on HSG.

PARTICIPANTS/MATERIALS, SETTING, METHODS

We included infertile women between 18 and 41 years old who were scheduled for tubal patency testing during their fertility work-up. Women with anovulatory cycles not responding to ovulation induction, endometriosis, severe male infertility or a known iodine contrast allergy were excluded. The primary outcome for the comparison of the HyFoSy- and HSG-based strategies was ongoing pregnancy leading to live birth within 12 months after inclusion in an intention-to-treat analysis.

MAIN RESULTS AND THE ROLE OF CHANCE

Between May 2015 and January 2019, 1026 women underwent HyFoSy and HSG. HyFoSy was inconclusive in 97 of them (9.5%), HSG was inconclusive in 30 (2.9%) and both were inconclusive in 9 (0.9%). In 747 women (73%) conclusive tests results were concordant. Of the 143/1026 (14%) with discordant results, 105 were randomized to clinical management based on the results of either HyFoSy or HSG. In this group, 22 of the 54 women (41%) allocated to management based on HyFoSy and 25 of 51 women (49%) allocated to management based on HSG had an ongoing pregnancy leading to live birth (Difference −8%; 95% CI: −27% to 10%). In total, clinical management based on the results of HyFoSy was estimated to lead to a live birth in 474 of 1026 women (46%) versus 486 of 1026 (47%) for management based on HSG (Difference −1.2%; 95% CI: −3.4% to 1.5%). Given the pre-defined margin of −2%, statistically significant non-inferiority of HyFoSy relative to HSG could not be demonstrated (P = 0.27). The mean pain score for HyFoSy on the 1–10 Visual Analogue Scale (VAS) was 3.1 (SD 2.2) and the mean VAS pain score for HSG was 5.4 (SD 2.5; P for difference < 0.001).

LIMITATIONS, REASONS FOR CAUTION

Since all women underwent both tubal patency tests, no conclusions on a direct therapeutic effect of tubal flushing could be drawn.

WIDER IMPLICATIONS OF THE FINDINGS

HyFoSy or HSG produce similar tubal pathology findings in a majority of infertile couples and, where they differ, a difference in findings does not lead to substantial difference in pregnancy outcome, while HyFoSy is associated with significantly less pain.

STUDY FUNDING/COMPETING INTEREST(S)

The FOAM study was an investigator-initiated study funded by ZonMw, The Netherlands organization for Health Research and Development (project number 837001504). ZonMw funded the whole project. IQ Medical Ventures provided the ExEm-foam^® kits free of charge. The funders had no role in study design, collection, analysis and interpretation of the data. K.D. reports travel and speaker fees from Guerbet. F.J.M.B. reports personal fees as a member of the external advisory board for Merck Serono, The Netherlands, and a research support grant from Merck Serono, outside the submitted work. C.B.L. reports speakers’ fee from Ferring in the past, and his department receives research grants from Ferring, Merck and Guerbet. J.S. reports a research agreement with Takeda on MR of motility outside the submitted work. M.V.W. reports leading The Netherlands Satellite of the Cochrane Gynaecology and Fertility Group. B.W.J.M. is supported by an NHMRC Investigator grant (GNT1176437). B.W.J.M. reports consultancy for Guerbet and research funding from Merck and Guerbet. V.M. reports non-financial support from IQ medicals ventures, during the conduct of the study; grants and personal fees from Guerbet, outside the submitted work. The other authors do not report conflicts of interest.

TRIAL REGISTRATION NUMBER

NTR4746/NL4587 (https://www.trialregister.nl)

TRIAL REGISTRATION DATE

19 August 2014

DATE OF FIRST PATIENT’S ENROLMENT

7 May 2015

Studies for the Evaluation of Diagnostic Tests–Part 28 of a Series on Evaluation of Scientific Publications

BACKGROUND

The accurate diagnosis of a disease is a prerequisite for its appropriate treatment. How well a medical test is able to correctly identify or rule out a target disease can be assessed by diagnostic accuracy studies.

METHODS

The main statistical parameters that are derived from diagnostic accuracy studies, and their proper interpretation, will be presented here in the light of publications retrieved by a selective literature search, supplemented by the authors' own experience. Aspects of study planning and the analysis of complex studies on diagnostic tests will also be discussed.

RESULTS

In the usual case, the findings of a diagnostic accuracy study are presented in a 2 × 2 contingency table containing the number of true-positive, true-negative, false-positive, and true-positive test results. This information allows the calculation of various statistical parameters, of which the most important are the two pairs sensitivity/ specificity and positive/negative predictive value. All of these parameters are quotients, with the number of true positive (resp. true negative) test results in the numerator; the denominator is, in the first pair, the total number of ill (resp. healthy) patients, and in the second pair, the total number of patients with a positive (resp. negative) test. The predictive values are the parameters of greatest interest to phy - sicians and patients, but their main disadvantage is that they can easily be misinterpreted. We will also present the receiver operating characteristic (ROC) curve and the area under the curve (AUC) as additional important measures for the assessment of diagnostic tests. Further topics are discussed in the supplementary materials.

CONCLUSION

The statistical parameters used to assess diagnostic tests are primarily based on 2 × 2 contingency tables. These parameters must be interpreted with care in order to draw correct conclusions for use in medical practice.

Randomized test-treatment studies with an outlook on adaptive designs

Background

Diagnostic accuracy studies aim to examine the diagnostic accuracy of a new experimental test, but do not address the actual merit of the resulting diagnostic information to a patient in clinical practice. In order to assess the impact of diagnostic information on subsequent treatment strategies regarding patient-relevant outcomes, randomized test-treatment studies were introduced. Various designs for randomized test-treatment studies, including an evaluation of biomarkers as part of randomized biomarker-guided treatment studies, are suggested in the literature, but the nomenclature is not consistent.

Methods

The aim was to provide a clear description of the different study designs within a pre-specified framework, considering their underlying assumptions, advantages as well as limitations and derivation of effect sizes required for sample size calculations. Furthermore, an outlook on adaptive designs within randomized test-treatment studies is given.

Results

The need to integrate adaptive design procedures in randomized test-treatment studies is apparent. The derivation of effect sizes induces that sample size calculation will always be based on rather vague assumptions resulting in over- or underpowered study results. Therefore, it might be advantageous to conduct a sample size re-estimation based on a nuisance parameter during the ongoing trial.

Conclusions

Due to their increased complexity, compared to common treatment trials, the implementation of randomized test-treatment studies poses practical challenges including a huge uncertainty regarding study parameters like the expected outcome in specific subgroups or disease prevalence which might affect the sample size calculation. Since research on adaptive designs within randomized test-treatment studies is limited so far, further research is recommended.

Supplementary Information

The online version contains supplementary material available at (10.1186/s12874-021-01293-y).

Evaluation of Diagnostic Tests

As technology advances, diagnostic tests continue to improve and each year, we are presented with new alternatives to standard procedures. Given the plethora of diagnostic alternatives, diagnostic tests must be evaluated to determine their place in the diagnostic armamentarium. The first step involves determining the accuracy of the test, including the sensitivity and specificity, positive and negative predictive values, likelihood ratios for positive and negative tests, and receiver operating characteristic (ROC) curves. The role of the test in a diagnostic pathway has then to be determined, following which the effect on patient outcome should be examined.

Methodologic considerations in randomized clinical trials in reproductive medicine

Randomized controlled trials (RCTs) are the cornerstone of evidence-based medicine. In this series in Fertility and Sterility, several aspects of RCTs are discussed, with contributions on multicenter RCTs, different international settings, and integrity of RCTs. The present contribution deals with methodologic issues. We discuss different types of RCTs based on null hypothesis (superiority vs. noninferiority vs. equivalence) as well as frequentist versus Bayesian interpretation. We also discuss the use of RCTs in the era of personalized medicine and RCTs to address diagnostic and prognostic questions. Finally, we address the use of big data compared with the use of RCTs.

Diagnostic Accuracy Studies

The medical community often assumes that the tests we use to diagnose various diseases are accurate, safe, and effective. However, the study designs traditionally used to determine whether such a diagnostic test is indeed accurate, safe, and effective are often at a higher risk of bias and are of lower methodological quality than those evaluating efficacy of therapeutic interventions. Several designs can be used to study diagnostic tests such as diagnostic accuracy cross-sectional studies, diagnostic accuracy case-control studies, and diagnostic accuracy comparative studies. Clinicians, researchers, and policy-makers may wish to consider moving toward higher quality study designs when studying new diagnostic modalities prior to their implementation in routine practice and diagnostic randomized trials are one such alternative.

The FOAM study: is Hysterosalpingo foam sonography (HyFoSy) a cost-effective alternative for hysterosalpingography (HSG) in assessing tubal patency in subfertile women? Study protocol for a randomized controlled trial

BACKGROUND

Tubal pathology is a causative factor in 20% of subfertile couples. Traditionally, tubal testing during fertility work-up is performed by hysterosalpingography (HSG). Hysterosalpingo-foam sonography (HyFoSy) is a new technique that is thought to have comparable accuracy as HSG, while it is less expensive and more patient friendly. HyFoSy would be an acceptable alternative for HSG, provided it has similar effectiveness in terms of patient outcomes.

METHODS/DESIGN

We aim to compare the effectiveness and costs of management guided by HyFoSy or by HSG. Consenting women will undergo tubal testing by both HyFoSy and HSG in a randomized order during fertility work-up. The study group will consist of 1163 subfertile women between 18 and 41 years old who are scheduled for tubal patency testing during their fertility work-up. Women with anovulatory cycles not responding to ovulation induction, endometriosis, severe male subfertility or a known contrast (iodine) allergy will be excluded. We anticipate that 7 % (N = 82) of the participants will have discordant test results for HyFoSy and HSG. These participants will be randomly allocated to either a management strategy based on HyFoSy or a management strategy based on HSG, resulting in either a diagnostic laparoscopy with chromopertubation or a strategy that assumes tubal patency (intrauterine insemination or expectant management). The primary outcome is ongoing pregnancy leading to live birth within 12 months after randomization. Secondary outcomes are patient pain scores, time to pregnancy, clinical pregnancy, miscarriage rate, multiple pregnancy rate, preterm birth rate and number of additional treatments. Costs will be estimated by counting resource use and calculating unit prices.

DISCUSSION

This trial will compare the effectiveness and costs of HyFoSy versus HSG in assessing tubal patency in subfertile women.

TRIAL REGISTRATION

Dutch Trial Register (NTR 4746, http://www.trialregister.nl ). Date of registration: 19 August 2014.

Filling the gap between lab and clinical impact: An open randomized diagnostic trial comparing urinary ethylglucuronide and ethanol in alcohol dependent outpatients

BACKGROUND

Efforts aimed at reducing alcohol-related harm include early detection of risky drinkers as well as detection of early relapse in patients with alcohol dependence. Ethyl glucuronide (EtG) has been proven to be a reliable biomarker for the detection of recent drinking; however, no randomized, diagnostic trial to date has tested its impact on drinking outcomes. The aim of this study was to assess, in a randomized design, the implications of EtG screening on alcohol outcomes, compared to screening with a low sensitivity biomarker such as ethanol.

METHODS

Alcohol dependent outpatients were randomized to either 24 weeks of continuous screening with EtG or ethanol. Patients were aware of screening methods and results. After 24 weeks, all participants were screened with EtG. Self-reports were also gathered. A logistic regression compared the rate of EtG positive results at study end between groups. Generalized estimating equations evaluated the descending monthly rate of EtG positive patients in the EtG group.

RESULTS

A total of 162 patients were randomized. During the study period, the ethanol group showed less patients with positive screens (19/64 (29.7%) vs 58/98 (59%)). After 24 weeks, the EtG group showed a greater number of patients having a negative screening test compared to ethanol subjects when they were all screened with EtG (5/62 (8.1%) vs 13/39 (33.3%)). A significant decrease in the rate of EtG positive patients was found for the first three months of the study.

CONCLUSIONS

Routine screening with EtG seems to reduce drinking and improve abstinence rates in alcohol dependent outpatients.

Beyond Correlations, Sensitivities, and Specificities: A Roadmap for Demonstrating Utility of Advanced Imaging in Oncology Treatment and Clinical Trial Design

Despite the widespread belief that advanced imaging should be very helpful in guiding oncology treatment decision and improving efficiency and success rates in treatment clinical trials, its acceptance has been slow. Part of this is likely attributable to gaps in study design and statistical methodology for these imaging studies. Also, results supporting the performance of the imaging in these roles have largely been insufficient to justify their use within the design of a clinical trial or in treatment decision making. Statistically significant correlations between the imaging results and clinical outcomes are often incorrectly taken as evidence of adequate performance. Assessments of whether the imaging can outperform standard techniques or meaningfully supplement them are also frequently neglected. This paper provides guidance on study designs and statistical analyses for evaluating the performance of advanced imaging in the various roles in treatment decision guidance and clinical trial conduct. Relevant methodology from the imaging literature is reviewed; gaps in the literature are addressed using related concepts from the more extensive genomic and in vitro biomarker literature.

[Benefit assessment of diagnostic procedures: quo vadimus?]

Not only therapeutic procedures, but also diagnostic procedures, have to demonstrate their patient-relevant benefits if they are to be reimbursed by public health insurance. Randomized trials comparing two diagnostic procedures allow us to assess these benefits directly if appropriate outcomes are used. However, owing to the widespread lack of such studies, it is now necessary to use the "linked evidence" approach as well, trying to predict the patient-relevant benefits from the results of comparative accuracy studies. Such a prediction is based on explicitly specifying our expectations with regard to the consequences of a change in diagnosis at the level of a single patient. We discuss the basic properties of these two approaches, which are relevant to the understanding of their possible role in the benefit assessment of diagnostic procedures. We try to predict the future roles of the two approaches and outline some of the issues on which a consensus is required to allow their successful use in benefit assessment. Furthermore, we indicate some of the developments related to the paradigm of individualized care that may influence the use of benefit assessments for diagnostic studies in the future.

Statistical controversies in clinical research: prognostic gene signatures are not (yet) useful in clinical practice

With the genomic revolution and the era of targeted therapy, prognostic and predictive gene signatures are becoming increasingly important in clinical research. They are expected to assist prognosis assessment and therapeutic decision making. Notwithstanding, an evidence-based approach is needed to bring gene signatures from the laboratory to clinical practice. In early breast cancer, multiple prognostic gene signatures are commercially available without having formally reached the highest levels of evidence-based criteria. We discuss specific concepts for developing and validating a prognostic signature and illustrate them with contemporary examples in breast cancer. When a prognostic signature has not been developed for predicting the magnitude of relative treatment benefit through an interaction effect, it may be wishful thinking to test its predictive value. We propose that new gene signatures be built specifically for predicting treatment effects for future patients and outline an approach for this using a cross-validation scheme in a standard phase III trial. Replication in an independent trial remains essential.

Evaluation of diagnostic tests

As technology advances, diagnostic tests continue to improve, and each year we are presented with new alternatives to standard procedures. Given the plethora of diagnostic alternatives, diagnostic tests must be evaluated to determine their place in the diagnostic armamentarium. The first step involves determining the accuracy of the test, including the sensitivity and specificity, positive and negative predictive values, likelihood ratios for positive and negative tests, and receiver operating characteristic (ROC) curves. The role of the test in a diagnostic pathway has then to be determined, following which the effect on patient outcome should be examined.

The move from accuracy studies to randomized trials in PET: current status and future directions

Since the influential study by van Tinteren et al. published in The Lancet in 2002, there have been an increasing number of diagnostic randomized controlled trials (RCTs) investigating the benefit of PET. If they provide valid and useful information on the benefit, these studies can play an important role in informing guideline developers and policy makers. Our aim was to investigate how far the nuclear medicine community has come on its way from accuracy studies to RCTs and which issues we have to take into account in planning future studies.

METHODS

We conducted a systematic review of diagnostic randomized trials, in which PET was applied in only one arm. We covered published studies as well as registered unpublished and planned studies. We considered 3 quality indicators related to the usefulness of a trial to generate evidence for a clinical benefit: use of patient-important outcome, sufficient sample size, and current standard as comparator.

RESULTS

Fourteen published and 15 planned studies were identified. Five of the published studies and 12 of the planned studies did not use a patient-important outcome. Sample sizes were often so small that a significant result could be expected only under the assumption of a substantial reduction in the event rate. Comparators typically reflected the current standard.

CONCLUSION

If we consider the traditional areas of primary diagnosis, staging, and follow-up, then the number and quality of RCTs on PET is currently not sufficient to provide a major source for evidence-based decisions on the clinical benefit of PET. There will also be a future need in these traditional areas to deduce the clinical benefit of PET from the results of accuracy studies. The situation may be more favorable for the areas of treatment planning and response evaluation. Choice of patient-important outcomes and sufficient sample sizes are crucial issues in planning RCTs to demonstrate the clinical benefit of using PET.