Evaluations of statistical methods for outlier detection when benchmarking in clinical registries: a systematic review

OBJECTIVES

Benchmarking is common in clinical registries to support the improvement of health outcomes by identifying underperforming clinician or health service providers. Despite the rise in clinical registries and interest in publicly reporting benchmarking results, appropriate methods for benchmarking and outlier detection within clinical registries are not well established, and the current application of methods is inconsistent. The aim of this review was to determine the current statistical methods of outlier detection that have been evaluated in the context of clinical registry benchmarking.

DESIGN

A systematic search for studies evaluating the performance of methods to detect outliers when benchmarking in clinical registries was conducted in five databases: EMBASE, ProQuest, Scopus, Web of Science and Google Scholar. A modified healthcare modelling evaluation tool was used to assess quality; data extracted from each study were summarised and presented in a narrative synthesis.

RESULTS

Nineteen studies evaluating a variety of statistical methods in 20 clinical registries were included. The majority of studies conducted application studies comparing outliers without statistical performance assessment (79%), while only few studies used simulations to conduct more rigorous evaluations (21%). A common comparison was between random effects and fixed effects regression, which provided mixed results. Registry population coverage, provider case volume minimum and missing data handling were all poorly reported.

CONCLUSIONS

The optimal methods for detecting outliers when benchmarking clinical registry data remains unclear, and the use of different models may provide vastly different results. Further research is needed to address the unresolved methodological considerations and evaluate methods across a range of registry conditions.

PROSPERO REGISTRATION NUMBER

CRD42022296520.

Complementing the P-value from null-hypothesis significance testing with a Bayes factor from null-hypothesis Bayesian testing

BACKGROUND

Classical frequentist statistics, including null-hypothesis significance testing (NHST), dominates nursing and medical research analysis. However, there is increasing recognition that null-hypothesis Bayesian testing (NHBT) merits inclusion in healthcare research analysis.

AIM

To recommend that researchers complement the P-value from NHST with a Bayes factor from NHBT in their research analysis.

DISCUSSION

Reporting the P-value and a Bayes factor clarifies results that may be difficult to interpret using the P-value alone.

CONCLUSION

NHBT offers statistical and practical advantages that complement NHST.

Surgery for women with endometrioma prior to in vitro fertilisation: proposal for a feasible multicentre randomised clinical trial in the UK

STUDY QUESTION

Is it feasible to undertake a randomised controlled trial to establish whether surgical removal of endometrioma or not, improves live birth rates from IVF?

SUMMARY ANSWER

A randomised controlled trial (RCT) comparing surgery versus no surgery to endometrioma prior to IVF is only feasible in UK if an adaptive rather than traditional study design is used; this would minimise resource wastage and complete the trial in an acceptable time frame.

WHAT IS KNOWN ALREADY

There is wide variation in the management of endometriomas prior to IVF, with decisions about treatment being influenced by personal preferences.

STUDY DESIGN, SIZE, AND DURATION

This was a mixed-methods study consisting of an online survey of clinicians, a focus group and individual interviews with potential trial participants.

PARTICIPANTS/MATERIALS, SETTING, METHODS

Endometriosis and fertility experts across the UK were invited to participate in an online anonymised questionnaire. Potential future trial participants were recruited from a tertiary care fertility centre and invited to participate in either individual interviews or focus groups.

MAIN RESULTS AND THE ROLE OF CHANCE

Clinicians and potential trial participants confirmed the need for an RCT to inform the management of an endometrioma prior to IVF. There were 126 clinicians who completed the survey, and the majority (70%) were willing to recruit to a trial. Half of those who responded indicated that they see at least 10 eligible women each year. The main barriers to recruitment were waiting lists for surgery and access to public funding for IVF. One focus group (n = 7) and five interviews were conducted with potential trial participants (n = 3) and their partners (n = 2). The findings from these discussions highlighted that recruitment and retention in a potential RCT could be improved by coordination between IVF and surgical services such that an operation does not delay IVF or affect access to public funding. Live birth was considered the most important outcome with an improvement of at least 10% considered the minimum acceptable by both patients and clinicians.

LIMITATIONS, REASONS FOR CAUTION

This feasibility study captured views of clinicians across the UK, but as patients were from a single Scottish centre, their views may not be representative of other areas with limited public funding for IVF.

WIDER IMPLICATIONS OF THE FINDINGS

There is a need for an appropriately powered RCT to establish whether or not surgical treatment of endometrioma prior to IVF improves live birth rates. There are logistical issues to be considered due to limited number of participants, funding of IVF and waiting times. These could be overcome in a RCT by using an adaptive design which would include a prospectively planned opportunity for modification of specified aspects of the study design based on interim analysis of the data, coordination of IVF treatments and endometriosis surgeries and international collaboration. Similar principles could be used for other questions in fertility where a traditional approach for randomised trials is not feasible.

STUDY FUNDING/COMPETING INTEREST(S)

Funding was received from the NHS Grampian R&D pump priming fund (RG14437-12). S.B. is Editor-in-Chief of HROPEN, and A.W.H. is Deputy Editor of HROPEN. Neither was involved in the review of this manuscript. L.S. reports grants from CSO and NIHR to do endometriosis research, outside the submitted work. K.C. reports grants from NIHR/HTA and CSO during the conduct of the study. J.H.e., A.W.H., J.D., S.B.r., K.B., G.B., J.H.u. and K.G. report no conflict of interest.