Meta-analysis of Diagnostic Accuracy and ROC Curves with Covariate Adjusted Semiparametric Mixtures

Nonparametric estimation of the random effects distribution for the risk or rate ratio in rare events meta-analysis with the arm-based and contrast-based approaches

Rare events are events which occur with low frequencies. These often arise in clinical trials or cohort studies where the data are arranged in binary contingency tables. In this article, we investigate the estimation of effect heterogeneity for the risk-ratio parameter in meta-analysis of rare events studies through two likelihood-based nonparametric mixture approaches: an arm-based and a contrast-based model. Maximum likelihood estimation is achieved using the EM algorithm. Special attention is given to the choice of initial values. Inspired by the classification likelihood, a strategy is implemented which repeatably uses random allocation of the studies to the mixture components as choice of initial values. The likelihoods under the contrast-based and arm-based approaches are compared and differences are highlighted. We use simulations to assess the performance of these two methods. Under the design of sampling studies with nested treatment groups, the results show that the nonparametric mixture model based on the contrast-based approach is more appropriate in terms of model selection criteria such as AIC and BIC. Under the arm-based design the results from the arm-based model performs well although in some cases it is also outperformed by the contrast-based model. Comparisons of the estimators are provided in terms of bias and mean squared error. Also included in the comparison is the mixed Poisson regression model as well as the classical DerSimonian-Laird model (using the Mantel-Haenszel estimator for the common effect). Using simulation, estimating effect heterogeneity in the case of the contrast-based method appears to behave better than the compared methods although differences become negligible for large within-study sample sizes. We illustrate the methodologies using several meta-analytic data sets in medicine.

Radiomics for differentiation of gliomas from primary central nervous system lymphomas: a systematic review and meta-analysis

Background and objective

Numerous radiomics-based models have been proposed to discriminate between central nervous system (CNS) gliomas and primary central nervous system lymphomas (PCNSLs). Given the heterogeneity of the existing models, we aimed to define their overall performance and identify the most critical variables to pilot future algorithms.

Methods

A systematic review of the literature and a meta-analysis were conducted, encompassing 12 studies and a total of 1779 patients, focusing on radiomics to differentiate gliomas from PCNSLs. A comprehensive literature search was performed through PubMed, Ovid MEDLINE, Ovid EMBASE, Web of Science, and Scopus databases. Overall sensitivity (SEN) and specificity (SPE) were estimated. Event rates were pooled using a random-effects meta-analysis, and the heterogeneity was assessed using the χ2 test.

Results

The overall SEN and SPE for differentiation between CNS gliomas and PCNSLs were 88% (95% CI = 0.83 - 0.91) and 87% (95% CI = 0.83 - 0.91), respectively. The best-performing features were the ones extracted from the Gray Level Run Length Matrix (GLRLM; ACC 97%), followed by those obtained from the Neighboring Gray Tone Difference Matrix (NGTDM; ACC 93%), and shape-based features (ACC 91%). The 18F-FDG-PET/CT was the best-performing imaging modality (ACC 97%), followed by the MRI CE-T1W (ACC 87% - 95%). Most studies applied a cross-validation analysis (92%).

Conclusion

The current SEN and SPE of radiomics to discriminate CNS gliomas from PCNSLs are high, making radiomics a helpful method to differentiate these tumor types. The best-performing features are the GLRLM, NGTDM, and shape-based features. The 18F-FDG-PET/CT imaging modality is the best-performing, while the MRI CE-T1W is the most used.

Comprehensive proteomics and meta-analysis of COVID-19 host response

COVID-19 is characterised by systemic immunological perturbations in the human body, which can lead to multi-organ damage. Many of these processes are considered to be mediated by the blood. Therefore, to better understand the systemic host response to SARS-CoV-2 infection, we performed systematic analyses of the circulating, soluble proteins in the blood through global proteomics by mass-spectrometry (MS) proteomics. Here, we show that a large part of the soluble blood proteome is altered in COVID-19, among them elevated levels of interferon-induced and proteasomal proteins. Some proteins that have alternating levels in human cells after a SARS-CoV-2 infection in vitro and in different organs of COVID-19 patients are deregulated in the blood, suggesting shared infection-related changes.The availability of different public proteomic resources on soluble blood proteome alterations leaves uncertainty about the change of a given protein during COVID-19. Hence, we performed a systematic review and meta-analysis of MS global proteomics studies of soluble blood proteomes, including up to 1706 individuals (1039 COVID-19 patients), to provide concluding estimates for the alteration of 1517 soluble blood proteins in COVID-19. Finally, based on the meta-analysis we developed CoViMAPP, an open-access resource for effect sizes of alterations and diagnostic potential of soluble blood proteins in COVID-19, which is publicly available for the research, clinical, and academic community.

Covariate-specific ROC curve analysis can accommodate differences between covariate subgroups in the evaluation of diagnostic accuracy

OBJECTIVES

We present an illustrative application of methods that account for covariates in receiver operating characteristic (ROC) curve analysis, using individual patient data on D-dimer testing for excluding pulmonary embolism.

STUDY DESIGN AND SETTING

Bayesian nonparametric covariate-specific ROC curves were constructed to examine the performance/positivity thresholds in covariate subgroups. Standard ROC curves were constructed. Three scenarios were outlined based on comparison between subgroups and standard ROC curve conclusion: (1) identical distribution/identical performance, (2) different distribution/identical performance, and (3) different distribution/different performance. Scenarios were illustrated using clinical covariates. Covariate-adjusted ROC curves were also constructed.

RESULTS

Age groups had prominent differences in D-dimer concentration, paired with differences in performance (Scenario 3). Different positivity thresholds were required to achieve the same level of sensitivity. D-dimer had identical performance, but different distributions for YEARS algorithm items (Scenario 2), and similar distributions for sex (Scenario 1). For the later covariates, comparable positivity thresholds achieved the same sensitivity. All covariate-adjusted models had AUCs comparable to the standard approach.

CONCLUSION

Subgroup differences in performance and distribution of results can indicate that the conventional ROC curve is not a fair representation of test performance. Estimating conditional ROC curves can improve the ability to select thresholds with greater applicability.

Estimation of Effect Heterogeneity in Rare Events Meta-Analysis

The paper outlines several approaches for dealing with meta-analyses of count outcome data. These counts are the accumulation of occurred events, and these events might be rare, so a special feature of the meta-analysis is dealing with low counts including zero-count studies. Emphasis is put on approaches which are state of the art for count data modelling including mixed log-linear (Poisson) and mixed logistic (binomial) regression as well as nonparametric mixture models for count data of Poisson and binomial type. A simulation study investigates the performance and capability of discrete mixture models in estimating effect heterogeneity. The approaches are exemplified on a meta-analytic case study investigating the acceptance of bibliotherapy.

Structured data vs. unstructured data in machine learning prediction models for suicidal behaviors: A systematic review and meta-analysis

Suicide remains a leading cause of preventable death worldwide, despite advances in research and decreases in mental health stigma through government health campaigns. Machine learning (ML), a type of artificial intelligence (AI), is the use of algorithms to simulate and imitate human cognition. Given the lack of improvement in clinician-based suicide prediction over time, advancements in technology have allowed for novel approaches to predicting suicide risk. This systematic review and meta-analysis aimed to synthesize current research regarding data sources in ML prediction of suicide risk, incorporating and comparing outcomes between structured data (human interpretable such as psychometric instruments) and unstructured data (only machine interpretable such as electronic health records). Online databases and gray literature were searched for studies relating to ML and suicide risk prediction. There were 31 eligible studies. The outcome for all studies combined was AUC = 0.860, structured data showed AUC = 0.873, and unstructured data was calculated at AUC = 0.866. There was substantial heterogeneity between the studies, the sources of which were unable to be defined. The studies showed good accuracy levels in the prediction of suicide risk behavior overall. Structured data and unstructured data also showed similar outcome accuracy according to meta-analysis, despite different volumes and types of input data.

Meta-analysis of diagnostic cell-free circulating microRNAs for breast cancer detection

Background

Breast cancer (BC) is the most frequently diagnosed cancer among women. Numerous studies explored cell-free circulating microRNAs as diagnostic biomarkers of BC. As inconsistent and rarely intersecting microRNA panels have been reported thus far, we aim to evaluate the overall diagnostic performance as well as the sources of heterogeneity between studies.

Methods

Based on the search of three online search engines performed up to March 21^st 2022, 56 eligible publications that investigated diagnostic circulating microRNAs by utilizing Real-Time Quantitative Reverse Transcription PCR (qRT-PCR) were obtained. Primary studies’ potential for bias was evaluated with the revised tool for the quality assessment of diagnostic accuracy studies (QUADAS-2). A bivariate generalized linear mixed-effects model was applied to obtain pooled sensitivity and specificity. A novel methodology was utilized in which the sample and study models’ characteristics were analysed to determine the potential preference of studies for sensitivity or specificity.

Results

Pooled sensitivity and specificity of 0.85 [0.81—0.88] and 0.83 [0.79—0.87] were obtained, respectively. Subgroup analysis showed a significantly better performance of multiple (sensitivity: 0.90 [0.86—0.93]; specificity: 0.86 [0.80—0.90]) vs single (sensitivity: 0.82 [0.77—0.86], specificity: 0.83 [0.78—0.87]) microRNA panels and a comparable pooled diagnostic performance between studies using serum (sensitivity: 0.87 [0.81—0.91]; specificity: 0.83 [0.78—0.87]) and plasma (sensitivity: 0.83 [0.77—0.87]; specificity: 0.85 [0.78—0.91]) as specimen type. In addition, based on bivariate and univariate analyses, miRNA(s) based on endogenous normalizers tend to have a higher diagnostic performance than miRNA(s) based on exogenous ones. Moreover, a slight tendency of studies to prefer specificity over sensitivity was observed.

Conclusions

In this study the diagnostic ability of circulating microRNAs to diagnose BC was reaffirmed. Nonetheless, some subgroup analyses showed between-study heterogeneity. Finally, lack of standardization and of result reproducibility remain the biggest issues regarding the diagnostic application of circulating cell-free microRNAs.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12885-022-09698-8.

Higher-order clinical risk factor interaction analysis for overall mortality in maintenance hemodialysis patients

BACKGROUND AND AIMS

In Taiwan, approximately 90% of patients with end-stage renal disease receive maintenance hemodialysis. Although studies have reported the survival predictability of multiclinical factors, the higher-order interactions among these factors have rarely been discussed. Conventional statistical approaches such as regression analysis are inadequate for detecting higher-order interactions. Therefore, this study integrated receiver operating characteristic, logistic regression, and balancing functions for adjusting the ratio in risk classes and classification errors for imbalanced cases and controls using multifactor-dimensionality reduction (MDR-ER) analyses to examine the impact of interaction effects between multiclinical factors on overall mortality in patients on maintenance hemodialysis.

METERIALS AND METHODS

In total, 781 patients who received outpatient hemodialysis dialysis three times per week before 1 January 2009 were included; their baseline clinical factor and mortality outcome data were retrospectively collected using an approved data protocol (201800595B0).

RESULTS

Consistent with conventional statistical approaches, the higher-order interaction model could indicate the impact of potential risk combination unique to patients on maintenance hemodialysis on the survival outcome, as described previously. Moreover, the MDR-based higher-order interaction model facilitated higher-order interaction effect detection among multiclinical factors and could determine more detailed mortality risk characteristics combinations.

CONCLUSION

Therefore, higher-order clinical risk interaction analysis is a reasonable strategy for detecting non-traditional risk factor interaction effects on survival outcome unique to patients on maintenance hemodialysis and thus clinically achieving whole-scale patient care.

Chemical shift imaging for evaluation of adrenal masses: a systematic review and meta-analysis

OBJECTIVES

To perform a systematic review and meta-analysis of published data to evaluate the utility of chemical shift imaging (CSI) for differentiating between adrenal adenomas and non-adenomas.

METHODS

A systematic search of the MEDLINE, Web of Science Core Collection, EMBASE and Cochrane Central Register of Controlled Trials electronic databases was performed. The methodological quality of the included studies was assessed by using the QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies) tool. A bivariate random effect model was used to determine summary and subgroup sensitivity and specificity and calculate summary receiver operating characteristic curves (SROC).

RESULTS

Eighteen studies with 1138 patients and 1280 lesions (859 adenomas, 421 non-adenomas) in total were included. In addition to summary analysis, quantitative analyses of the adrenal signal intensity index (SII, 978 lesions, 14 studies), adrenal-to-spleen ratio (ASR; 394 lesions, 7 studies) and visual analysis (560 lesions, 5 studies) were performed. The resultant data showed considerable heterogeneity (inconsistency index I² of 94%, based on the diagnostic odds ratio, DOR). The pooled sensitivity of CSI for adenoma was 0.94 [95% confidence interval (CI) 0.88-0.97] and pooled specificity was 0.95 (95% CI 0.89-0.97). The area (AUC) under the SROC curve was 0.98 (95% CI 0.96-0.99). The corresponding AUCs were 0.98, 0.99 and 0.95 for SII, ASR and visual evaluation, respectively.

CONCLUSION

CSI has high sensitivity, specificity and accuracy for adrenal adenoma. Diagnostic performance does not improve when quantitative indices are used.

KEY POINTS

• Inclusion of CSI in abdominal MRI protocols provides an effective solution for classifying adrenal masses discovered on MR exams • Visual evaluation of adrenal CSI is sufficient; use of quantitative indices does not improve diagnostic accuracy.

Distinguishing variation in referral accuracy from referral threshold: analysis of a national dataset of referrals for suspected cancer

OBJECTIVES

To distinguish between variation in referral threshold and variation in accurate selection of patients for referral in fast-track referrals for possible cancer. To examine factors associated with threshold and accuracy and model the effects of changing thresholds.

DESIGN

Analysis of national data on cancer referrals from general practices in England over a 5-year period. We developed a new method to estimate specificity of referral to complement existing sensitivity. We used bivariate meta-analysis to produce summary measures and described practices in relation to these.

SETTING

5479 general practitioner (GP) practices with data relating to more than 50 cancer cases diagnosed over the 5 years.

OUTCOMES

Number of practices whose 95% confidence regions for sensitivity and specificity indicated that they were outliers in terms of either referral threshold or decision accuracy.

RESULTS

2019 practices (36.8%) were outliers in relation to referral threshold compared with 1205 practices (22%) in relation to decision accuracy. Practice age profile, cancer incidence and deprivation showed a modest association with decision accuracy but not with thresholds. If all practices shared the referral behaviour of those in the highest quintile of age-standardised referral rate, there would be a 3.3% increase in cancers detected through fast-track pathways at the cost of a 36.9% increase in urgent referrals.

CONCLUSION

This new method permits variation in referral to be described more precisely and quality improvement activities to be targeted. Changing referral thresholds without increasing accuracy will result in modest effects on detection rates and a large increase in demand on diagnostic services.