Diagnostic test accuracy in longitudinal study settings: theoretical approaches with use cases from clinical practice

OBJECTIVES

In this study, we evaluate how to estimate diagnostic test accuracy (DTA) correctly in the presence of longitudinal patient data (ie, repeated test applications per patient).

STUDY DESIGN AND SETTING

We used a nonparametric approach to estimate the sensitivity and specificity of three tests for different target conditions with varying characteristics (ie, episode length and disease-free intervals between episodes): 1) systemic inflammatory response syndrome (n = 36), 2) depression (n = 33), and 3) epilepsy (n = 30). DTA was estimated on the levels 'time', 'block', and 'patient-time' for each diagnosis, representing different research questions. The estimation was conducted for the time units per minute, per hour, and per day.

RESULTS

A comparison of DTA per and across use cases showed variations in the estimates, which resulted from the used level, the time unit, the resulting number of observations per patient, and the diagnosis-specific characteristics. Intra- and inter-use-case comparisons showed that the time-level had the highest DTA, particularly the larger the time unit, and that the patient-time-level approximated 50% sensitivity and specificity.

CONCLUSION

Researchers need to predefine their choices (ie, estimation levels and time units) based on their individual research aims, estimands, and diagnosis-specific characteristics of the target outcomes to make sure that unbiased and clinically relevant measures are communicated. In cases of uncertainty, researchers could report the DTA of the test using more than one estimation level and/or time unit.

Sensitivity and specificity of alternative screening methods for systematic reviews using text mining tools

OBJECTIVES

To evaluate the impact of text mining (TM) on the sensitivity and specificity of title and abstract screening strategies for systematic reviews (SRs).

STUDY DESIGN AND SETTING

Twenty reviewers each evaluated a 500-citation set. We compared five screening methods: conventional double screen (CDS), single screen, double screen with TM, combined double screen and single screen with TM, and single screen with TM. Rayyan, Abstrackr, and SWIFT-Review were used for each TM method. The results of a published SR were used as the reference standard.

RESULTS

The mean sensitivity and specificity achieved by CDS were 97.0% (95% confidence interval [CI]: 94.7, 99.3) and 95.0% (95% CI: 93.0, 97.1). When compared with single screen, CDS provided a greater sensitivity without a decrease in specificity. Rayyan, Abstrackr, and SWIFT-Review identified all relevant studies. Specificity was often higher for TM-assisted methods than that for CDS, although with mean differences of only one-to-two percentage points. For every 500 citations not requiring manual screening, 216 minutes (95% CI: 169, 264) could be saved.

CONCLUSION

TM-assisted screening methods resulted in similar sensitivity and modestly improved specificity as compared to CDS. The time saved with TM makes this a promising new tool for SR.

Development and validation of a clinical prediction model for detecting coronary heart disease in middle-aged and elderly people: a diagnostic study

OBJECTIVE

To develop and validate a multivariate prediction model to estimate the risk of coronary heart disease (CHD) in middle-aged and elderly people and to provide a feasible method for early screening and diagnosis in middle-aged and elderly CHD patients.

METHODS

This study was a single-center, retrospective, case-control study. Admission data of 932 consecutive patients with suspected CHD were retrospectively assessed from September 1, 2020 to December 31, 2021 in the Department of Integrative Cardiology at China-Japan Friendship Hospital. A total of 839 eligible patients were included in this study, and 588 patients were assigned to the derivation set and 251 as the validation set at a 7:3 ratio. Clinical characteristics of included patients were compared between derivation set and validation set by univariate analysis. The least absolute shrinkage and selection operator (Lasso) regression analysis method was performed to avoid collinearity and identify key potential predictors. Multivariate logistic regression analysis was used to construct a clinical prediction model with identified predictors for clinical practice. Bootstrap validation was used to test performance and eventually we obtained the actual model. And the Hosmer-Lemeshow test was carried out to evaluate the goodness-fit of the constructed model. The area under curve (AUC) of receiver operating characteristic (ROC), calibration curve, decision curve analysis (DCA), and clinical impact curve (CIC) were plotted and utilized with validation set to comprehensively evaluate the predictive accuracy and clinical value of the model.

RESULTS

A total of eight indicators were identified as risk factors for the development of CHD in middle-aged and elderly people by univariate analysis. Of these candidate predictors, four key parameters were defined to be significantly related to CHD by Lasso regression analysis, including age (OR 1.034, 95% CI 1.002 ~ 1.067, P = 0.040), hemoglobin A1c (OR 1.380, 95% CI 1.078 ~ 1.768, P = 0.011), ankle-brachial index (OR 0.078, 95% CI 0.012 ~ 0.522, P = 0.009), and brachial artery flow-mediated vasodilatation (OR 0.848, 95% CI 0.726 ~ 0.990, P = 0.037). The Hosmer-Lemeshow test showed a good calibration performance of the clinical prediction model (derivation set, χ2 = 7.865, P = 0.447; validation set, χ2 = 11.132, P = 0.194). The ROCs of the nomogram in the derivation set and validation set were 0.722 and 0.783, respectively, suggesting excellent predictive power and suitable performance. The clinical prediction model presented a greater net benefit and clinical impact based on DCA and CIC analysis.

CONCLUSION

Overall, the development and validation of the multivariate model combined the laboratory and clinical parameters of patients with CHD, which could be beneficial to the individualized prediction of middle-aged and elderly people, and helped to facilitate clinical assessments and decisions during treatment and management of CHD.

Missing values and inconclusive results in diagnostic studies - A scoping review of methods

Most diagnostic studies exclude missing values and inconclusive results from the analysis or apply simple methods resulting in biased accuracy estimates. This may be due to the lack of availability or awareness of appropriate methods. This scoping review aimed to provide an overview of strategies to handle missing values and inconclusive results in the reference standard or index test in diagnostic accuracy studies. Conducting a systematic literature search in MEDLINE, Cochrane Library, and Web of Science, we could identify many articles proposing methods for addressing missing values in the reference standard. There are also several articles describing methods regarding missing values or inconclusive results in the index test. The latter encompass imputation, frequentist and Bayesian likelihood, model-based, and latent class methods. While methods for missing values in the reference standard are regularly applied in practice, this is not true for methods addressing missing values and inconclusive results in the index test. Our comprehensive overview and description of available methods may raise further awareness of these methods and will enhance their application. Future research is needed to compare the performance of these methods under different conditions to give valid and robust recommendations for their usage in various diagnostic accuracy research scenarios.

Circulating Tumor Cells as Diagnostic Markers of Early Gastric Cancer and Gastric Precancerous Lesions

INTRODUCTION

Circulating tumor cells (CTCs) may be potential diagnostic biomarkers of various malignancies including gastric cancer. This study aimed to evaluate whether CTCs could be used to facilitate the diagnosis of early gastric cancer (EGC) or precancerous gastric lesions.

METHODS

The diagnostic study included consecutive patients with EGC, gastric precancerous lesions, or fundic gland polyps admitted to the Gastroenterology Department, Beijing Friendship Hospital Affiliated to Capital Medical University (National Center for Digestive Diseases) between October 2016 and January 2018.

RESULTS

A total of 92 patients were enrolled, including 57 patients with EGC, 14 patients with gastric precancerous lesions, and 21 patients with fundic gland polyps (control group). CTCs were detected in 47.89% (34/71) of patients with EGC/gastric precancerous lesions and 4.76% (1/21) of patients with fundic gland polyps (p < 0.001). CTC detection distinguished EGC/precancerous lesions from fundic gland polyps with an area under the receiver operating characteristic curve of 0.740 (95% confidence interval, 0.640-0.840; p = 0.001), a sensitivity of 49.10%, a specificity of 95.00%, a positive predictive value of 97.00%, and a negative predictive value of 64.90%.

CONCLUSIONS

Peripheral blood CTCs are more common in patients with EGC or gastric precancerous lesions than in patients with fundic gland polyps. Measurement of CTCs may be a useful tool to aid in the diagnosis of EGC and precancerous lesions.

Development of a diagnostic score using FeNO and symptoms to predict asthma

BACKGROUND

Fractional exhaled nitric oxide (FeNO) is known as effective for ruling-in asthma. The diagnostic value might be increased in combination with clinical signs and symptoms (CSS). The aim was to develop a new model for ruling-in and ruling-out asthma.

METHODS

Diagnostic multi-centre study in three practices of pneumologists in Germany. Whole-body plethysmography was combined with bronchodilation tests or bronchial provocation as diagnostic reference standard. Follow-up was performed after 3 months. An expert committee evaluated test results, symptoms, and course of disease for the final diagnosis. Relevant CSS known from guidelines were used to enable combinatorial development of decision rules. Outcomes of multiple logistic regression modeling were translated into a diagnostic score and internally validated by ten-fold cross validation.

RESULTS

308 patients with complete follow-up were included. 186 (60.4%) were female, average age was 44.7 years and 161 (52.5%) had asthma. The average area under the receiver operating curve (AUC) of the diagnostic score was 0.755 (interquartile range 0.721-0.814). Allergic rhinitis, wheezing, dyspnea on exertion, coughing attacks at night, and awakening by shortness of breath were leading symptoms for ruling-in asthma. Frequent coughing and frequent respiratory infections were leading symptoms for ruling-out. The combination of FeNO and CSS allowed ruling-in asthma with a probability of up to 99%, and ruling-out with a post-test probability down to 9%.

CONCLUSION

The diagnostic scoring model increased the diagnostic value of FeNO in combination with CSS. The new decision rule allowed to rule-in asthma with high certainty, and also to rule-out with acceptable certainty.

Systematic review identifies deficiencies in reporting of diagnostic test accuracy among clinical decision support systems

OBJECTIVES

This systematic review assesses the reporting quality and risk of bias in studies evaluating the diagnostic test accuracy (DTA) of clinical decision support systems (CDSS).

STUDY DESIGN AND SETTING

The Cochrane Library, PubMed/MEDLINE, Scopus, and Web of Science were searched for studies, published between January 1, 2016 and May 31, 2021, evaluating the DTA of CDSS for human patients. Articles using a patient's self-diagnosis, assessing disease severity, focusing on treatment/follow-up, or comparing pre-post CDSS implementation periods were excluded. All eligible studies were assessed for reporting quality using STARD 2015 and for risk of bias using QUADAS-2. Item ratings were presented using heat maps. This study was reported as per PRISMA-DTA.

RESULTS

In total, 158 of 2,820 screened articles were included in the analysis. The studies were heterogeneous in terms of study characteristics, reporting quality, risk of biases, and applicability concerns with few highly rated studies. Mostly the overall quality was deficient for items addressing the domains 'methodology,' 'results,' and 'other information'.

CONCLUSION

Our analysis revealed shortcomings in critical domains of reporting quality and risk of bias, indicating the need for additional guidance and training in an interdisciplinary scientific field with mixed biostatistical expertise.

Skin thickness affects the result of tuberculin skin test in systemic sclerosis

Background

Skin thickness is a prominent clinical feature of systemic sclerosis (SSc), but there is no consensus on the cut-off for a positive tuberculin skin test (TST) size and the limitation of the TST for a diagnosis of tuberculosis in SSc. We aimed to identify the cut-off size of an indurated TST and the sensitivity and specificity of the test for the diagnosis of tuberculosis in SSc patients.

Methods

A cross-sectional study of 168 adult Thai SSc patients was conducted. The TST was done using 0.1 ml of purified protein derivatives via intradermal injection. The test was interpreted 72 h after testing.

Results

The median age was 57.2 years. The majority (71.8%) had the diffuse cutaneous SSc subset. All the patients had a BCG vaccination at birth, and 17 (10.1%) had a tuberculosis infection. An indurated skin reaction size of 20 mm had the highest specificity for tuberculosis (99.3%: 95%CI 96.4–100) (ROC 0.53). The skin thickness—assessed using the modified Rodnan skin score (mRSS)—had a significant negative correlation with the reaction size (Rho -0.23; p = 0.003).

Conclusion

The TST is not sufficiently sensitive for detecting TB infection in SSc patients, albeit a skin induration of ≥ 15 mm indicates a high specificity for tuberculosis infection. A high mRSS resulted in a smaller skin reaction size when using the TST, which has limited utility as a diagnostic for tuberculosis among SSc patients with severe skin thickness.

The manuscript was presented as a poster presentation at the Annual European Congress of Rheumatology EULAR 2019 Madrid 12–15 June 2019. (Ann Rheum Dis. 2019;78(suppl 2): abstract FRI0347) http://dx.doi.org/10.1136/annrheumdis-2019-eular.1456

Supplementary Information

The online version contains supplementary material available at 10.1186/s41927-022-00278-8.

Diagnostic accuracy of FeNO in asthma and predictive value for inhaled corticosteroid responsiveness: A prospective, multicentre study

BACKGROUND

Fractional exhaled nitric oxide (FeNO) is promising for diagnosing asthma and could replace bronchial provocation (BP). To date, cut-off values have been derived by post hoc analysis only. The aim was to validate the diagnostic accuracy for predefined FeNO cut-off values and the predictive value for responsiveness to inhaled corticosteroids (ICS).

METHODS

We conducted a prospective, diagnostic, multicentre study with patients attending three private practices of pneumologists in Upper Bavaria, Germany, from July 3, 2020 to Jan 21, 2022. Index test was FENO measurement. Reference standard was Tiffeneau ratio (FEV₁/VC) or airway resistance as assessed by whole body plethysmography, with additional BP or bronchodilation test. Follow-up was performed after 12 weeks. Analyses of Receiver Operating Characteristics curves were conducted to determine the diagnostic accuracy and predictive value of FeNO.

FINDINGS

308 patients with complete follow-up were recruited, 186 (60·4%) were female, average age was 44·7 years, 161 (52·3%) had asthma. Regarding diagnostic accuracy, the area under the curve (AUC) was 0·718 (95% CI 0·661-0·775; p < 0·001). Sensitivity at FeNO >50 ppb was 0·24 (95% CI 0·18-0·32), specificity 0·99 (0·95-1·0), positive predictive value (PPV) 0·95 (0·84-0·99), negative predictive value (NPV) 0·54 (0·48-0·60). In 66 patients with ´wheezing´ and ´allergic rhinitis´, the sensitivity at FeNO >33 ppb was 0·49 (0·34-0·64), specificity 0·88 (0·64-0·99), PPV 0·92 (0·75-0·99), NPV 0·38 (0·23-0·54). In 68 patients with ICS medication, responsiveness was predicted at the cut-off >43 ppb, with a sensitivity of 0·55 (95%CI 0·36-0·74), specificity 0·82 (0·66-0·92), PPV 0·70 (0·47-0·87), NPV 0·71 (0·56-0·84).

INTERPRETATION

FeNO measurement allows a valid ruling-in of an asthma diagnosis, whereas ruling-out of asthma is not possible. Enhanced probability of ICS responsiveness is also given with increased FeNO values.

FUNDING

Circassia Germany gave 25% discount on the purchase of three NIOX VERO devices.

The accuracy of a recombinant antigen immunochromatographic test for the detection of Strongyloides stercoralis infection in migrants from sub-Saharan Africa

Background

Strongyloidiasis, a nematode infection which is mainly caused by Strongyloides stercoralis in humans, can lead to a fatal syndrome in immunocompromised individuals. Its diagnosis is challenging due to the absence of a diagnostic gold standard. The infection is highly prevalent in migrants from endemic countries in tropical and subtropical areas, and a rapid diagnostic test would be helpful for screening purposes. The aim of this study was to estimate the accuracy of a novel immunochromatographic test (ICT) for the diagnosis of S. stercoralis infection.

Methods

A single-centre diagnostic accuracy study was undertaken using well-characterized frozen sera available from the biobank of a referral hospital for parasitic diseases in Italy. The included sera were from migrants from sub-Saharan Africa, and matching results were available for agar plate culture and/or polymerase chain reaction for S. stercoralis; moreover, the results of both a commercial enzyme-linked immunosorbent assay test and an in-house immunofluorescence test for strongyloidiasis were made available. Laboratory staff who read the ICT results were blinded as regards the results of the other tests. Two readers independently read the ICT, and a third one was involved when results were discrepant. The accuracy of the ICT was assessed both against the results of the panel of faecal tests and by latent class analysis (LCA).

Results

Agreement between the readers was excellent [Cohen’s κ = 92.7%, 95% confidence interval (CI) 88.3–97.1%]. When assessed against the results of the faecal tests, the sensitivity and specificity of the ICT were 82.4% (95% CI 75.7–89.0%) and 73.8% (95% CI 66.8–80.9%), respectively. According to the LCA, the sensitivity and specificity were 86.3% (95% CI 80.1–92.5%) and 73.9% (95% CI 67.0–80.8%), respectively.

Conclusions

The results of the ICT demonstrated ease of interpretation. The accuracy proved good, though the sensitivity might be further improved for screening purposes.

Graphical Abstract

Supplementary Information

The online version contains supplementary material available at 10.1186/s13071-022-05249-z.

Evaluation of the clinical effect of an artificial intelligence-assisted diagnosis and treatment system for neonatal seizures in the real world: a multicenter clinical study protocol

Neonatal seizures are the most common clinical manifestations of critically ill neonates and often suggest serious diseases and complicated etiologies. The precise diagnosis of this disease can optimize the use of anti-seizure medication, reduce hospital costs, and improve the long-term neurodevelopmental outcomes. Currently, a few artificial intelligence-assisted diagnosis and treatment systems have been developed for neonatal seizures, but there is still a lack of high-level evidence for the diagnosis and treatment value in the real world. Based on an artificial intelligence-assisted diagnosis and treatment systems that has been developed for neonatal seizures, this study plans to recruit 370 neonates at a high risk of seizures from 6 neonatal intensive care units (NICUs) in China, in order to evaluate the effect of the system on the diagnosis, treatment, and prognosis of neonatal seizures in neonates with different gestational ages in the NICU. In this study, a diagnostic study protocol is used to evaluate the diagnostic value of the system, and a randomized parallel-controlled trial is designed to evaluate the effect of the system on the treatment and prognosis of neonates at a high risk of seizures. This multicenter prospective study will provide high-level evidence for the clinical application of artificial intelligence-assisted diagnosis and treatment systems for neonatal seizures in the real world.

Non-invasive detection of a femoral-to-radial arterial pressure gradient in intensive care patients with vasoactive agents

BACKGROUND

In patient requiring vasopressors, the radial artery pressure may underestimate the true central aortic pressure leading to unnecessary interventions. When using a femoral and a radial arterial line, this femoral-to-radial arterial pressure gradient (FR-APG) can be detected. Our main objective was to assess the accuracy of non-invasive blood pressure (NIBP) measures; specifically, measuring the gradient between the NIBP obtained at the brachial artery and the radial artery pressure and calculating the non-invasive brachial-to-radial arterial pressure gradient (NIBR-APG) to detect an FR-APG. The secondary objective was to assess the prevalence of the FR-APG in a targeted sample of critically ill patients.

METHODS

Adult patients in an intensive care unit requiring vasopressors and instrumented with a femoral and a radial artery line were selected. We recorded invasive radial and femoral arterial pressure, and brachial NIBP. Measurements were repeated each hour for 2 h. A significant FR-APG (our reference standard) was defined by either a mean arterial pressure (MAP) difference of more than 10 mmHg or a systolic arterial pressure (SAP) difference of more than 25 mmHg. The diagnostic accuracy of the NIBR-APG (our index test) to detect a significant FR-APG was estimated and the prevalence of an FR-APG was measured and correlated with the NIBR-APG.

RESULTS

Eighty-one patients aged 68 [IQR 58-75] years and an SAPS2 score of 35 (SD 7) were included from which 228 measurements were obtained. A significant FR-APG occurred in 15 patients with a prevalence of 18.5% [95%CI 10.8-28.7%]. Diabetes was significantly associated with a significant FR-APG. The use of a 11 mmHg difference in MAP between the NIBP at the brachial artery and the MAP of the radial artery led to a specificity of 92% [67; 100], a sensitivity of 100% [95%CI 83; 100] and an AUC ROC of 0.93 [95%CI 0.81-0.99] to detect a significant FR-APG. SAP and MAP FR-APG correlated with SAP (r2 = 0.36; p < 0.001) and MAP (r2 = 0.34; p < 0.001) NIBR-APG.

CONCLUSION

NIBR-APG assessment can be used to detect a significant FR-APG which occur in one in every five critically ill patients requiring vasoactive agents.

Optical coherence tomography angiography in healthy children: A comparison of macular structure

PURPOSE

The purpose of the present study was to investigate the perfusion density (PD) of macular superficial (SCP) and deep capillary plexus (DCP), the size of foveal avascular zone (FAZ) and central macular thickness (CMT) in healthy children using optical coherence tomography angiography (OCT-A).

PATIENTS AND METHODS

About 206 eyes of 111 children were analyzed. The correlation of gestational age (GA), birth weight (BW), age, sex, refractive errors, and visual acuity (VA) with OCT-A parameters were investigated.

RESULTS

The mean PD of the fovea and the mean FAZ area of SCP were 17.1% (DS: 4.26) and 234.47 (DS: 106.39) µm². The mean PD of the fovea and the mean FAZ area of DCP were 13.5% (DS: 5.23) and 298.32 (DS: 112.37) µm². Superficial and deep FAZ areas were not correlated with sex, age, BW, refractive errors, or VA. FAZ area of SCP was correlated with foveal PD (r = -0.76) and with CMT (r = -0.68). FAZ area of DCP was correlated with foveal's PD (r = -0.61). There was no correlation between CMT and refractive errors.

CONCLUSION

OCT-A may provide a non-invasive and reliable approach to evaluate macular perfusion in children. As the FAZ area, PD, and CMT change during the growth period, we performed established a reference range for different ages.

Diagnostic accuracy of a novel enzyme-linked immunoassay for the detection of IgG and IgG4 against Strongyloides stercoralis based on the recombinant antigens NIE/SsIR

Background

The diagnosis of strongyloidiasis is challenging. Serological tests are acknowledged to have high sensitivity, but issues due to cross-reactions with other parasites, native parasite antigen supply and intrinsic test variability do occur. Assays based on recombinant antigens could represent an improvement. The aim of this study was to assess the sensitivity and specificity of two novel immunoglobulin (Ig)G and IgG4 enzyme-linked immunosorbent assays (ELISAs) based on the recombinant antigens NIE/SsIR for the diagnosis of strongyloidiasis.

Methods

This was a retrospective diagnostic accuracy study. We included serum samples collected from immigrants from strongyloidiasis endemic areas for whom there was a matched result for Strongyloides stercoralis on agar plate culture and/or PCR assay, or a positive microscopy for S. stercoralis larvae. For the included samples, results were also available from an in-house indirect fluorescent antibody test (IFAT) and a commercial (Bordier ELISA; Bordier Affinity Products SA) ELISA. We excluded: (i) samples with insufficient serum volume; (ii) samples from patients treated with ivermectin in the previous 6 months; and (iii) sera from patients for whom only routine coproparasitology was performed after formol–ether concentration, if negative for S. stercoralis larvae. The performance of the novel assays was assessed against: (i) a primary reference standard, with samples classified as negative/positive on the basis of the results of fecal tests; (ii) a composite reference standard (CRS), which also considered patients to be positive who had concordant positive results for the IFAT and Bordier ELISA or with a single “high titer” positive result for the IFAT or Bordier ELISA. Samples with a single positive test, either for the IFAT or Bordier ELISA, at low titer, were considered to be “indeterminate,” and analyses were carried out with and without their inclusion.

Results

When assessed against the primary reference standard, the sensitivities of the IgG and IgG4 ELISAs were 92% (95% confidence interval [CI]: 88–97%) and 81% (95% CI: 74–87%), respectively, and the specificities were 91% (95% CI: 88–95%) and 94% (95% CI: 91–97%), respectively. When tested against the CRS, the IgG ELISA performed best, with 78% sensitivity (95% CI: 72–83%) and 98% specificity (95% CI: 96–100%), when a cut-off of 0.675 was applied and the indeterminate samples were excluded from the analysis.

Conclusion

The NIE-SsIR IgG ELISA demonstrated better accuracy than the IgG4 assay and was deemed promising particularly for serosurveys in endemic areas.

Graphical abstract

Supplementary Information

The online version contains supplementary material available at 10.1186/s13071-021-04916-x.

Clinical evaluation of an interoperable clinical decision-support system for the detection of systemic inflammatory response syndrome in critically ill children

Background

Systemic inflammatory response syndrome (SIRS) is defined as a non-specific inflammatory process in the absence of infection. SIRS increases susceptibility for organ dysfunction, and frequently affects the clinical outcome of affected patients. We evaluated a knowledge-based, interoperable clinical decision-support system (CDSS) for SIRS detection on a pediatric intensive care unit (PICU).

Methods

The CDSS developed retrieves routine data, previously transformed into an interoperable format, by using model-based queries and guideline- and knowledge-based rules. We evaluated the CDSS in a prospective diagnostic study from 08/2018–03/2019. 168 patients from a pediatric intensive care unit of a tertiary university hospital, aged 0 to 18 years, were assessed for SIRS by the CDSS and by physicians during clinical routine. Sensitivity and specificity (when compared to the reference standard) with 95% Wald confidence intervals (CI) were estimated on the level of patients and patient-days.

Results

Sensitivity and specificity was 91.7% (95% CI 85.5–95.4%) and 54.1% (95% CI 45.4–62.5%) on patient level, and 97.5% (95% CI 95.1–98.7%) and 91.5% (95% CI 89.3–93.3%) on the level of patient-days. Physicians’ SIRS recognition during clinical routine was considerably less accurate (sensitivity of 62.0% (95% CI 56.8–66.9%)/specificity of 83.3% (95% CI 80.4–85.9%)) when measurd on the level of patient-days. Evaluation revealed valuable insights for the general design of the CDSS as well as specific rule modifications. Despite a lower than expected specificity, diagnostic accuracy was higher than the one in daily routine ratings, thus, demonstrating high potentials of using our CDSS to help to detect SIRS in clinical routine.

Conclusions

We successfully evaluated an interoperable CDSS for SIRS detection in PICU. Our study demonstrated the general feasibility and potentials of the implemented algorithms but also some limitations. In the next step, the CDSS will be optimized to overcome these limitations and will be evaluated in a multi-center study.

Trial registration: NCT03661450 (ClinicalTrials.gov); registered September 7, 2018.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-021-01428-7.

Combining multiple biomarkers to linearly maximize the diagnostic accuracy under ordered multi-class setting

Either in clinical study or biomedical research, it is a common practice to combine multiple biomarkers to improve the overall diagnostic performance. Despite the fact there exist a large number of statistical methods for biomarker combination under binary classification, research on this topic under multi-class setting is sparse. The overall diagnostic accuracy, i.e. the sum of correct classification rates, directly measures the classification accuracy of the combined biomarkers. Hence the overall accuracy can serve as an important objective function for biomarker combination, especially when the combined biomarkers are used for the purpose of making medical diagnosis. In this paper, we address the problem of combining multiple biomarkers to directly maximize the overall diagnostic accuracy by presenting several grid search methods and derivation-based methods. A comprehensive simulation study was conducted to compare the performances of these methods. An ovarian cancer data set is analyzed in the end.

Diagnostic accuracy of FeNO [fractional exhaled nitric oxide] and asthma symptoms increased when evaluated with a superior reference standard

OBJECTIVES

The objective of the study is to determine the impact of changing reference standards (RS), namely spirometry vs. whole-body plethysmography (WBP), on estimation of the diagnostic accuracy of fractional exhaled nitric oxide (FeNO) and clinical signs and symptoms (CSS) as index tests regarding asthma diagnosis.

STUDY DESIGN AND SETTING

This was a diagnostic study conducted in 393 patients attending a private practice of pneumologists with complaints suspicious of asthma. First, the index tests were compared with the diagnostic results of spirometry in terms of forced expiratory volume in the first second (FEV₁) responsiveness. Second, the index tests were compared with the results of WBP in terms of specific airway resistance and FEV₁ responsiveness. Areas under the curve (AUC) were compared with a generalized estimating equation approach based on binary logistic regression.

RESULTS

FeNO values and CSS 'wheezing' and 'allergic rhinitis' showed higher specificities (P < 0.001) and sensitivities (not significant) when evaluated with WBP; also, Youden indices increased in these CSS (P < 0.05). AUC of FeNO in combination with 'wheezing' and 'allergic rhinitis' when WBP was used as RS (AUC = 0.724; 95% confidence interval 0.672 to 0.776) was higher compared with spirometry as RS (AUC = 0.654; 95% confidence interval 0.585 to 0.722) (P < 0.001).

CONCLUSION

In case of asthma, superior RS led to more favorable assessment of index tests. FeNO measurement might have been underestimated in some previous studies.

External Validation of a Periodontal Prediction Model for Identification of Diabetes among Saudi Adults

AIM AND OBJECTIVE

To externally validate the performance of a novel periodontal prediction model (PPM) for identification of diabetes among Saudi adults.

MATERIALS AND METHODS

The study was carried out among 150 adults attending primary care clinics in Riyadh (Saudi Arabia). The study adopted a temporal external validation approach, where the performance of the PPM was evaluated in the same location as the development study, but at a later time to allow for some variation between samples. A case-control approach was adopted, where diabetes status was first ascertained, followed by the completion of the Finnish Diabetes Risk Score (FINDRISC), Canadian Diabetes Risk (CANRISK) tools, and periodontal examinations.

RESULTS

The area under the curve (AUC) of the PPM (based on the number of missing teeth, the proportion of sites with pocket probing depth ≥6 mm, and mean pocket probing depth) was 0.514 (95% CI: 0.385, 0.642). The FINDRISC and CANRISK tools had AUC values of 0.871 (95% CI: 0.811-0.931) and 0.927 (95% CI: 0.884-0.971), respectively. The addition of the PPM did not improve the AUC of FINDRISC (p = 0.479) or CANRISK (p = 0.920). The decision curve analysis showed that there was no clinical benefit in adding the PPM to either tool. The PPM was updated with an overall adjustment factor for all existing predictors and three more periodontal measures.

CONCLUSION

In an external sample, the PPM had poor performance for identification of diabetes and no added value when combined with FINDRISC and CANRISK. The performance of the PPM improved after recalibration and extension.

CLINICAL SIGNIFICANCE

The results underscore the value of externally validating prediction models before applying them in clinical dental practice.

Role of preoperative magnetic resonance imaging in patients with quiescent septic arthritis undergoing primary total hip arthroplasty

AIM

to evaluate the role of preoperative magnetic resonance imaging (MRI) in assessing patients with a history of quiescent hip septic arthritis undergoing total hip arthroplasty.

MATERIALS AND METHODS

retrospective consecutive study of patients with previous history of septic arthritis who underwent MRI scans of their hips prior to primary hip arthroplasty surgery and who also had minimum 2 years follow up postoperatively. Detailed radiographic examinations were obtained, demographic and microbiological data collected. The primary outcome measure was whether a preoperative MRI scan had influenced the surgical decision-making and planning. Rate of recurrence of infection and complications was also collected at final follow up.

RESULTS

sixteen patients with quiescent hip septic arthritis were included. There were 4 males and 12 females with average age at time of primary hip arthroplasty 51.7 years (range 22-75). Five patients had childhood septic arthritis with no documented microbiology data. Eleven patients had adult onset septic arthritis. In patients with childhood septic arthritis the MRI findings were similar to those with degenerative joint disease and had no added value to the routine surgical work up. MRIs of patients with adult onset septic arthritis showed persistent findings of effusion, marrow oedema and soft tissue oedema and had no added value to the surgical planning. All but one underwent single stage total hip arthroplasty. At final follow up, with average 4.6 years (range 2-8), none had a recurrence of infection.

CONCLUSION

In our experience, preoperative MRI scans did not influence the surgical decision making and are not recommended for routine practice in the surgical work up of quiescent septic arthritis prior to total hip arthroplasty.

Issues and solutions in biomarker evaluation when subclasses are involved under binary classification

In practice, it is common to evaluate biomarkers in binary classification settings (e.g. non-cancer vs. cancer) where one or both main classes involve multiple subclasses. For example, non-cancer class might consist of healthy subjects and benign cases, while cancer class might consist of subjects at early and late stages. The standard practice is pooling within each main class, i.e. all non-cancer subclasses are pooled together to create a control group, and all cancer subclasses are pooled together to create a case group. Based on the pooled data, the area under ROC curve (AUC) and other characteristics are estimated under binary classification for the purpose of biomarker evaluation. Despite the popularity of this pooling strategy in practice, its validity and implication in biomarker evaluation have never been carefully inspected. This paper aims to demonstrate that pooling strategy can be seriously misleading in biomarker evaluation. Furthermore, we present a new diagnostic framework as well as new accuracy measures appropriate for biomaker evaluation under such settings. In the end, an ovarian cancer data set is analyzed.

Inter-rater reliability of amplitude-integrated EEG for the detection of neonatal seizures

BACKGROUND

Amplitude-integrated electroencephalogram (aEEG) is being used increasingly for seizure detection in neonates. However, data regarding inter-rater reliability among neonatologists for the use of aEEG for the detection of neonatal seizures is lacking.

METHODS

Term and late-preterm infants at risk of seizures were monitored simultaneously with 24-h video-electroencephalography (vEEG) and aEEG. vEEG was interpreted by an experienced neurologist. Five neonatologists with experience in aEEG interpretation from four different neonatal units interpreted aEEG recordings independently. The Brennan and Prediger kappa coefficient and Intra-class Correlation Coefficients (ICC) were used to assess inter-rater reliability between the neonatologists.

RESULTS

Thirty-five infants at risk of seizure with gestational age at birth 35-42 weeks were recruited for the study after informed parental consent. vEEG detected seizures in seven infants with a total of 169 individual seizure episodes. Neonatologists detected seizures in 10 to 15 infants on aEEG. The sensitivities for the detection of individual seizures by neonatologists ranged from 18% to 38%. The inter-rater reliability for detection of: individual seizure was "fair" (kappa = 0.37; 95% CI: 0.32-0.42), infant with seizure was "moderate" (kappa = 0.60; 95% CI: 0.44-0.75), duration of individual seizure (ICC: 0.22; 95% CI: 0.18-0.28) and total duration of seizures in an infant (ICC: 0.46; 95% CI: 0.30-0.63) was "poor". The neonatologists missed 77-90% of the duration of seizures.

CONCLUSION

The inter-rater reliability of aEEG for the detection of neonatal seizures was suboptimal. Even when interpreted by experienced and trained clinicians, seizure detection with aEEG has limitations and can miss large number and duration of seizures.

Accuracy of surgeon prediction of appendicitis severity in pediatric patients

PURPOSE

Clinical prediction of disease severity is important as one considers nonoperative management of simple appendicitis. This study assesses the accuracy of surgeons' prediction of appendicitis severity.

METHODS

From February to August 2016, pediatric surgeons at a single institution were asked to predict whether patients had simple or complex appendicitis preoperatively based on clinical data, imaging, and general assessment. Receiver operating characteristic curves were generated to determine area under the curve (AUC) and optimal cutoff points of clinical findings for diagnosing simple appendicitis. Outcomes included sensitivity and specificity of variables to identify simple appendicitis. Predictions were compared to operative findings using χ². A p-value<0.05 was considered statistically significant.

RESULTS

Of 125 cases (median age 9 years [IQR 7-13], 58% male), simple appendicitis was predicted in 77 (62%) and complex appendicitis in 48 (38%). Predictions were accurate in 59 (77%) simple cases and 45 (94%) complex cases. Although surgeon prediction was more accurate than individual imaging or clinical findings and was highly sensitive (95%) for diagnosing simple appendicitis, specificity was only 71%. Lower WBC (<15.5 × 10³/μL, AUC 0.61, p = 0.05), afebrile (<100.4 °F, AUC 0.86, p < 0.01), and shorter symptom duration (≤ 1.5 days, AUC 0.71, p < 0.001) were associated with simple appendicitis. Of 18 complex cases (14%) inaccurately predicted as simple, 17 (94%) lacked diffuse tenderness, 15 (83%) were well-appearing, 11 (61%) had ultrasound findings of simple appendicitis, 11 (61%) had ≤2 days of symptoms, and 8 (44%) were afebrile (<100.4 °F).

CONCLUSION

While surgeon prediction of simple appendicitis is more accurate than ultrasound or clinical data alone, diagnostic accuracy is still limited.

TYPE OF STUDY

Prospective survey.

LEVEL OF EVIDENCE

II.

Diagnostic accuracy of capnovolumetry for the identification of airway obstruction - results of a diagnostic study in ambulatory care

BACKGROUND

One of the known weaknesses of spirometry is its dependence on patients' cooperation, which can only partially be alleviated by educational efforts. Therefore, procedures less dependent on cooperation might be of value in clinical practice. We investigated the diagnostic accuracy of ultrasound-based capnovolumetry for the identification of airway obstruction.

METHODS

Consecutive patients from a pulmonary outpatient clinic were included in the diagnostic study. As reference standard, the presence of airway obstruction was evaluated via spirometry and bodyplethysmography. Capnovolumetry was performed as index test with an ultrasound spirometer providing a surrogate measure of exhaled carbon dioxide. Receiver operating characteristic (ROC) analysis was performed using the ratio of slopes of expiratory phases 3 and 2 (s3/s2) ≥ 0.10 as primary capnovolumetric parameter for the recognition of airway obstruction. Logistic regression was performed as secondary analysis to identify further useful capnovolumetric parameters. The diagnostic potential of capnovolumetry to identify more severe degrees of airway obstruction was evaluated additionally.

RESULTS

Of 1400 patients recruited, 1287 patients were included into the analysis. Airway obstruction was present in 29% of patients. The area under the ROC-curve (AUC) of s3/s2 was 0.678 (95% CI 0.645, 0.710); sensitivity of s3/s2 ≥ 0.10 was 47.7 (95% CI 42.7, 52.8)%, specificity 79.0 (95% CI 76.3, 81.6)%. When combining this parameter with three other parameters derived from regression analysis (ratio area/volume phase 3, slope phase 3, volume phase 2), an AUC of 0.772 (95% CI 0.743, 0.801) was obtained. For severe airway obstruction (FEV₁ ≤ 50% predicted) sensitivity of s3/s2 ≥ 0.10 was 75.9 (95% CI 67.1, 83.0)%, specificity 75.8 (95% CI 73.3, 78.1)%; for very severe airway obstruction (FEV₁ ≤ 30% predicted) sensitivity was 86.7 (95% CI 70.3, 94.7)%, specificity 72.8 (95% CI 70.3, 75.2)%. Sensitivities increased and specificities decreased considerably when the combined capnovolumetric score was used as index test.

CONCLUSIONS

Capnovolumetry by way of an ultrasound spirometer had a statistically significant albeit moderate potential for the recognition of airway obstruction in a heterogeneous population of patients typically found in clinical practice. Diagnostic accuracy of the capnovolumetric device increased with the severity of airway obstruction.

TRIAL REGISTRATION

The study is registered under DRKS00013935 at German Clinical Trials Register (DRKS).

Evaluation of a simple tool to assess the results of Ponseti treatment for use by clubfoot therapists: a diagnostic accuracy study

Background

We aimed to develop and evaluate a tool for clubfoot therapists in low resource settings to assess the results of Ponseti treatment of congenital talipes equinovarus, or clubfoot, in children of walking age.

Method

A literature review and a Delphi process based on the opinions of 35 Ponseti trainers in Africa were used to develop the Assessing Clubfoot Treatment (ACT) tool and score. We followed up children with clubfoot from a cohort treated between 2011 and 2013, in 2017. A full clinical assessment was conducted to decide if treatment was successful or if further treatment was required. The ACT score was then calculated for each child. Inter-observer variation for the ACT tool was assessed. Sensitivity, specificity, positive and negative predictive values were calculated for the ACT score compared to full clinical assessment (gold standard). Predictors of a successful outcome were explored.

Results

The follow up rate was 31.2% (68 children). The ACT tool consisted of 4 questions; each scored from 0 to 3, giving a total from 0 to 12 where 12 is the ideal result. The 4 questions included one physical assessment and three parent reported outcome measures. It took 5 min to administer and had excellent inter-observer agreement.

An ACT score of 8 or less demonstrated 79% sensitivity and 100% specificity in identifying children that required further intervention, with a positive predictive value of 100% and negative predictive value of 90%. Children who completed two or more years of bracing were four times more likely to achieve an ACT score of 9 or more compared to those who did not (OR: 4.08, 95% CI: 1.31–12.65, p = 0.02).

Conclusions

The ACT tool is simple to administer, had excellent observer agreement, and good sensitivity and specificity in identifying children who need further intervention. The score can be used to identify those children who definitely need referral and further treatment (score 8 or less) and those with a definite successful outcome (score 11 or more), however further discrimination is needed to decide how to manage children with a borderline ACT score of 9 or 10.

Level of evidence

Level II, Diagnostic Study.

Electronic supplementary material

The online version of this article (10.1186/s13047-019-0323-4) contains supplementary material, which is available to authorized users.

Ultrasensitive detection of oncogenic human papillomavirus in oropharyngeal tissue swabs

BACKGROUND

The incidence of oropharyngeal squamous cell carcinoma (OPSCC) caused by oncogenic human papillomavirus (HPV) is rising worldwide. HPV-OPSCC is commonly diagnosed by RT-qPCR of HPV E6 and E7 oncoproteins or by p16 immunohistochemistry (IHC). Droplet digital PCR (ddPCR) has been recently reported as an ultra-sensitive and highly precise method of nucleic acid quantification for biomarker analysis. To validate the use of a minimally invasive assay for detection of oncogenic HPV based on oropharyngeal swabs using ddPCR. Secondary objectives were to compare the accuracy of ddPCR swabs to fresh tissue p16 IHC and RT-qPCR, and to compare the cost of ddPCR with p16 IHC.

METHODS

We prospectively included patients with p16⁺ oral cavity/oropharyngeal cancer (OC/OPSCC), and two control groups: p16^- OC/OPSCC patients, and healthy controls undergoing tonsillectomy. All underwent an oropharyngeal swab with ddPCR for quantitative detection of E6 and E7 mRNA. Surgical specimens had p16 IHC performed. Agreement between ddPCR and p16 IHC was determined for patients with p16 positive and negative OC/OPSCC as well as for healthy control patients. The sensitivity and specificity of ddPCR of oropharyngeal swabs were calculated against p16 IHC for OPSCC.

RESULTS

122 patients were included: 36 patients with p16⁺OPSCC, 16 patients with p16^-OPSCC, 4 patients with p16⁺OCSCC, 41 patients with p16^-OCSCC, and 25 healthy controls. The sensitivity and specificity of ddPCR of oropharyngeal swabs against p16 IHC were 92 and 98% respectively, using 20-50 times less RNA than that required for conventional RT-qPCR. Overall agreement between ddPCR of tissue swabs and p16 of tumor tissue was high at ĸ = 0.826 [0.662-0.989].

CONCLUSION

Oropharyngeal swabs analyzed by ddPCR is a quantitative, rapid, and effective method for minimally invasive oncogenic HPV detection. This assay represents the most sensitive and accurate mode of HPV detection in OPSCC without a tissue biopsy in the available literature.

Diagnostic accuracy of audio-based seizure detection in patients with severe epilepsy and an intellectual disability

We evaluated the performance of audio-based detection of major seizures (tonic-clonic and long generalized tonic) in adult patients with intellectual disability living in an institute for residential care.

METHODS

First, we checked in a random sample (n=17, 102 major seizures) how many patients have recognizable sounds during these seizures. In the second part of this trial, we followed 10 patients (who had major seizures with recognizable sounds) during four weeks with an acoustic monitoring system developed by CLB ('CLB-monitor') and video camera. In week 1, we adapted the sound detection threshold until, per night, a maximum of 20 sounds was found. During weeks 2-4, we selected the epilepsy-related sounds and performed independent video verification and labeling ('snoring', 'laryngeal contraction') of the seizures. The video images were also fully screened for false negatives. In the third part, algorithms in the CLB-monitor detected one specific sound (sleep-related snoring) to illustrate the value of automatic sound recognition.

RESULTS

Part 1: recognizable sounds (louder than whispering) occurred in 23 (51%) of the 45 major seizures, 20 seizures (45%) were below this threshold, and 2 (4%) were without any sound. Part 2: in the follow-up group (n=10, 112 major seizures; mean: 11.2, range: 1-30), we found a mean sensitivity of 0.81 (range: 0.33-1.00) and a mean positive predictive value of 0.40 (range: 0.06-1.00). All false positive alarms (mean value: 1.29 per night) were due to minor seizures. We missed 4 seizures (3%) because of lack of sound and 10 (9%) because of sounds below the system threshold. Part 3: the machine-learning algorithms in the CLB-monitor resulted in an overall accuracy for 'snoring' of 98.3%.

CONCLUSIONS

Audio detection of major seizures is possible in half of the patients. Lower sound detection thresholds may increase the proportion of suitable candidates. Human selection of seizure-related sounds has a high sensitivity and moderate positive predictive value because of minor seizures which do not need intervention. Algorithms in the CLB-monitor detect seizure-related sounds and may be used alone or in multimodal systems.

The accuracy of different methods for diagnosing septal deviation in patients undergoing septorhinoplasty: A prospective study

OBJECTIVE

This study aimed to determine the diagnostic accuracy of different diagnostic tests in predicting nasal septum deformities during preoperative planning for septorhinoplasty.

METHODS

Consecutive patients who underwent septorhinoplasty between June 2011 and August 2012 were included (n = 30) and underwent a protocol of diagnostic tests, including nasal speculoscopy, craniofacial computed tomography (CT), three-dimensional (3D) reconstruction of the nasal septum by CT and nasal endoscopy. A modified Guyuron classification of septal deformities was used for classifying the septal deviations. Direct surgical assessment of the nasal septum during open septorhinoplasty was the reference standard with which each of the diagnostic tests was compared. Sensitivity, specificity and predictive values of each test were calculated.

RESULTS

The preoperative diagnosis was nasal bone fracture in 11 patients, nasal septal fracture in 15 and post-traumatic nasal deformity in four. For type A deviations (localised), craniofacial CT showed the highest performance with a sensitivity of 100%, specificity of 100%, positive predictive value (PPV) of 100% and negative predictive value (NPV) of 99%. For type B septal deformations (C shape), nasal endoscopy (sensitivity, 100%; specificity, 87.5%; PPV, 87.7%; and NPV, 100%) showed the highest performance. For type C deformities (S shape), nasal endoscopy (sensitivity, 70%; specificity, 100%; PPV, 100%; and NPV, 87%) showed the highest performance. The accuracy for nasal endoscopy was 27/30 (90%), 26/30 (87%) for craniofacial CT, 22/30 (73%) for 3D reconstruction and 10/28 (36%) for speculoscopy.

CONCLUSIONS

Nasal endoscopy and craniofacial CT were more accurate and precise than nasal speculoscopy and 3D reconstruction for preoperative evaluation of the nasal septum, thus enabling more appropriate surgical planning for septorhinoplasty.

Added value of cost-utility analysis in simple diagnostic studies of accuracy: (18)F-fluoromethylcholine PET/CT in prostate cancer staging

Diagnostic studies of accuracy targeting sensitivity and specificity are commonly done in a paired design in which all modalities are applied in each patient, whereas cost-effectiveness and cost-utility analyses are usually assessed either directly alongside to or indirectly by means of stochastic modeling based on larger randomized controlled trials (RCTs). However the conduct of RCTs is hampered in an environment such as ours, in which technology is rapidly evolving. As such, there is a relatively limited number of RCTs. Therefore, we investigated as to which extent paired diagnostic studies of accuracy can be also used to shed light on economic implications when considering a new diagnostic test. We propose a simple decision tree model-based cost-utility analysis of a diagnostic test when compared to the current standard procedure and exemplify this approach with published data from lymph node staging of prostate cancer. Average procedure costs were taken from the Danish Diagnosis Related Groups Tariff in 2013 and life expectancy was estimated for an ideal 60 year old patient based on prostate cancer stage and prostatectomy or radiation and chemotherapy. Quality-adjusted life-years (QALYs) were deduced from the literature, and an incremental cost-effectiveness ratio (ICER) was used to compare lymph node dissection with respective histopathological examination (reference standard) and (18)F-fluoromethylcholine positron emission tomography/computed tomography (FCH-PET/CT). Lower bounds of sensitivity and specificity of FCH-PET/CT were established at which the replacement of the reference standard by FCH-PET/CT comes with a trade-off between worse effectiveness and lower costs. Compared to the reference standard in a diagnostic accuracy study, any imperfections in accuracy of a diagnostic test imply that replacing the reference standard generates a loss in effectiveness and utility. We conclude that diagnostic studies of accuracy can be put to a more extensive use, over and above a mere indication of sensitivity and specificity of an imaging test, and that health economic considerations should be undertaken when planning a prospective diagnostic accuracy study. These endeavors will prove especially fruitful when comparing several imaging techniques with one another, or the same imaging technique using different tracers, with an independent reference standard for the evaluation of results.