Replicability of simulation studies for the investigation of statistical methods: the RepliSims project

Results of simulation studies evaluating the performance of statistical methods can have a major impact on the way empirical research is implemented. However, so far there is limited evidence of the replicability of simulation studies. Eight highly cited statistical simulation studies were selected, and their replicability was assessed by teams of replicators with formal training in quantitative methodology. The teams used information in the original publications to write simulation code with the aim of replicating the results. The primary outcome was to determine the feasibility of replicability based on reported information in the original publications and supplementary materials. Replicasility varied greatly: some original studies provided detailed information leading to almost perfect replication of results, whereas other studies did not provide enough information to implement any of the reported simulations. Factors facilitating replication included availability of code, detailed reporting or visualization of data-generating procedures and methods, and replicator expertise. Replicability of statistical simulation studies was mainly impeded by lack of information and sustainability of information sources. We encourage researchers publishing simulation studies to transparently report all relevant implementation details either in the research paper itself or in easily accessible supplementary material and to make their simulation code publicly available using permanent links.

Perspective; recommendations for improved patient participation in decision-making for geriatric patients in acute surgical settings

Geriatric patients often present to the hospital in acute surgical settings. In these settings, shared decision-making as equal partners can be challenging. Surgeons should recognize that geriatric patients, and frail patients in particular, may sometimes benefit from de-escalation of care in a palliative setting rather than curative treatment. To provide more person-centred care, better strategies for improved shared decision-making need to be developed and implemented in clinical practice. A shift in thinking from a disease-oriented paradigm to a patient-goal-oriented paradigm is required to provide better person-centred care for older patients. We may greatly improve the collaboration with patients if we move parts of the decision-making process to the pre-acute phase. In the pre-acute phase appointing legal representatives, having goals of care conversations, and advance care planning can help give physicians an idea of what is important to the patient in acute settings. When making decisions as equal partners is not possible, a greater degree of physician responsibility may be appropriate. Physicians should tailor the "sharedness" of the decision-making process to the needs of the patient and their family.

Concepts, utilization, and perspectives on the Dutch Nationwide Trauma registry: a position paper

Over the last decades, the Dutch trauma care have seen major improvements. To assess the performance of the Dutch trauma system, in 2007, the Dutch Nationwide Trauma Registry (DNTR) was established, which developed into rich source of information for quality assessment, quality improvement of the trauma system, and for research purposes. The DNTR is one of the most comprehensive trauma registries in the world as it includes 100% of all trauma patients admitted to the hospital through the emergency department. This inclusive trauma registry has shown its benefit over less inclusive systems; however, it comes with a high workload for high-quality data collection and thus more expenses. The comprehensive prospectively collected data in the DNTR allows multiple types of studies to be performed. Recent changes in legislation allow the DNTR to include the citizen service numbers, which enables new possibilities and eases patient follow-up. However, in order to maximally exploit the possibilities of the DNTR, further development is required, for example, regarding data quality improvement and routine incorporation of health-related quality of life questionnaires. This would improve the quality assessment and scientific output from the DNTR. Finally, the DNTR and all other (European) trauma registries should strive to ensure that the trauma registries are eligible for comparisons between countries and healthcare systems, with the goal to improve trauma patient care worldwide.

Effects of reduction technique for acute anterior shoulder dislocation without sedation or intra-articular pain management: a systematic review and meta-analysis

INTRODUCTION

Anterior shoulder dislocations are commonly seen in the emergency department for which several closed reduction techniques exist. The aim of this systematic review is to identify the most successful principle of closed reduction techniques for an acute anterior shoulder dislocation in the emergency department without the use of sedation or intra-articular lidocaine injection.

METHODS

A literature search was conducted up to 15-08-2022 in the electronic databases of PubMed, Embase and CENTRAL for randomized and observational studies comparing two or more closed reduction techniques for anterior shoulder dislocations. Included techniques were grouped based on their main operating mechanism resulting in a traction-countertraction (TCT), leverage and biomechanical reduction technique (BRT) group. The primary outcome was success rate and secondary outcomes were reduction time and endured pain scores. Meta-analyses were conducted between reduction groups and for the primary outcome a network meta-analysis was performed.

RESULTS

A total of 3118 articles were screened on title and abstract, of which 9 were included, with a total of 987 patients. Success rates were 0.80 (95% CI 0.74; 0.85), 0.81 (95% CI 0.63; 0.92) and 0.80 (95% CI 0.56; 0.93) for BRT, leverage and TCT, respectively. No differences in success rates were observed between the three separate reduction groups. In the network meta-analysis, similar yet more precise effect estimates were found. However, in a post hoc analysis the BRT group was more successful than the combined leverage and TCT group with a relative risk of 1.33 (95% CI 1.19, 1.48).

CONCLUSION

All included techniques showed good results with regard to success of reduction. The BRT might be the preferred technique for the reduction of an anterior shoulder dislocation, as patients experience the least pain and it results in the fastest reduction.

Measurement error in echocardiographic assessment of aortic stenosis: an epidemiological consideration of research methodology and clinical practice

Background/Introduction

Current international guidelines on treatment of valvular heart disease contain recommendations for aortic valve replacement based exclusively on abnormal echocardiographic parameters in asymptomatic patients with aortic stenosis (AS). To prevent misclassification and subsequent mistreatment of patients, these recommendations require accurate and unbiased measurements of hemodynamic parameters.

Purpose

To illustrate consideration and impact of measurement error in echocardiographic assessment of aortic stenosis in both the research and clinical setting.

Methods

First, a systematic review was performed to investigate the recognition of and correction for measurement error in clinical studies on the prognostic value of peak aortic jet velocity (Vmax), mean pressure gradient (MPG), and effective orifice area (EOA). Second, all potential erroneous sources in the calculation of those primary parameters were listed stratified to random or systematic measurement error, and subsequently their magnitude was quantified. Third, the impact of various types of measurement error on current thresholds for intervention was graphically illustrated in different clinical scenarios.

Results

The presence of measurement error was acknowledged in 44% of the 36 included studies, while none utilized methods to correct for it. Interobserver variability ranged between 0.9–8.3% for Vmax and MPG but was substantially higher for EOA (range 7.7–12.7%) implying lower reliability (Figure 1). Furthermore, the invalid assumption of a circular left ventricular outflow tract area resulted in a median underestimation in EOA of 22.5% compared to 3D-transesophageal echocardiography (3D-TEE), computed tomography (CT), and cardiovascular magnetic resonance (CMR) planimetry. Figure 2 illustrates the impact of this discrepancy on the classification of AS using one-sided t-tests to determine the areas under the curve; the proportion of patients with non-severe AS in a hypothetical cohort (based on values from the PARTNER 3 trial [1[) increased by 42%.

Conclusion(s)

Measurement error is underrecognized in studies of echocardiographic assessment of aortic stenosis. This review demonstrates that random and systematic measurement errors affect echocardiographic assessment, leading to potential misdiagnosis and subsequent mistreatment. Clinicians and scientists should be aware of the implications of measurement errors to enhance rightful clinical decision-making and assure research validity.

Funding Acknowledgement

Type of funding sources: None.

Publication rates in small German trials remained low five years after trial completion

OBJECTIVE

To investigate publication rates in small trials and to explore which factors are associated with publication rates in small trials, including sample size, the type and number of primary and secondary outcomes.

STUDY DESIGN AND SETTING

We studied a subgroup of 'small' trials from a pre-existing dataset (IntoValue), containing German trials completed between 2009 and 2017. Small trials were defined as phase II-III, III and IV trials with 150 or fewer participants. We performed an updated publication search and collected additional data from online trial records.

RESULTS

Out of 499 trials, 325 (65%) trials published their results in a journal article or dissertation. Median time-to-publication was 3.41 years (95% CI: 3.04-4.10). Planned sample size was not associated with publication rates, but the difference between planned and achieved sample size was (per 10% unsuccessfully recruited participants, HR = 0.95, 95% CI: 0.91-1.00). Phase III vs. II-III trials, studied intervention (device vs. other) and clearly vs. unclearly defined primary outcomes predicted a higher likelihood of earlier publication.

CONCLUSION

About 35% of small trials in Germany remain unpublished, even after an extensive follow-up period of over 9 years. Publication rates are low and were associated with sample size, trial phase and type of intervention.

Real-World Metastatic Renal Cell Carcinoma Treatment Patterns and Clinical Outcomes in The Netherlands

The number of treatment options for patients with metastatic renal cell carcinoma (mRCC) has significantly grown in the last 15 years. Although randomized controlled trials are fundamental in investigating mRCC treatment efficacy, their external validity can be limited. Therefore, the efficacy of the different treatment options should also be evaluated in clinical practice. We performed a chart review of electronic health records using text mining software to study the current treatment patterns and outcomes. mRCC patients from two large hospitals in the Netherlands, starting treatment between January 2015 and May 2020, were included. Data were collected from electronic health records using a validated text mining tool. Primary endpoints were progression-free survival (PFS) and overall survival (OS). Statistical analyses were performed using the Kaplan-Meier method. Most frequent first-line treatments were pazopanib (n = 70), sunitinib (n = 34), and nivolumab with ipilimumab (n = 28). The overall median PFS values for first-line treatment were 15.7 months (95% confidence interval [95%CI], 8.8-20.7), 16.3 months (95%CI, 9.3-not estimable [NE]) for pazopanib, and 6.9 months (95% CI, 4.4-NE) for sunitinib. The overall median OS values were 33.4 months (95%CI, 28.1-50.9 months), 39.3 months (95%CI, 29.5-NE) for pazopanib, and 28.1 months (95%CI, 7.0-NE) for sunitinib. For nivolumab with ipilimumab, median PFS and median OS were not reached. Of the patients who finished first- and second-line treatments, 64 and 62% received follow-up treatments, respectively. With most patients starting on pazopanib and sunitinib, these real-world treatment outcomes were most likely better than in pivotal trials, which may be due to extensive follow-up treatments.

[Trials, observational research and the real world]

The treatment effect found in a randomized trial does not always correspond to the effect of the treatment in daily practice. To estimate the applicability of the results of a trial, a comparison can be made with the results of observational research. In this commentary we discuss such a comparison between the results of the TIME trial and the analysis of the observational DUCA database. Both compared open and minimally invasive oesophageal resection, yet results were strikingly different. We discuss nine possible explanations for the differences found in the effects of the two treatments.

Effectiveness and toxicity of lenvatinib in refractory thyroid cancer: Dutch real-life data

OBJECTIVE

The SELECT trial showed progression-free survival (PFS) benefit for lenvatinib for advanced radioiodine-refractory differentiated thyroid cancer (RAI-refractory or RR-DTC) patients, on which current clinical practice is based. We assessed whether the effectiveness and toxicity of lenvatinib in real-life clinical practice in the Netherlands were comparable to the pivotal SELECT trial.

METHODS

From three Dutch centres Electronic Health Records (EHRs) of patients treated in the lenvatinib compassionate use program or as standard of care were reviewed and checked for SELECT eligibility criteria. Baseline characteristics, safety, and efficacy measures were compared and PFS and overall survival (OS) were calculated. Furthermore, PFS was compared to estimates of PFS reported in other studies.

RESULTS

A total of 39 DTC patients with a median age of 62 years were analysed. Of these, 27 patients (69%) did not fulfil the SELECT eligibility criteria. The most common grade ≥3 toxicities were hypertension (n = 11, 28%), diarrhoea (n = 7, 18%), vomiting (n = 4, 10%), and gallbladder disease (n = 3, 8%). Median PFS and median OS were 9.7 (95% confidence interval (CI): 4.0-15.5) and 18.3 (95% CI: 4.9-31.7) months, respectively, response rate was 38% (95% CI: 23-54%). PFS in the Dutch real-life situation was comparable to previous real-life studies, but inferior to PFS as shown in the SELECT trial (P = 0.04).

CONCLUSIONS

PFS in our non-trial population was significantly shorter than in the SELECT trial population. In the interpretation of results, differences in the real-life population and the SELECT study population regarding patient characteristics should be taken into account.

Measurement error in continuous endpoints in randomised trials: Problems and solutions

In randomised trials, continuous endpoints are often measured with some degree of error. This study explores the impact of ignoring measurement error and proposes methods to improve statistical inference in the presence of measurement error. Three main types of measurement error in continuous endpoints are considered: classical, systematic, and differential. For each measurement error type, a corrected effect estimator is proposed. The corrected estimators and several methods for confidence interval estimation are tested in a simulation study. These methods combine information about error‐prone and error‐free measurements of the endpoint in individuals not included in the trial (external calibration sample). We show that, if measurement error in continuous endpoints is ignored, the treatment effect estimator is unbiased when measurement error is classical, while Type‐II error is increased at a given sample size. Conversely, the estimator can be substantially biased when measurement error is systematic or differential. In those cases, bias can largely be prevented and inferences improved upon using information from an external calibration sample, of which the required sample size increases as the strength of the association between the error‐prone and error‐free endpoint decreases. Measurement error correction using already a small (external) calibration sample is shown to improve inferences and should be considered in trials with error‐prone endpoints. Implementation of the proposed correction methods is accommodated by a new software package for R.

Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective

It is widely acknowledged that the predictive performance of clinical prediction models should be studied in patients that were not part of the data in which the model was derived. Out‐of‐sample performance can be hampered when predictors are measured differently at derivation and external validation. This may occur, for instance, when predictors are measured using different measurement protocols or when tests are produced by different manufacturers. Although such heterogeneity in predictor measurement between derivation and validation data is common, the impact on the out‐of‐sample performance is not well studied. Using analytical and simulation approaches, we examined out‐of‐sample performance of prediction models under various scenarios of heterogeneous predictor measurement. These scenarios were defined and clarified using an established taxonomy of measurement error models. The results of our simulations indicate that predictor measurement heterogeneity can induce miscalibration of prediction and affects discrimination and overall predictive accuracy, to extents that the prediction model may no longer be considered clinically useful. The measurement error taxonomy was found to be helpful in identifying and predicting effects of heterogeneous predictor measurements between settings of prediction model derivation and validation. Our work indicates that homogeneity of measurement strategies across settings is of paramount importance in prediction research.

[Significance of p-values: misinterpreted and overrated]

- An often provided interpretation of a significant p-value (p < 0.05) is that 'the probability the conclusion is incorrect, is only 5%'. This interpretation is incorrect.- It can be shown that for observational studies, in case of p < 0.05, the probability of a false positive signal is around 50%. This means that significant p-values give us much less certainty about the reliability of a conclusion than we like to believe.- Much would be gained already if the emphasis on p-values would be replaced by: (a) an estimation of the effect size in combination with the corresponding statistical uncertainty (represented by the confidence interval), (b) an assessment of the clinical relevance of that effect, and

[Power, you can never have enough of it]

- In randomised trials on the effects of a medical treatment, the power of the trial corresponds to the conditional probability that the conclusion of the trial will be that the treatment is effective, given a certain treatment effect.- The time to consider the power of a trial is before conducting the study, to ensure that the design of the trial is such that there is a reasonable chance of demonstrating a clinically relevant effect.

[On the timing of randomised studies: the sooner initiated, the better?]

The study design for randomised double-blind studies is powerful. Randomisation and blinding ensure that the groups that are compared are truly exchangeable. Any differences in health outcomes can be attributed rightfully to the one aspect on which the study groups differ: the treatment. In this commentary, we argue that the use of this powerful study design at the wrong moment can lead to undesirable situations.

The effects of misclassification in routine healthcare databases on the accuracy of prognostic prediction models: a case study of the CHA2DS2-VASc score in atrial fibrillation

BACKGROUND

Research on prognostic prediction models frequently uses data from routine healthcare. However, potential misclassification of predictors when using such data may strongly affect the studied associations. There is no doubt that such misclassification could lead to the derivation of suboptimal prediction models. The extent to which misclassification affects the validation of existing prediction models is currently unclear.We aimed to quantify the amount of misclassification in routine care data and its effect on the validation of the existing risk prediction model. As an illustrative example, we validated the CHA2DS2-VASc prediction rule for predicting mortality in patients with atrial fibrillation (AF).

METHODS

In a prospective cohort in general practice in the Netherlands, we used computerized retrieved data from the electronic medical records of patients known with AF as index predictors. Additionally, manually collected data after scrutinizing all complete medical files were used as reference predictors. Comparing the index with the reference predictors, we assessed misclassification in individual predictors by calculating Cohen's kappas and other diagnostic test accuracy measures. Predictive performance was quantified by the c-statistic and by determining calibration of multivariable models.

RESULTS

In total, 2363 AF patients were included. After a median follow-up of 2.7 (IQR 2.3-3.0) years, 368 patients died (incidence rate 6.2 deaths per 100 person-years). Misclassification in individual predictors ranged from substantial (Cohen's kappa 0.56 for prior history of heart failure) to minor (kappa 0.90 for a history of type 2 diabetes). The overall model performance was not affected when using either index or reference predictors, with a c-statistic of 0.684 and 0.681, respectively, and similar calibration.

CONCLUSION

In a case study validating the CHA2DS2-VASc prediction model, we found substantial predictor misclassification in routine healthcare data with only limited effect on overall model performance. Our study should be repeated for other often applied prediction models to further evaluate the usefulness of routinely available healthcare data for validating prognostic models in the presence of predictor misclassification.

[Added value of observational studies in surgery: the hierarchical structure of study designs requires a more refined approach]

The randomised placebo-controlled trial (RCT) is the gold standard for the evaluation of medical interventions. Observational studies, on the other hand, usually do not get much credit. For studies investigating surgical interventions this does not always seem entirely justified. A more refined approach might be needed for the often-used hierarchical structure of research designs. Instead of a strict separation of results from RCTs and other designs, results of the different designs should rather be regarded as complementary to each other when evaluating surgical interventions in traumatology.

Quality of life from a randomized trial of open and endovascular repair for abdominal aortic aneurysm

Background

Long-term survival is similar after open or endovascular repair of abdominal aortic aneurysm. Few data exist on the effect of either procedure on long-term health-related quality of life (HRQoL) and health status.

Methods

Patients enrolled in a multicentre randomized clinical trial (DREAM trial; 2000–2003) in Europe of open repair versus endovascular repair (EVAR) of abdominal aortic aneurysm were asked to complete questionnaires on health status and HRQoL. HRQoL scores were assessed at baseline and at 13 time points thereafter, using generic tools, the Medical Outcomes Study 36-Item Short-Form Health Survey (SF-36®) and EuroQol 5D (EQ-5D™). Physical (PCS) and mental component summary scores were also calculated. Follow-up was 5 years.

Results

Some 332 of 351 patients enrolled in the trial returned questionnaires. More than 70 per cent of questionnaires were returned at each time point. Both surgical interventions had a short-term negative effect on HRQoL and health status. This was less severe in the EVAR group than in the open repair group. In the longer term the physical domains of SF-36® favoured open repair: mean difference in PCS score between open repair and EVAR −1·98 (95 per cent c.i. −3·56 to −0·41). EQ-5D™ descriptive and EQ-5D™ visual analogue scale scores for open repair were also superior to those for EVAR after the initial 6-week interval: mean difference −0·06 (−0·10 to −0·02) and −4·09 (−6·91 to −1·27) respectively.

Conclusion

In this study EVAR appeared to be associated with less severe disruption to HRQoL and health status in the short term. However, during longer-term follow-up to 5 years, patients receiving open repair appeared to have improved quality of life and health status.

Chemotherapy effectiveness and mortality prediction in surgically treated osteosarcoma dogs: A validation study

Canine osteosarcoma is the most common bone cancer, and an important cause of mortality and morbidity, in large purebred dogs. Previously we constructed two multivariable models to predict a dog's 5-month or 1-year mortality risk after surgical treatment for osteosarcoma. According to the 5-month model, dogs with a relatively low risk of 5-month mortality benefited most from additional chemotherapy treatment. In the present study, we externally validated these results using an independent cohort study of 794 dogs. External performance of our prediction models showed some disagreement between observed and predicted risk, mean difference: -0.11 (95% confidence interval [95% CI]-0.29; 0.08) for 5-month risk and 0.25 (95%CI 0.10; 0.40) for 1-year mortality risk. After updating the intercept, agreement improved: -0.0004 (95%CI-0.16; 0.16) and -0.002 (95%CI-0.15; 0.15). The chemotherapy by predicted mortality risk interaction (P-value=0.01) showed that the chemotherapy compared to no chemotherapy effectiveness was modified by 5-month mortality risk: dogs with a relatively lower risk of mortality benefited most from additional chemotherapy. Chemotherapy effectiveness on 1-year mortality was not significantly modified by predicted risk (P-value=0.28). In conclusion, this external validation study confirmed that our multivariable risk prediction models can predict a patient's mortality risk and that dogs with a relatively lower risk of 5-month mortality seem to benefit most from chemotherapy.

Breast-feeding and health consequences in early childhood: is there an impact of time-dependent confounding?

BACKGROUND

Estimated effects of breast-feeding on childhood health vary between studies, possibly due to confounding by baseline maternal and child characteristics. Possible time-dependent confounding has received little consideration. Our aim was to evaluate the impact of such confounding.

METHODS

We estimated the relationship between cumulative exclusive breast-feeding up to 6 months and wheezing, rash and body mass index (BMI) at 12 months [in the Whistler cohort (n = 494) and PROBIT (n = 11,463)], and wheezing, rash, asthma, hay fever, eczema, allergy and BMI at age 6.5 years (PROBIT). We adjusted for time-dependent confounding by weight, length, rash, respiratory illness and day care attendance using marginal structural models (MSMs).

RESULTS

Weight and day care attendance appeared potential time-dependent confounders, since these predicted breast-feeding status and were influenced by previous breast-feeding. However, adjustment for time-dependent confounders did not markedly change the estimated associations. For example, in PROBIT the adjusted increase in BMI at 12 months per 1-month increase in exclusive breast-feeding was 0.04 (95% CI -0.09 to 0.01) using logistic regression and -0.06 (95% CI -0.11 to -0.01) using MSM. In Whistler, these estimates were each -0.05 (95% CI -0.10 to 0.00).

CONCLUSIONS

In two cohort studies, there was little evidence of time-dependent confounding by weight, length, rash, respiratory illness or day care attendance of the effects of breast-feeding on early childhood health.

Effectiveness of progestogens to improve perinatal outcome in twin pregnancies: an individual participant data meta-analysis

BACKGROUND

In twin pregnancies, the rates of adverse perinatal outcome and subsequent long-term morbidity are substantial, and mainly result from preterm birth (PTB).

OBJECTIVES

To assess the effectiveness of progestogen treatment in the prevention of neonatal morbidity or PTB in twin pregnancies using individual participant data meta-analysis (IPDMA).

SEARCH STRATEGY

We searched international scientific databases, trial registration websites, and references of identified articles.

SELECTION CRITERIA

Randomised clinical trials (RCTs) of 17-hydroxyprogesterone caproate (17Pc) or vaginally administered natural progesterone, compared with placebo or no treatment.

DATA COLLECTION AND ANALYSIS

Investigators of identified RCTs were asked to share their IPD. The primary outcome was a composite of perinatal mortality and severe neonatal morbidity. Prespecified subgroup analyses were performed for chorionicity, cervical length, and prior spontaneous PTB.

MAIN RESULTS

Thirteen trials included 3768 women and their 7536 babies. Neither 17Pc nor vaginal progesterone reduced the incidence of adverse perinatal outcome (17Pc relative risk, RR 1.1; 95% confidence interval, 95% CI 0.97-1.4, vaginal progesterone RR 0.97; 95% CI 0.77-1.2). In a subgroup of women with a cervical length of ≤25 mm, vaginal progesterone reduced adverse perinatal outcome when cervical length was measured at randomisation (15/56 versus 22/60; RR 0.57; 95% CI 0.47-0.70) or before 24 weeks of gestation (14/52 versus 21/56; RR 0.56; 95% CI 0.42-0.75).

AUTHOR'S CONCLUSIONS

In unselected women with an uncomplicated twin gestation, treatment with progestogens (intramuscular 17Pc or vaginal natural progesterone) does not improve perinatal outcome. Vaginal progesterone may be effective in the reduction of adverse perinatal outcome in women with a cervical length of ≤25 mm; however, further research is warranted to confirm this finding.

The Dutch hospital standardised mortality ratio (HSMR) method and cardiac surgery: benchmarking in a national cohort using hospital administration data versus a clinical database

Objective

To compare the accuracy of data from hospital administration databases and a national clinical cardiac surgery database and to compare the performance of the Dutch hospital standardised mortality ratio (HSMR) method and the logistic European System for Cardiac Operative Risk Evaluation, for the purpose of benchmarking of mortality across hospitals.

Methods

Information on all patients undergoing cardiac surgery between 1 January 2007 and 31 December 2010 in 10 centres was extracted from The Netherlands Association for Cardio-Thoracic Surgery database and the Hospital Discharge Registry. The number of cardiac surgery interventions was compared between both databases. The European System for Cardiac Operative Risk Evaluation and hospital standardised mortality ratio models were updated in the study population and compared using the C-statistic, calibration plots and the Brier-score.

Results

The number of cardiac surgery interventions performed could not be assessed using the administrative database as the intervention code was incorrect in 1.4–26.3%, depending on the type of intervention. In 7.3% no intervention code was registered. The updated administrative model was inferior to the updated clinical model with respect to discrimination (c-statistic of 0.77 vs 0.85, p<0.001) and calibration (Brier Score of 2.8% vs 2.6%, p<0.001, maximum score 3.0%). Two average performing hospitals according to the clinical model became outliers when benchmarking was performed using the administrative model.

Conclusions

In cardiac surgery, administrative data are less suitable than clinical data for the purpose of benchmarking. The use of either administrative or clinical risk-adjustment models can affect the outlier status of hospitals. Risk-adjustment models including procedure-specific clinical risk factors are recommended.

A clinical prediction model to assess the risk of operative delivery

OBJECTIVE

To predict instrumental vaginal delivery or caesarean section for suspected fetal distress or failure to progress.

DESIGN

Secondary analysis of a randomised trial.

SETTING

Three academic and six non-academic teaching hospitals in the Netherlands.

POPULATION

5667 labouring women with a singleton term pregnancy in cephalic presentation.

METHODS

We developed multinomial prediction models to assess the risk of operative delivery using both antepartum (model 1) and antepartum plus intrapartum characteristics (model 2). The models were validated by bootstrapping techniques and adjusted for overfitting. Predictive performance was assessed by calibration and discrimination (area under the receiver operating characteristic), and easy-to-use nomograms were developed.

MAIN OUTCOME MEASURES

Incidence of instrumental vaginal delivery or caesarean section for fetal distress or failure to progress with respect to a spontaneous vaginal delivery (reference).

RESULTS

  375 (6.6%) and 212 (3.6%) women had an instrumental vaginal delivery or caesarean section due to fetal distress, and 433 (7.6%) and 571 (10.1%) due to failure to progress, respectively. Predictors were age, parity, previous caesarean section, diabetes, gestational age, gender, estimated birthweight (model 1) and induction of labour, oxytocin augmentation, intrapartum fever, prolonged rupture of membranes, meconium stained amniotic fluid, epidural anaesthesia, and use of ST-analysis (model 2). Both models showed excellent calibration and the receiver operating characteristics areas were 0.70-0.78 and 0.73-0.81, respectively.

CONCLUSION

  In Dutch women with a singleton term pregnancy in cephalic presentation, antepartum and intrapartum characteristics can assist in the prediction of the need for an instrumental vaginal delivery or caesarean section for fetal distress or failure to progress.

How to deal with measures of association: a short guide for the clinician

When reading medical literature as a clinician, many different measures of association are presented. To judge whether results of studies can be applied to clinical practice, it is essential to understand and to be able to interpret the measure of association reported in the article. In this paper, we will present how to deal with the most commonly used measures of association including the risk and rate difference, number needed to treat, risk and rate ratio, hazard ratio and odds ratio. By means of examples, we will discuss the different measures of association for the three main study designs used in clinical research: randomized controlled trial, observational cohort study and case-control study.

Association between cannabis and psychiatric hospitalization

OBJECTIVE

To investigate the relationship between cannabis use and mental health.

METHOD

A cross-sectional analysis in a sample of 17 698 individuals with a mean age of 22 years (SD: 4.2). Participants provided information on the amount and initial age of cannabis use and history of psychiatric hospitalizations through a web-based questionnaire. To quantify Δ(9) -tetrahydrocannabinol exposure, we operationalized cannabis use as the amount of money spent on cannabis per week over the last month. The odds ratio of having a history of psychiatric hospitalizations was the primary outcome measure.

RESULTS

We found a dose-response relationship between the amount of cannabis use and the odds for psychiatric hospitalization. Adjusted odds ratios for hospitalization increased with the amount of cannabis consumed from 1.6 (95% CI: 1.1-2.3) in incidental users to 6.2 (95% CI: 4.3-8.9) in heavy users (>€25/week). Our data suggested that concomitant drug use was an intermediate factor. Exposure to cannabis before the age of 12 years was found to carry a 4.8 (95% CI: 2.9-7.8) times increased odds for past psychiatric hospitalizations.

CONCLUSION

We conclude that early and heavy uses of cannabis are each and independently associated with poor mental health in its users.

Quantifying the potential role of unmeasured confounders: the example of influenza vaccination

BACKGROUND

The validity of non-randomized studies using healthcare databases is often challenged because they lack information on potentially important confounders, such as functional health status and socioeconomic status. In a study quantifying the effects of influenza vaccination among community-dwelling elderly we assessed whether additional information on not routinely available covariates was indeed associated with exposure to influenza vaccination and could, therefore, have led to residual confounding in healthcare databases.

METHODS

We randomly selected 500 persons aged 65 years and older from the computerized Utrecht General Practitioner database. Information on exposure status and on demographics, co-morbidity status, prior healthcare use and medication use was extracted from the database. A questionnaire was used to obtain additional information on not routinely available risk factors [e.g. functional health status (SF-20), smoking status and alcohol consumption]. Missing data from the questionnaire was imputed and multivariable logistic regression analysis was applied to quantify the influence of covariates on the prediction of exposure to influenza vaccination. Within an existing dataset the potential impact of functional health status on the relation between influenza vaccination and mortality was simulated.

RESULTS

We obtained questionnaire data from 365 of 500 (73%) subjects. The model including routinely available data from the database appeared accurate in predicting exposure to influenza vaccination (c-statistic 0.86, 95% CI: 0.82-0.89). Functional health status was the only additional characteristic measured with the questionnaire that was not similar in vaccinated and unvaccinated subjects. However, extending the multivariable regression model with functional health status did not significantly improve the prediction of exposure to influenza vaccination, nor did it affect the relation between influenza vaccination and mortality.

CONCLUSION

The potential for unmeasured confounding on the association between influenza vaccination and health outcomes as quantified in healthcare databases seems small for non-randomized intervention studies within extensive and reliable databases.