Generalizability of predictive models for Clostridioides difficile infection, severity and recurrence at an urban safety-net hospital

INTRODUCTION

Predictive models for Clostridioides difficile infection can identify high-risk patients and aid clinicians in preventing infection. Issues of generalizability regarding current predictive models have been acknowledged but, to the authors' knowledge, have never been quantified.

METHODS

C. difficile infection, severity and recurrence predictive models were created using multi-variate logistic regression through case-control sampling from an urban safety-net hospital. Models were validated using five-fold cross-validation, and inverse probability weights (IPW) based on two different catchment area definitions were used to improve external validity. Akaike Information Criterion (AIC), area under the receiver operating characteristic curve (AUROC), and sensitivity and specificity with bootstrapped confidence intervals (CI) were used to assess and compare model fit and performance.

RESULTS

Changes in performance before and after weighting were small across all models, although differences were more apparent after weighting the recurrence model (AUROC values of 0.78, 0.76 and 0.71 for the unweighted and two weighted models, respectively). Overall, the infection model performed the best (AUROC 0.82, 95% CI 0.78-0.85), followed by the recurrence model (AUROC 0.78, 95% CI 0.69-0.86) and then the severity model (AUROC 0.70, 95% CI 0.63-0.78).

CONCLUSIONS

The performance of the models after weighting did not change drastically, suggesting that the models predicting C. difficile infection, severity and recurrence may not be impacted by patient selection factors. However, other researchers may wish to consider addressing these catchment forces using IPW.

Initial evidence of delay discounting's predictive utility for alcohol self-administration in ecologically valid contexts among young adults who drink heavily

BACKGROUND

While delay discounting is robustly associated with alcohol use disorder, whether discounting predicts real-time alcohol use behaviors is unclear. Existing support comes from laboratory studies using intravenous alcohol self-administration methods, thus limiting ecological validity and generalizability. The present study evaluated whether delay discounting predicted real-time alcohol use in naturalistic settings with and without probabilistic negative consequences for consuming larger amounts of alcohol.

METHODS

This secondary analysis utilized data from three laboratory alcohol self-administration studies with young adults who engaged in frequent heavy drinking (N=206, 45% female). Participants completed a delay discounting measure before an alcohol self-administration session in an actual or simulated bar with (n=187) or without (n=19) probabilistic negative consequences (compensation loss) tied to performance on cognitive and psychomotor tasks after alcohol self-administration. Bootstrapped (unstandardized) coefficient estimates and 95% confidence intervals were utilized due to the sample size discrepancy.

RESULTS

Multiple regressions revealed that delay discounting did not significantly predict estimated blood alcohol concentration (eBAC) or number of drinks consumed when procedures included probabilistic negative consequences. Among participants who completed procedures without probabilistic negative consequences, delay discounting was positively associated with peak eBAC.

CONCLUSION

Counter to hypotheses, steeper delay discounting did not predict real-time alcohol use in contexts with probabilistic negative consequences, whereas preliminary evidence suggests that delay discounting predicts real-time alcohol use behaviors in contexts without probabilistic negative consequences. The specific discounting task may have impacted study findings, thus future research should consider how the sign (gain vs. loss), outcome certainty, and delay relate to alcohol consumption.

Comparing clinical trial population representativeness to real-world users of 17 biologics approved for immune-mediated inflammatory diseases: An external validity analysis of 66,639 biologic users from the Italian VALORE project

To date, no population-based studies have specifically explored the external validity of pivotal randomized clinical trials (RCTs) of biologics simultaneously for a broad spectrum of immuno-mediated inflammatory diseases (IMIDs). The aims of this study were, firstly, to compare the patients' characteristics and median treatment duration of biologics approved for IMIDs between RCTs' and real-world setting (RW); secondly, to assess the extent of biologic users treated for IMIDs in the real-world setting that would not have been eligible for inclusion into pivotal RCT for each indication of use. Using the Italian VALORE distributed database (66,639 incident biologic users), adult patients with IMIDs treated with biologics in the Italian real-world setting were substantially older (mean age ± SD: 50 ± 15 years) compared to those enrolled in pivotal RCTs (45 ± 15 years). In the real-world setting, certolizumab pegol was more commonly used by adult women with psoriasis/ankylosing spondylitis (F/M ratio: 1.8-1.9) compared to RCTs (F/M ratio: 0.5-0.6). The median treatment duration (weeks) of incident biologic users in RW was significantly higher than the duration of pivotal RCTs in almost all indications for use and most biologics (4-100 vs. 6-167). Furthermore, almost half (46.4%) of biologic users from RW settings would have been ineligible for inclusion in the respective indication-specific pivotal RCTs. The main reasons were: advanced age, recent history of cancer and presence of other concomitant IMIDs. These findings suggest that post-marketing surveillance of biologics should be prioritized for those patients.

The tricot approach: an agile framework for decentralized on-farm testing supported by citizen science. A retrospective

Matching crop varieties to their target use context and user preferences is a challenge faced by many plant breeding programs serving smallholder agriculture. Numerous participatory approaches proposed by CGIAR and other research teams over the last four decades have attempted to capture farmers' priorities/preferences and crop variety field performance in representative growing environments through experimental trials with higher external validity. Yet none have overcome the challenges of scalability, data validity and reliability, and difficulties in capturing socio-economic and environmental heterogeneity. Building on the strengths of these attempts, we developed a new data-generation approach, called triadic comparison of technology options (tricot). Tricot is a decentralized experimental approach supported by crowdsourced citizen science. In this article, we review the development, validation, and evolution of the tricot approach, through our own research results and reviewing the literature in which tricot approaches have been successfully applied. The first results indicated that tricot-aggregated farmer-led assessments contained information with adequate validity and that reliability could be achieved with a large sample. Costs were lower than current participatory approaches. Scaling the tricot approach into a large on-farm testing network successfully registered specific climatic effects of crop variety performance in representative growing environments. Tricot's recent application in plant breeding networks in relation to decision-making has (i) advanced plant breeding lines recognizing socio-economic heterogeneity, and (ii) identified consumers' preferences and market demands, generating alternative breeding design priorities. We review lessons learned from tricot applications that have enabled a large scaling effort, which should lead to stronger decision-making in crop improvement and increased use of improved varieties in smallholder agriculture.

A new methodology to extrapolate disease freedom to an area using surveillance results from selected aquatic populations

According to Chapter 1.4 of the World Organisation for Animal Health (WOAH) Aquatic Animal Health Code, an entire country or zone can be classified as free of a disease only if there is compelling evidence that all susceptible populations within the country or zone are free. However, the methods for achieving freedom are not prescribed in the WOAH standards and guidelines. Within this context, this paper describes a novel methodology to determine if surveillance results can be extrapolated from a study population to a target population. A framework of six criteria was developed to standardize a method for extrapolating surveillance results to other susceptible populations that have not been sampled. Criteria 1 assesses the internal validity for the freedom claim on the source population. Criteria 2 assesses which other susceptible populations have a non-negligible probability of exposure. Criteria 3 assesses whether the risk of infection upon exposure of the source population is the same or greater than each of the other susceptible populations. Finally, Criteria 4, 5 and 6 assess if the other susceptible populations would transmit the infection to the source population or if they have the same exposure pathways as the source population. We illustrate the use of this novel methodology using two hypothetical case scenarios. The presented methodology has the advantage of being applicable either retrospectively or prospectively. When applied retrospectively, it can be used to assess if the surveillance results of the source population can be extrapolated to the target population. When applied prospectively it can be used to design a more efficient surveillance system by selecting source populations from which it is easier to extrapolate surveillance results to the rest of the target population. Conclusions drawn using this methodology depend on the validity of the assumptions made when working through the methodology. We therefore recommend cautious application of the criteria and thorough review of all assumptions.

The quality and pattern of rehabilitation interventions prescribed for post-COVID-19 infection patients: A systematic review and meta-analysis

Patients with prior COVID-19 infection may present with lasting multisystem symptoms that require intervention and includes exercise rehabilitation. The aim of this systematic review was to investigate the quality of articles, with emphasis on exercise rehabilitation, and conduct a meta-analysis on experimental and observational rehabilitation intervention studies on cardiorespiratory fitness and pulmonary function in post-COVID-19 infection patients. This systematic review used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 guidelines for evaluating rehabilitation interventions in clinical practice. A methodical search of cohort and experimental studies occurred from January 2019 up to March 2023. Thirty-two studies were included for complete analysis. The quality of the eligible studies for complete review was fair overall. The studies did not provide a detailed account of key descriptors of exercise such as volume, progression, motivational strategies, adherence and replication. There was a significant difference in the 6-minute walk test (Mean difference (MD) = 51.69 m; confidence intervals (CIs) = 36.99 to 66.38; p < 0.001, level of heterogeneity (I2) = 0), percent of predicted forced expiratory volume in one second (MD = 7.95%; CIs = 3.46 to 12.44; p = 0.0005, I2 = 65%) and percent of predicted forced vital capacity (MD = 3.49%; 95% CI = 1.25 to 5.73; p = 0.002; I2 = 37%) between the experiment and control groups. The current evidence suggests that rehabilitation interventions significantly improve cardiorespiratory fitness and pulmonary function in post-COVID-19 infection patients; however, there is a need for conceptualising high-quality and long-term rehabilitation interventions, especially exercise interventions.

Racial and Ethnic Representation in Preventive Intervention Research: a Methodological Study

Individuals who are Asian or Asian American, Black or African American, Native American or American Indian or Alaska Native, Native Hawaiian or Pacific Islander, and Hispanic or Latino (i.e., presently considered racial ethnic minoritized groups in the USA) lacked equal access to resources for mitigating risk during COVID-19, which highlighted public health disparities and exacerbated inequities rooted in structural racism that have contributed to many injustices, such as failing public school systems and unsafe neighborhoods. Minoritized groups are also vulnerable to climate change wherein the most severe harms disproportionately fall upon underserved communities. While systemic changes are needed to address these pervasive syndemic conditions, immediate efforts involve examining strategies to promote equitable health and well-being-which served as the impetus for this study. We conducted a descriptive analysis on the prevalence of culturally tailored interventions and reporting of sample characteristics among 885 programs with evaluations published from 2010 to 2021 and recorded in the Blueprints for Healthy Youth Development registry. Inferential analyses also examined (1) reporting time trends and (2) the relationship between study quality (i.e., strong methods, beneficial effects) and culturally tailored programs and racial ethnic enrollment. Two percent of programs were developed for Black or African American youth, and 4% targeted Hispanic or Latino populations. For the 77% of studies that reported race, most enrollees were White (35%) followed by Black or African American (28%), and 31% collapsed across race or categorized race with ethnicity. In the 64% of studies that reported ethnicity, 32% of enrollees were Hispanic or Latino. Reporting has not improved, and there was no relationship between high-quality studies and programs developed for racial ethnic youth, or samples with high proportions of racial ethnic enrollees. Research gaps on racial ethnic groups call for clear reporting and better representation to reduce disparities and improve the utility of interventions.

Let's play pretend: Towards effective modelling in experimental psycho(patho)logy

BACKGROUND AND OBJECTIVES

Models are omnipresent in science. We introduce a novel framework to achieve more effective modelling practices in general psychology and experimental psychopathology.

METHOD

We analyze three types of models that are common in psychology: laboratory models, computational models, and cognitive models. We then evaluate two common ways to assess the translational value of models (phenomenological similarity and deep similarity) as well as an arguably underappreciated way (functional similarity). Functional similarity is based on an assessment of whether variables (e.g., the administration of a pharmacological substance) have a similar effect (a) in the model (e.g., in a fear conditioning procedure) and (b) on the real-life target phenomenon (e.g., on real-life anxiety complaints).

CONCLUSIONS

We argue that the assessment of functional similarity is a powerful tool to assess the translational value of models in the field of experimental psychopathology and beyond.

Revisiting the link between the sustained attention to response task (SART) and daily-life cognitive failures

In this study, we examined the relationship betweenerrors of commissionon theSustained Attention to Response Task(SART)andscores on the Cognitive Failures Questionnaire (CFQ). The goal was to assess theecological validity of the SARTin a sample of people scoring high on fatigue complaints.SART errors of commission were positively associated with CFQ scores and this finding remained after controlling for fatigue level, age, and SART reaction times.Thus, our results generally supported the ecological validity of the SART. However, when examining subsamples separately, we found the association between SART and CFQ only in our subsample of employees, not in our subsample of university students. The three subscales of the CFQ showed the same pattern of findings. Our results imply that, when using the SART to draw conclusions about everyday life, it is crucial to consider the characteristics of one's sample and control for relevant confounding variables.

Are we enrolling the right patients? A scoping review of external validity and generalizability of clinical trials in bloodstream infections

BACKGROUND

Having a representative population in randomized clinical trials (RCTs) improves external validity and generalizability of trial results. There are limited data examining differences between RCT-enrolled and real-world populations in bloodstream infections (BSI).

OBJECTIVES

We conducted a scoping review aiming to review studies assessing generalizability of BSI RCT populations, to identify sub-groups that have been systematically under-represented and to explore approaches to improve external validity of future RCTs.

SOURCES

MEDLINE, Embase, and Cochrane Library databases were searched for terms related to external validity or generalizability, BSI, and clinical trials in papers published up to 1 August 2023. Studies comparing enrolled versus nonenrolled patients, or papers discussing external validity or generalizability in the context of BSI RCTs were included.

CONTENT

Sixteen papers were included in the final review. Five compared RCT-enrolled and nonenrolled participants from the same source population. There were significant differences between the two groups in all studies, with nonenrolled patients having a greater comorbidity burden and consistently worse outcomes including mortality. We identified several barriers to improving generalizability of RCT populations and outlined potential approaches to reduce these barriers, such as alternative/simplified consent processes, streamlining eligibility criteria and follow-up procedures, quota-based sampling techniques, and ensuring diversity in site and study team selection.

IMPLICATIONS

Study cohorts in BSI RCTs are not representative of the general BSI patient population. As we increasingly adopt large pragmatic trials in infectious diseases, it is important to recognize the importance of maximizing generalizability to ensure that our research findings are of direct relevance to our patients.

Underreporting contextual factors preclude the applicability appraisal in primary care randomized controlled trials

OBJECTIVES

To assess applicability reporting in randomized controlled trials (RCTs) conducted in primary care (PC).

STUDY DESIGN AND SETTING

We used a random sample of PC RCTs published between 2000 and 2020 to assess applicability. We extracted data related to setting, population, intervention (including implementation), comparator, outcomes, and context. Based on data availability, we assessed whether the five predefined applicability questions were adequately addressed by each PC RCT.

RESULTS

Adequately described elements that were reported frequently (>50%) included the responsible organization for intervention provision (97, 93.3%), study population characteristics (94, 90.4%), intervention implementation including monitoring and evaluation (92, 88.5%), intervention components (89, 85.6%), time frame (82, 78.8%), baseline prevalence (58, 55.8%), and the type of setting and location (53, 51%). Elements that were often underreported included contextual factors, that is, evidence of differential effects across sociodemographic or other groupings (2, 1.9%), intervention components tailored for specific settings (7, 6.7%), health system structure (32, 30.8%), factors affecting implementation (40, 38.5%) and organization structure (50, 48.1%). The proportion of trials that adequately addressed each applicability question ranged between 1% and 20.2%, while none RCT could address all of them.

CONCLUSION

Underreporting contextual factors jeopardize the appraisal of applicability in PC RCTs.

Disparities between randomized clinical trial participants and sepsis patients in real-world

BACKGROUND

The external validity or "generalizability" of randomized controlled trials (RCTs) often needs be considered when making treatment decisions. We evaluate whether participants in large multicenter RCTs investigating sepsis were similar in age, disease severity, comorbidities, and mortality to the general population of sepsis patients.

METHODS

Using MEDLINE, PubMed, and the Cochrane Central Register of Controlled Trials, RCTs that enrolled 100 or more adult sepsis patients from two or more sites published from 01 January 2000 to 04 August 2019 were identified. The weighted mean age of trial participants was calculated as the main variable, and it was compared with the mean ages of the overall populations derived from the MIMIC database and the EICU database. Two researchers independently screened all abstracts and performed data extraction, then aggregated the data using a random effects model. Multiple linear regression was used to determine whether any factors were significantly associated with age disparities.

RESULTS

The mean age of the 60,577 participants in the 94 trials included in the analysis was significantly lower than those of the patients in the MIMIC and EICU databases (weighted mean age 62.28 years vs. 64.47 years for MIMIC and 65.20 years for EICU; both p < 0.001). Trial participants were less likely to have known comorbidities such as diabetes (13.96% vs. 30.64% for MIMIC and 35.75% for EICU; both p < 0.001). The weighted mortality rate in trial participants was higher than that in patients in the MIMIC and EICU databases (29.33% vs. 20.72% for MIMIC and 17.53% for EICU; both p < 0.001). Differences in age, severity score, and comorbidities remained statistically significant in sensitivity analyses. Multivariable regression suggested that commercially supported trials were more likely to include patients with higher severity scores (p = 0.002), but after adjustment for study region and sepsis diagnosis inclusion in such trials was not significantly associated with age.

CONCLUSIONS

On average, trial participants were younger than the general sepsis patient population. Commercial support influenced patient selection. Efforts to understand and address the above-described patient disparities are necessary to improve the generalizability of RCT results.

TRIAL REGISTRATION

PROSPERO CRD42019145692.

GRADE concept 4: rating the certainty of evidence when study interventions or comparators differ from PICO targets

OBJECTIVES

This Grading of Recommendations Assessment, Development and Evaluation (GRADE) concept article offers systematic reviewers, guideline authors, and other users of evidence assistance in addressing randomized trial situations in which interventions or comparators differ from those in the target people, interventions, comparators, and outcomes. To clarify what GRADE considers under indirectness of interventions and comparators, we focus on a particular example: when comparator arm participants receive some or all aspects of the intervention management strategy (treatment switching).

STUDY DESIGN AND SETTING

An interdisciplinary panel of the GRADE working group members developed this concept article through an iterative review of examples in multiple teleconferences, small group sessions, and e-mail correspondence. After presentation at a GRADE working group meeting in November 2022, attendees approved the final concept paper, which we support with examples from systematic reviews and individual trials.

RESULTS

In the presence of safeguards against risk of bias, trials provide unbiased estimates of the effect of an intervention on the people as enrolled, the interventions as implemented, the comparators as implemented, and the outcomes as measured. Within the GRADE framework, differences in the people, interventions, comparators, and outcomes elements between the review or guideline recommendation targets and the trials as implemented constitute issues of indirectness. The intervention or comparator group management strategy as implemented, when it differs from the target comparator, constitutes one potential source of indirectness: Indirectness of interventions and comparators-comparator group receipt of the intervention constitutes a specific subcategory of said indirectness. The proportion of comparator arm participants that received the intervention and the apparent magnitude of effect bear on whether one should rate down, and if one does, to what extent.

CONCLUSION

Treatment switching and other differences between review or guideline recommendation target interventions and comparators vs. interventions and comparators as implemented in otherwise relevant trials are best considered issues of indirectness.

External validation of postoperative nausea and vomiting risk scores in patients with liver cancer: A single-centre prospective cohort study

OBJECTIVES

This study aimed to test the external validity of postoperative nausea and vomiting (PONV) risk assessment tools in patients undergoing hepatectomy, and to guide healthcare professionals' assessment of postoperative patients.

BACKGROUND

The identification of PONV risk is particularly important in the context of prevention. However, the predictive performance of the current PONV risk scores has not been confirmed in patients with liver cancer, and its applicability is still unknown. These uncertainties pose difficulties in performing routine risk assessment of PONV for patients with liver cancer in a clinical practice setting.

METHODS

Patients diagnosed with liver cancer and undergoing hepatectomy were prospectively consecutively recruited. All enrolled patients received PONV assessments and PONV risk assessments via the Apfel risk score and the Koivuranta risk score. Receiver operating characteristic curves (ROC curves) and calibration curves were used to assess the external validity. This study was reported according to the TRIPOD Checklist.

RESULTS

Among 214 PONV-assessed patients, 114 patients (53.3%) developed PONV. For the Apfel simplified risk score, the ROC area was 0.612 (95% confidence interval [CI]: 0.543-0.678) in the validation dataset, which demonstrated imperfect discrimination; the calibration curve showed poor calibration with a slope of 0.49. For the Koivuranta score, the ROC area was 0.628 (CI: 0.559-0.693) in the validation dataset, which showed limited discrimination; the calibration curve indicated an unsatisfactory calibration with a slope of 0.71.

CONCLUSIONS

The Apfel risk score and the Koivuranta risk score were not well validated in our study and disease-specific risk factors should be taken into account when updating or developing PONV risk stratification instruments.

How Effect Measure Choice Influences Minimally Sufficient Adjustment Sets for External Validity

Epidemiologic researchers generalizing or transporting effect estimates from a study to a target must account for effect measure modifiers (EMMs) on the scale of interest. Little attention is paid to how the EMMs required may vary depending on the mathematical nuances of each effect measure, however. We defined two types of EMM: marginal EMM, where the effect on the scale of interest differs across levels of a variable; and conditional EMM, where the effect differs conditional on other variables associated with the outcome. These types define three classes of variables: Class 1 (conditional EMM), Class 2 (marginal, but not conditional, EMM), or Class 3 (neither marginal nor conditional EMM). Class 1 variables are necessary to achieve a valid estimate of the RD in a target, while a RR requires Class 1 and Class 2 and an OR requires Class 1, Class 2, and Class 3 (i.e., all variables associated with the outcome). This does not mean that fewer variables are required for an externally valid RD (because variables may not modify effects on all scale) but does suggest researchers should consider the scale of the effect measure when identifying EMM necessary for an externally valid treatment effect estimate.

REPRESENT: REPresentativeness of RESearch data obtained through the 'General Informed ConsENT'

BACKGROUND

We assessed potential consent bias in a cohort of > 40,000 adult patients asked by mail after hospitalization to consent to the use of past, present and future clinical and biological data in an ongoing 'general consent' program at a large tertiary hospital in Switzerland.

METHODS

In this retrospective cohort study, all adult patients hospitalized between April 2019 and March 2020 were invited to participate to the general consent program. Demographic and clinical characteristics were extracted from patients' electronic health records (EHR). Data of those who provided written consent (signatories) and non-responders were compared and analyzed with R studio.

RESULTS

Of 44,819 patients approached, 10,299 (23%) signed the form. Signatories were older (median age 54 [IQR 38-72] vs. 44 years [IQR 32-60], p < .0001), more comorbid (2614/10,299 [25.4%] vs. 4912/28,676 [17.1%] with Charlson comorbidity index ≤ 4, p < .0001), and more often of Swiss nationality (6592/10,299 [64%] vs. 13,813/28,676 [48.2%], p < .0001).

CONCLUSIONS

Our results suggest that actively seeking consent creates a bias and compromises the external validity of data obtained via 'general consent' programs. Other options, such as opt-out consent procedures, should be further assessed.

External Validity of a Randomized Controlled Trial on Duration of Antibiotics for the Treatment of Gram-Negative Bacteremia

INTRODUCTION

Reports regarding the external validity of randomized controlled trials (RCTs) are scarce. We aimed to assess the population external validity of an investigator-initiated RCT on the duration of antibiotics for the treatment of Gram-negative bacteremia by comparing patients included in the RCT to patients that were not included in the trial.

METHODS

Hospitalized patients with Gram-negative bacteremia were recruited into an RCT and randomized to receive 7 or 14 days of covering antibiotic therapy in Israel and Italy from 2013 to 2017. In a concomitant observational study, RCT participants were compared with patients who fulfilled the inclusion criteria but were not included in the trial due to participation in other trials, discharge before approached by researchers, refusal to participate, or unwillingness of the treating physician to allow participants' recruitment.

RESULTS

Six hundred and four RCT patients were compared with 613 nonincluded patients. Almost 50% of nonincluded patients (288/613) were dependent on others for activities of daily living at baseline compared to 37.7% of RCT participants (228/604). Dementia was nearly 2-fold more frequent in nonincluded patients than those included (5.9% [36/613] versus 3.6% [22/604], p = 0.07). Patients who were not included in the RCT were more likely to acquire their infection in the hospital (53.3% [327/613] versus 29.1% [176/604], p < 0.001). The primary composite outcome of mortality, clinical failure, readmissions, or extended hospitalization at 90 days occurred in 353 of 613 nonincluded patients (57.6%) compared to 299 of 604 RCT participants (49.6%), p = 0.005. However, on multivariate analysis noninclusion in the RCT was not an independent risk factor for clinical failure and mortality.

CONCLUSIONS

RCTs, even with broad eligibility criteria, do not represent the whole spectrum of patients and leave out a population with more severe illness for whom the evidence is lacking.

[Systematic reviews in five steps: V. Interpreting the findings]

The last step in a systematic review is the interpretation of the findings. The important findings need to be explicitly identified. A level of strength of evidence should be assigned to support each key finding, based on factors such as study design, methodological quality and risk of publication bias. Variations in the magnitude of associations observed also need to be explored. The aim of this analysis is to determine in which clinical groups the intervention is more or less effective, the impact of exposure is greater or lesser, or a diagnostic test is more useful. At this stage, for better interpretation of the findings, the magnitude of the association can be estimated either globally or stratified according to the characteristics of the participants. All this is helpful in formulating recommendations for clinical practice and policy.

Lack of sex-related analysis and reporting in Cochrane Reviews: a cross-sectional study

BACKGROUND

Sex-specific analysis and reporting may allow a better understanding of intervention effects and can support the decision-making process. Well-conducted systematic reviews (SRs), like those carried out by the Cochrane Collaboration, provide clinical responses transparently and stress gaps of knowledge. This study aimed to describe the extent to which sex is analysed and reported in a cross-section of Cochrane SRs of interventions, and assess the association with the gender of main authorships.

METHODS

We searched SRs published during 2018 within the Cochrane Database of Systematic Reviews. An investigator appraised the sex-related analysis and reporting across sections of SRs and collected data on gender and country of affiliation of the review first and last authors, and a second checked for accuracy. We conducted descriptive statistics and bivariate logistic regression to explore the association between the gender of the authors and sex-related analysis and reporting.

RESULTS

Six hundred and ten Cochrane SRs were identified. After removing those that met no eligibility criteria, 516 reviews of interventions were included. Fifty-six reviews included sex-related reporting in the abstract, 90 considered sex in their design, 380 provided sex-disaggregated descriptive data, 142 reported main outcomes or performed subgroup analyses by sex, and 76 discussed the potential impact of sex or the lack of such on the interpretations of findings. Women represented 53.1 and 42.2% of first and last authorships, respectively. Women authors (in first and last position) had a higher possibility to report sex in at least one of the review sections (OR 2.05; CI 95% 1.12-3.75, P=0.020) than having none.

CONCLUSIONS

Sex consideration amongst Cochrane SRs was frequently missing. Structured guidance to sex-related analysis and reporting is needed to enhance the external validity of findings. Likewise, including gender diversity within the research workforce and relevant authorship positions may foster equity in the evidence generated.

Can healthcare choice be predicted using stated preference data? The role of model complexity in a discrete choice experiment about colorectal cancer screening

INTRODUCTION

The validity of discrete choice experiments (DCEs) is crucial to its usage in healthcare decision-making, but there is only a limited number of health contexts in which external validity is demonstrated. This study aims to assess the internal and external validity of the DCE in the context of colorectal cancer (CRC) screening, and gather insights into the discrepancy between stated and revealed preferences.

METHODS

Stated and revealed preferences were elicited on an individual level from Dutch residents eligible for CRC screening in a DCE and a field experiment, respectively (N = 568). To identify the determinants of CRC screening participation and their relative importance, five random utility maximisation models that varied in complexity were used. We assessed the accuracy with which the models based on stated preferences predict individual-level screening choice in a holdout task (internal validity) and in the actual screening choice (external validity). Insights into the discrepancy between stated and revealed preferences were gathered by comparing groups of respondents.

RESULTS

Our findings show high internal and external validity. Choices could be accurately predicted for 95% of the respondents in the holdout task, and 90% in the actual screening choice. When scale and preference heterogeneity were taken into account model fit improved; individual-level prediction accuracy slightly increased for the holdout task but not for the actual screening choice. Respondents for whom stated preferences matched revealed preferences were generally in better health and found the GP's support for their screening decision more important.

DISCUSSION

Evidence was found that revealed preferences can be predicted accurately on an individual level. Incorporating heterogeneity improved internal validity but not external validity. Differences between stated and revealed preferences can be attributed to respondents' health and the support of their GP. We suggest researchers to continue investigating the internal and external validity of discrete choice experiments, and the role of model complexity.

Trial-level bias score versus mean bias score: Comparison of the reliability and external validity using dot-probe task among daily smokers

INTRODUCTION

Attentional bias (AB) is an individual difference risk factor that represents the extent to which cigarette cues capture one's attention. AB is typically indexed by mean bias score (MBS), theoretically assuming that AB is static. However, poor reliability of MBS has threatened valid interpretation of the results on AB. Based on observed trial-by-trial temporal fluctuation and variability of attentional allocation, trial-level bias score (TLBS) has been introduced as an alternative index with evidence of better psychometric properties in various populations, as compared to MBS. However, such evidence is limited among daily smokers. The current study aimed to replicate and extend extant findings in a sample of daily smokers by hypothesizing that TLBS, as compared to MBS, would demonstrate superior reliability and external validity.

METHODS

Forty-eight daily smokers completed self-reports, ad-libitum smoking, and a dot-probe task three times, which was comprised of 36 pairs of pictorial stimuli of cigarette and neutral cues, yielding 144 total trials.

RESULTS

The TLBS demonstrated superior internal (range intra class correlation [ICC] = 0.79-0.95) and test-retest reliability (range ICC = 0.64-0.88) compared to MBS (range ICC = 0.31-0.40 and 0.06-0.16, respectively). However, few significant relations between either the MBS or TLBS and measures of biobehavioral and self-report indices of smoking reinforcement were observed.

CONCLUSIONS

The current findings demonstrate that TLBS, as compared to MBS, is a more reliable measure of AB among daily smokers, while evidence of its external validity is limited.

External Validity of Randomised Controlled Trials on Carotid Revascularisation: Trial Populations May Not Always Reflect Patients in Clinical Practice

OBJECTIVE

The external validity of randomised controlled trials (RCTs) and their transferability to clinical practice is under investigated. This study aimed to analyse the exclusion criteria of recent carotid RCTs comparing carotid endarterectomy (CEA) and carotid artery stenting, and to assess the eligibility of consecutive clinical practice cohorts to those RCTs.

METHODS

An analysis of the clinical and anatomical exclusion criteria of RCTs for asymptomatic (SPACE-2, ACST-2, CREST-1, and CREST-2) and symptomatic carotid stenosis (SPACE-1, CREST-1, ICSS, and EVA-3S) was performed. Two hundred consecutive asymptomatic and 200 consecutive symptomatic patients, treated by CEA, or transfemoral or transcarotid artery stenting at a tertiary referral university centre were assessed for their potential eligibility for each corresponding RCT. RCT patient data were pooled and differences from the clinical practice cohort analysed. Statistics were descriptive and comparative using Fisher's exact and t tests.

RESULTS

The number of clinical and anatomical exclusion criteria differed widely between RCTs. Potential eligibility rates of the clinical practice cohort for RCTs with regard to asymptomatic carotid stenosis were 80.5% (ACST-2), 79.5% (SPACE-2), 47% (CREST-1), and 20% (CREST-2). For RCTs on symptomatic carotid stenosis the eligibility rates were 89% (ICSS), 86.5% (EVA-3S), 64% (SPACE-1), and 39% (CREST-1). Both clinical practice cohorts were older by about three years and patients were more often male vs. the RCTs. Furthermore, a history of smoking (asymptomatic patients), hypertension (symptomatic patients), and atrial fibrillation was diagnosed more often, whereas hypercholesterolaemia and coronary heart disease (asymptomatic patients) were less prevalent. More clinical practice patients were on antiplatelets, anticoagulants, and lipid lowering drugs. Symptomatic clinical practice patients presented more often with retinal ischaemia and less often with minor hemispheric strokes than patients in the RCTs.

CONCLUSION

The external validity of contemporary carotid RCTs varies considerably. Patients in routine clinical practice differ from RCT populations with respect to age, comorbidities, and medication. These data are of interest for clinicians and guideline authors and may be relevant for the design of future comparative trials.

Who is in the sample? An analysis of real and surrogate users as participants in user study research in the information technology fields

BACKGROUND

Constructing a sample of real users as participants in user studies is considered by most researchers to be vital for the validity, usefulness, and applicability of research findings. However, how often user studies reported in information technology academic literature sample real users or surrogate users is unknown. Therefore, it is uncertain whether or not the use of surrogate users in place of real users is a widespread problem within user study practice.

OBJECTIVE

To determine how often user studies reported in peer-reviewed information technology literature sample real users or surrogate users as participants.

METHOD

We analyzed 725 user studies reported in 628 peer-reviewed articles published from 2013 through 2021 in 233 unique conference and journal outlets, retrieved from the ACM Digital Library, IEEE Xplore, and Web of Science archives. To study the sample selection choices, we categorized each study as generic (i.e., users are from the general population) or targeted (i.e., users are from a specific subpopulation), and the sampled study participants as real users (i.e., from the study population) or surrogate users (i.e., other than real users).

RESULTS

Our analysis of all 725 user studies shows that roughly two-thirds (75.4%) sampled real users. However, of the targeted studies, only around half (58.4%) sampled real users. Of the targeted studies sampling surrogate users, the majority (69.7%) used students, around one-in-four (23.6%) sampled through crowdsourcing, and the remaining 6.7% of studies used researchers or did not specify who the participants were.

CONCLUSIONS

Key findings are as follows: (a) the state of sampling real users in information technology research has substantial room for improvement for targeted studies; (b) researchers often do not explicitly characterize their study participants in adequate detail, which is probably the most disconcerting finding; and (c) suggestions are provided for recruiting real users, which may be challenging for researchers.

IMPLICATIONS

The results imply a need for standard guidelines for reporting the types of users sampled for a user study. We provide a template for reporting user study sampling with examples.

Assessment of the external validity of the National Comprehensive Cancer Network (NCCN) guidelines for pancreatic ductal adenocarcinoma in a population of older patients aged 70 years and older

INTRODUCTION

Pancreatic ductal adenocarcinoma (PDAC) mainly occurs in older adults. Since randomized clinical trials (RCTs) provide the highest-quality evidence incorporated in NCCN recommendations, the underrepresentation of older patients in RCTs challenges guidelines' external validity and limits the solidity of evidence in this specific population.

MATERIALS AND METHODS

The study aimed to investigate external validity of NCCN guidelines for PDCA and the impact of reference studies eligibility on overall survival (OS) in a real-world older population. We retrieved RCTs supporting NCCN recommendations for management of PDAC and identified ten topics. We matched a cohort of 707 PDAC patients aged ≥70 years from the Moffitt Cancer Center database with eligibility criteria of 96 reference RCTs to check the proportion of patients eligible for at least two RCTs. Eligibility >60% was rated full validity, 30%-60% partial validity and < 30% limited validity. We also performed log-rank test to assessed whether "eligibility" status affects OS, stratifying by age (70-74; 75-79; ≥80).

RESULTS AND DISCUSSION

We found full validity for neoadjuvant (57/73 patients; 69.86%), locally advanced (28/39; 71.79%) and second line (88/110; 80%) treatment, while lowest validity was found for adjuvant chemotherapy (37/86; 43%). Eligible status was correlated with a significant OS benefit for adjuvant chemoradiation (p = 0.002) in all-comers and for first-line polychemotherapy in patients aged ≥80 (p = 0.01). Our analysis supports the limitation of guidelines' external validity in older patients, and hints at possible correlations with survival, although no definitive conclusions can be drawn at this stage. Renewing RCT design with broader eligibility criteria might help increase inclusion of older and thus strengthen the evidence.

Under-representation of key demographic groups in opioid use disorder trials

Background

The extent to which clinical trials of medications for opioid use disorder (MOUD) are representative or not is unknown. Some patient characteristics modify MOUD effectiveness; if these same characteristics differ in distribution between the trial population and usual-care population, this could contribute to lack of generalizability-a discrepancy between trial and usual-care effectiveness. Our objective was to identify interpretable, multidimensional subgroups who were prescribed MOUD in substance use treatment programs in the US but who were not represented or under-represented by clinical trial participants.

Methods

This was a secondary descriptive analysis of trial and real-world data. The trial data included twenty-seven US opioid treatment programs in the National Drug Abuse Treatment Clinical Trials Network, N = 2,199 patients. The real-world data included US substance use treatment programs that receive public funding, N = 740,015 patients. We characterized real-world patient populations who were non-represented and under-represented in the trial data in terms of sociodemographic and clinical characteristics that could modify MOUD effectiveness.

Results

We found that 10.7% of MOUD patients in TEDS-A were not represented in the three clinical trials. As expected, pregnant MOUD patients (n = 19,490) were not represented. Excluding pregnancy, education and marital status from the characteristics, 2.6% of MOUD patients were not represented. Patients aged 65 years and older (n = 11,204), and those 50-64 years who identified as other (non-White, non-Black, and non-Hispanic) race/ethnicity or multi-racial (n = 7,281) were under-represented.

Conclusions

Quantifying and characterizing non- or under-represented subgroups in trials can provide the data necessary to improve representation in future trials and address research-to-practice gaps.

Reply to: "Research on agroforestry systems and biodiversity conservation: what can we conclude so far and what should we improve?" by Boinot et al. 2022

In our article ‘European agroforestry has no unequivocal effect on biodiversity: a time-cumulative meta-analysis’ (BMC Ecology and Evolution, 2021) we synthesize the effect of agroforestry on biodiversity. Boinot et al. (BMC Ecology and Evolution, 2022) criticise our approach arguing that our definitions of agroforestry and biodiversity are too narrow; that we use inappropriate control sites for primary studies lacking distance to the treatment sites; that there are too few studies for a meta-analysis in silvoarable systems; and that local practice should be emphasized. We agree on the importance of local practices and that more studies would improve the robustness of our meta-analytical results. However, our conclusions are robust to removing studies criticised for inappropriate controls. We also recognize the problem of different definitions of agroforestry and using species richness as sole proxy for biodiversity. We appreciate being given the opportunity to clarify our results and to trigger future discussions about definitions and the interpretation of results from meta-analysis.

Identification of tools used to assess the external validity of randomized controlled trials in reviews: a systematic review of measurement properties

Background

Internal and external validity are the most relevant components when critically appraising randomized controlled trials (RCTs) for systematic reviews. However, there is no gold standard to assess external validity. This might be related to the heterogeneity of the terminology as well as to unclear evidence of the measurement properties of available tools. The aim of this review was to identify tools to assess the external validity of RCTs. It was further, to evaluate the quality of identified tools and to recommend the use of individual tools to assess the external validity of RCTs in future systematic reviews.

Methods

A two-phase systematic literature search was performed in four databases: PubMed, Scopus, PsycINFO via OVID, and CINAHL via EBSCO. First, tools to assess the external validity of RCTs were identified. Second, studies investigating the measurement properties of these tools were selected. The measurement properties of each included tool were appraised using an adapted version of the COnsensus based Standards for the selection of health Measurement INstruments (COSMIN) guidelines.

Results

38 publications reporting on the development or validation of 28 included tools were included. For 61% (17/28) of the included tools, there was no evidence for measurement properties. For the remaining tools, reliability was the most frequently assessed property. Reliability was judged as “sufficient” for three tools (very low certainty of evidence). Content validity was rated as “sufficient” for one tool (moderate certainty of evidence).

Conclusions

Based on these results, no available tool can be fully recommended to assess the external validity of RCTs in systematic reviews. Several steps are required to overcome the identified difficulties to either adapt and validate available tools or to develop a better suitable tool.

Trial registration

Prospective registration at Open Science Framework (OSF): 10.17605/OSF.IO/PTG4D.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01561-5.

Principles and framework for assessing the risk of bias for studies included in comparative quantitative environmental systematic reviews

The internal validity of conclusions about effectiveness or impact in systematic reviews, and of decisions based on them, depends on risk of bias assessments being conducted appropriately. However, a random sample of 50 recently-published articles claiming to be quantitative environmental systematic reviews found 64% did not include any risk of bias assessment, whilst nearly all that did omitted key sources of bias. Other limitations included lack of transparency, conflation of quality constructs, and incomplete application of risk of bias assessments to the data synthesis. This paper addresses deficiencies in risk of bias assessments by highlighting core principles that are required for risk of bias assessments to be fit-for-purpose, and presenting a framework based on these principles to guide review teams on conducting risk of bias assessments appropriately and consistently. The core principles require that risk of bias assessments be Focused, Extensive, Applied and Transparent (FEAT). These principles support risk of bias assessments, appraisal of risk of bias tools, and the development of new tools. The framework follows a Plan-Conduct-Apply-Report approach covering all stages of risk of bias assessment. The scope of this paper is comparative quantitative environmental systematic reviews which address PICO or PECO-type questions including, but not limited to, topic areas such as environmental management, conservation, ecosystem restoration, and analyses of environmental interventions, exposures, impacts and risks.

Differential adherence to peer and nurse components of a supportive care package-The appeal of peer support may be related to women's health and psychological status

OBJECTIVE

Knowledge of factors associated with intervention non-adherence may provide insights into the clinical utility of non-pharmacologic interventions.

METHODS

This study compared complete and incomplete adherers to two separate components of a novel intervention package for women undergoing curative intent radiotherapy for gynaecological cancer on socio-demographic, clinical and pre-radiotherapy patient-reported outcomes data.

RESULTS

Adherence to the tailored specialist nurse consultations was satisfactory (71% participated in all available sessions, 19% participated in all but one). Adherence to the telephone peer support sessions was less satisfactory (47% participated in all available sessions, 24% participated in all but one session). Complete adherers to the peer sessions reported significantly lower levels of psychological distress and significantly higher levels of physical, emotional and functional wellbeing before radiotherapy. No other statistically significant differences were observed between complete and incomplete adherers to the nurse- or peer-led sessions.

CONCLUSION

Women's ability or motivation to engage with peer support may be influenced by their health and psychological status. Further, the extent of intervention non-adherence to the peer-led component may have compromised the assessment of its efficacy.

PRACTICE IMPLICATIONS

Peer support may be less acceptable or appropriate for women with more complex care needs. Such women may prefer specialised care from trained professionals.

How Within-Study Comparisons can Inform Research and Evaluation Design for the Third Sector

Within-study comparisons (WSCs) compare quasi-experimental results to an experimental benchmark and assess the extent to which quasi-experiments (QEs) sacrifice internal validity for external validity. WSCs inform decisions about when to use experiments or QEs, as well as methodological decisions (e.g., covariate selection) about how to implement QEs when they are necessary or preferred. We review the methodological literature on WSCs and present the findings from a WSC of a school voucher program as an example of the use of WSCs to inform policy debates involving the third sector. Prior literature and our results suggest that QE evaluations of third-sector institutions can produce low levels of bias, but only if analytic models include key proxy variables for voluntary self-selection into programs.

Conventional laboratory housing increases morbidity and mortality in research rodents: results of a meta-analysis

BACKGROUND

Over 120 million mice and rats are used annually in research, conventionally housed in shoebox-sized cages that restrict natural behaviours (e.g. nesting and burrowing). This can reduce physical fitness, impair thermoregulation and reduce welfare (e.g. inducing abnormal stereotypic behaviours). In humans, chronic stress has biological costs, increasing disease risks and potentially shortening life. Using a pre-registered protocol ( https://atrium.lib.uoguelph.ca/xmlui/handle/10214/17955 ), this meta-analysis therefore tested the hypothesis that, compared to rodents in 'enriched' housing that better meets their needs, conventional housing increases stress-related morbidity and all-cause mortality.

RESULTS

Comprehensive searches (via Ovid, CABI, Web of Science, Proquest and SCOPUS on May 24 2020) yielded 10,094 publications. Screening for inclusion criteria (published in English, using mice or rats and providing 'enrichments' in long-term housing) yielded 214 studies (within 165 articles, using 6495 animals: 59.1% mice; 68.2% male; 31.8% isolation-housed), and data on all-cause mortality plus five experimentally induced stress-sensitive diseases: anxiety, cancer, cardiovascular disease, depression and stroke. The Systematic Review Center for Laboratory animal Experimentation (SYRCLE) tool assessed individual studies' risks of bias. Random-effects meta-analyses supported the hypothesis: conventional housing significantly exacerbated disease severity with medium to large effect sizes: cancer (SMD = 0.71, 95% CI = 0.54-0.88); cardiovascular disease (SMD = 0.72, 95% CI = 0.35-1.09); stroke (SMD = 0.87, 95% CI = 0.59-1.15); signs of anxiety (SMD = 0.91, 95% CI = 0.56-1.25); signs of depression (SMD = 1.24, 95% CI = 0.98-1.49). It also increased mortality rates (hazard ratio = 1.48, 95% CI = 1.25-1.74; relative median survival = 0.91, 95% CI = 0.89-0.94). Meta-regressions indicated that such housing effects were ubiquitous across species and sexes, but could not identify the most impactful improvements to conventional housing. Data variability (assessed via coefficient of variation) was also not increased by 'enriched' housing.

CONCLUSIONS

Conventional housing appears sufficiently distressing to compromise rodent health, raising ethical concerns. Results also add to previous work to show that research rodents are typically CRAMPED (cold, rotund, abnormal, male-biased, poorly surviving, enclosed and distressed), raising questions about the validity and generalisability of the data they generate. This research was funded by NSERC, Canada.

Generality of Findings From Single-Case Designs: It's Not All About the "N"

There is a common misconception in applied research that generalizations from a study to a specific client can only be made with a large sample size. In single-case design research, however, generalizations are made from a line of replication studies rather than from a single large-N study. In this brief tutorial, we summarize how generalizations are made from single-case design research, and provide a model elevator speech to assist behavior analysts in talking about single-case design research with others.

Appraising clinical applicability of studies: mapping and synthesis of current frameworks, and proposal of the FrACAS framework and VICORT checklist

BACKGROUND

Not all research findings are translated to clinical practice. Reasons for lack of applicability are varied, and multiple frameworks and criteria exist to appraise the general applicability of epidemiological and clinical research. In this two-part study, we identify, map, and synthesize frameworks and criteria; we develop a framework to assist clinicians to appraise applicability specifically from a clinical perspective.

METHODS

We conducted a literature search in PubMed and Embase to identify frameworks appraising applicability of study results. Conceptual thematic analysis was used to synthesize frameworks and criteria. We carried out a framework development process integrating contemporary debates in epidemiology, findings from the literature search and synthesis, iterative pilot-testing, and brainstorming and consensus discussions to propose a concise framework to appraise clinical applicability.

RESULTS

Of the 4622 references retrieved, we identified 26 unique frameworks featuring 21 criteria. Frameworks and criteria varied by scope and level of aggregation of the evidence appraised, target user, and specific area of applicability (internal validity, clinical applicability, external validity, and system applicability). Our proposed Framework Appraising the Clinical Applicability of Studies (FrACAS) classifies studies in three domains (research, practice informing, and practice changing) by examining six criteria sequentially: Validity, Indication-informativeness, Clinical relevance, Originality, Risk-benefit comprehensiveness, and Transposability (VICORT checklist).

CONCLUSIONS

Existing frameworks to applicability vary by scope, target user, and area of applicability. We introduce FrACAS to specifically assess applicability from a clinical perspective. Our framework can be used as a tool for the design, appraisal, and interpretation of epidemiological and clinical studies.

Generalisability of Randomised Controlled Trials in Heart Failure with Reduced Ejection Fraction

Background

Heart failure (HF) trials have stringent inclusion and exclusion criteria, but limited data exist regarding generalizability of trials. We compared patient characteristics and outcomes between patients with HF and reduced ejection fraction (HFrEF) in trials and observational registries.

Methods and Results

Individual patient data for 16 922 patients from five randomized clinical trials and 46 914 patients from two HF registries were included. The registry patients were categorized into trial-eligible and non-eligible groups using the most commonly used inclusion and exclusion criteria. A total of 26 104 (56%) registry patients fulfilled the eligibility criteria. Unadjusted all-cause mortality rates at 1 year were lowest in the trial population (7%), followed by trial-eligible patients (12%) and trial-non-eligible registry patients (26%). After adjustment for age and sex, all-cause mortality rates were similar between trial participants and trial-eligible registry patients [standardized mortality ratio (SMR) 0.97; 95% confidence interval (CI) 0.92–1.03] but cardiovascular mortality was higher in trial participants (SMR 1.19; 1.12–1.27). After full case-mix adjustment, the SMR for cardiovascular mortality remained higher in the trials at 1.28 (1.20–1.37) compared to RCT-eligible registry patients.

Conclusion

In contemporary HF registries, over half of HFrEF patients would have been eligible for trial enrolment. Crude clinical event rates were lower in the trials, but, after adjustment for case-mix, trial participants had similar rates of survival as registries. Despite this, they had about 30% higher cardiovascular mortality rates. Age and sex were the main drivers of differences in clinical outcomes between HF trials and observational HF registries.

Predictive processing models and affective neuroscience

The neural bases of affective experience remain elusive. Early neuroscience models of affect searched for specific brain regions that uniquely carried out the computations that underlie dimensions of valence and arousal. However, a growing body of work has failed to identify these circuits. Research turned to multivariate analyses, but these strategies, too, have made limited progress. Predictive processing models offer exciting new directions to address this problem. Here, we use predictive processing models as a lens to critique prevailing functional neuroimaging research practices in affective neuroscience. Our review highlights how much work relies on rigid assumptions that are inconsistent with a predictive processing approach. We outline the central aspects of a predictive processing model and draw out their implications for research in affective and cognitive neuroscience. Predictive models motivate a reformulation of "reverse inference" in cognitive neuroscience, and placing a greater emphasis on external validity in experimental design.

Baseline characteristics and comparability of older multimorbid patients with polypharmacy and general practitioners participating in a randomized controlled primary care trial

Objectives

Recruiting general practitioners (GPs) and their multimorbid older patients for trials is challenging for multiple reasons (e.g., high workload, limited mobility). The comparability of study participants is important for interpreting study findings. This manuscript describes the baseline characteristics of GPs and patients participating in the ‘Optimizing PharmacoTherapy in older multimorbid adults In primary CAre’ (OPTICA) trial, a study of optimization of pharmacotherapy for multimorbid older adults. The overall aim of this study was to determine if the GPs and patients participating in the OPTICA trial are comparable to the real-world population in Swiss primary care.

Design

Analysis of baseline data from GPs and patients in the OPTICA trial and a reference cohort from the FIRE (‘Family medicine ICPC Research using Electronic medical records’) project.

Setting

Primary care, Switzerland.

Participants

Three hundred twenty-three multimorbid (≥ 3 chronic conditions) patients with polypharmacy (≥ 5 regular medications) aged ≥ 65 years and 43 GPs recruited for the OPTICA trial were compared to 22,907 older multimorbid patients with polypharmacy and 227 GPs from the FIRE database.

Methods

We compared the characteristics of GPs and patients participating in the OPTICA trial with other GPs and other older multimorbid adults with polypharmacy in the FIRE database. We described the baseline willingness to have medications deprescribed of the patients participating in the OPTICA trial using the revised Patients’ Attitudes Towards Deprescribing (rPATD) questionnaire.

Results

The GPs in the FIRE project and OPTICA were similar in terms of sociodemographic characteristics and their work as a GP (e.g. aged in their fifties, ≥ 10 years of experience, ≥ 60% are self-employed, ≥ 80% work in a group practice). The median age of patients in the OPTICA trial was 77 years and 45% of trial participants were women. Patients participating in the OPTICA trial and patients in the FIRE database were comparable in terms of age, certain clinical characteristics (e.g. systolic blood pressure, body mass index) and health services use (e.g. selected lab and vital data measurements). More than 80% of older multimorbid patients reported to be willing to stop ≥ 1 of their medications if their doctor said that this would be possible.

Conclusion

The characteristics of patients and GPs recruited into the OPTICA trial are relatively comparable to characteristics of a real-world Swiss population, which indicates that recruiting a generalizable patient sample is possible in the primary care setting. Multimorbid patients in the OPTICA trial reported a high willingness to have medications deprescribed.

Trial registration

Clinicaltrials.gov (NCT03724539), KOFAM (Swiss national portal) (SNCTP000003060), Universal Trial Number (U1111-1226-8013)

Supplementary Information

The online version contains supplementary material available at 10.1186/s12875-021-01488-8.

FIT FOR PURPOSE IN ACTION: DESIGN, IMPLEMENTATION, AND EVALUATION OF THE NATIONAL INTERNET FLU SURVEY

Researchers strive to design and implement high-quality surveys to maximize the utility of the data collected. The definitions of quality and usefulness, however, vary from survey to survey and depend on the analytic needs. Survey teams must evaluate the trade-offs of various decisions, such as when results are needed and their required level of precision, in addition to practical constraints like budget, before finalizing the design. Characteristics within the concept of fit for purpose (FfP) can provide the framework for considering the trade-offs. Furthermore, this tool can enable an evaluation of quality for the resulting estimates. Implementation of a FfP framework in this context, however, is not straightforward. In this article, we provide the reader with a glimpse of a FfP framework in action for obtaining estimates on early season influenza vaccination coverage estimates and on knowledge, attitudes, behaviors, and barriers related to influenza and influenza prevention among civilian noninstitutionalized adults aged 18 years and older in the United States. The result is the National Internet Flu Survey (NIFS), an annual, two-week internet survey sponsored by the US Centers for Disease Control and Prevention. In addition to critical design decisions, we use the established NIFS FfP framework to discuss the quality of the NIFS in meeting the intended objectives. We highlight aspects that work well and other survey traits requiring further evaluation. Differences found in comparing the NIFS to the National Flu Survey, the National Health Interview Survey, and Behavioral Risk Factor Surveillance System are discussed via their respective FfP characteristics. The findings presented here highlight the importance of the FfP framework for designing surveys, defining data quality, and providing a set a metrics used to advertise the intended use of the survey data and results.

Predicting clinical outcomes via human fear conditioning: A narrative review

A common assumption in human fear conditioning research is that findings are informative for the etiology and treatment of clinical anxiety. One way to empirically evaluate the external validity of fear conditioning is by prospective studies. We review available prospective research investigating whether individual performance in fear conditioning predicts individual differences in anxiety levels and exposure-based treatment outcome. We focus on fear extinction, generalization, acquisition, and avoidance. Results suggest that reduced extinction and broader generalization predict higher anxiety levels. Results with respect to the predictive value of acquisition for anxiety levels are mixed. With regard to predicting exposure-based treatment outcome, some studies do find an association with extinction whereas others do not. The majority of studies does not find an association with acquisition. Evidence on extinction recall is limited and not consistent. The interpretation of these results requires caution. The number of available studies is limited. It is possible that not all work, in particular studies with only null effects, has found its way to publication. Future research on this topic will benefit from large sample sizes, preregistered hypotheses, full transparency about the conducted analyses and the publication of high-quality studies with null effects.

Developing the INCLUDE Ethnicity Framework-a tool to help trialists design trials that better reflect the communities they serve

Background

Ensuring that a trial is designed so that its participants reflect those who might benefit from the results, or be spared harms, is key to the potential benefits of the trial reaching all they should. This paper describes the process, facilitated by Trial Forge, that was used between July 2019 and October 2020 to develop the INCLUDE Ethnicity Framework, part of the wider INCLUDE initiative from the National Institute for Health Research to improve inclusion of under-served groups in clinical research studies.

Methods

Development of the Framework was done in seven phases: (1) outline, (2) initial draft, (3) stakeholder meeting, (4) modify draft, (5) Stakeholder feedback, (6) applying the Framework and (7) packaging. Phases 2 and 3 were face-to-face meetings. Consultation with stakeholders was iterative, especially phases 4 to 6. Movement to the next phase was done once all or most stakeholders were comfortable with the results of the current phase. When there was a version of the Framework that could be considered final, the Framework was applied to six trials to create a set of examples (phase 6). Finally, the Framework, guidance and examples were packaged ready for dissemination (phase 7).

Results

A total of 40 people from stakeholder groups including patient and public partners, clinicians, funders, academics working with various ethnic groups, trial managers and methodologists contributed to the seven phases of development. The Framework comprises two parts. The first part is a list of four key questions:

Who should my trial apply to?

Are the groups identified likely to respond in different ways?

Will my study intervention make it harder for some groups to engage?

Will the way I have designed the study make it harder for some groups to engage?

The second part is a set of worksheets to help trial teams address these questions. The Framework can be used for any stage of trial, for a healthcare intervention in any disease area. The Framework was launched on 1st October 2020 and is available open access at the Trial Forge website: https://www.trialforge.org/trial-forge-centre/include/.

Conclusion

Thinking about the number of people in our trials is not enough: we need to start thinking more carefully about who our participants are.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13063-021-05276-8.

Population Diversity Challenge the External Validity of the European Randomized Controlled Trials Comparing Laparoscopic Gastric Bypass and Sleeve Gastrectomy

INTRODUCTION

Two randomized controlled trials (RCTs) from Europe recently showed similar weight loss and rates of type 2 diabetes (T2D) remission following laparoscopic gastric bypass (LRYGB) and laparoscopic sleeve gastrectomy (LSG). However, results from observational studies in the United States (US) have discordant results. We compared 1-year weight loss and T2D remission between LRYGB and LSG in a heterogeneous patient cohort from the US, albeit with similar inclusion and exclusion criteria to the European RCTs.

METHODS

Logistic regression was used to propensity match LSG and LRYGB patients according to age, gender, race, preoperative BMI, and T2D. Inclusion and exclusion criteria were adopted from the two European RCTs. Demographic, anthropometric, weight outcomes, and comorbidities prevalence were compared at baseline and 1-year follow-up.

RESULTS

We included 278 patients (139 LSG and 139 RYGB; median age 42 years, 89% female, 57% black race, 22% with public health insurance, and 25% with T2D). One year after surgery, mean %EWL was 77.3 ± 19.5% with LRYGB and 63.1 ± 21% with LSG (P < 0.001). Mean %TWL was 34.2 ± 7.3% after LRYGB and 28.1 ± 8.2% after LSG, (P < 0.001). The proportion of patients who achieved T2D remission was comparable between surgeries (LRGYB: 68.6% vs. LSG: 66.7%, P = 0.89). LSG, older age, black race, and higher preoperative BMI were independently associated with lower %EWL. Independent correlates of weight loss were different for LRYGB and LSG.

CONCLUSIONS

Weight loss, but not the likelihood of T2D remission, was greater with LRYGB than LSG in a diverse patient cohort in the US. Further research efforts connecting population diversity to discordant results across studies is needed to better counsel patients with regards to expected postoperative outcomes.

External validity of phase III trials on vaccines against SARS-CoV-2 to a middle-aged and elderly Western European population

Initial results from various phase-III trials on vaccines against SARS-CoV-2 are promising. For proper translation of these results to clinical guidelines, it is essential to determine how well the general population is reflected in the study populations of these trials. This study was conducted among 7162 participants (age-range: 51-106 years; 58% women) from the Rotterdam Study. We quantified the proportion of participants that would be eligible for the nine ongoing phase-III trials. We further quantified the eligibility among participants at high risk to develop severe COVID-19. Since many trials were not explicit in their exclusion criterion with respect to 'acute' or 'unstable preexisting' diseases, we performed two analyses. First, we included all participants irrespective of this criterion. Second, we excluded persons with acute or 'unstable preexisting' diseases. 97% of 7162 participants was eligible for any trial with eligibility for separate trials ranging between 11-97%. For high-risk individuals the corresponding numbers were 96% for any trial with separate trials ranging from 5-96%. Importantly, considering persons ineligible due to 'acute' or 'unstable pre-existing' disease drastically dropped the eligibilities for all trials below 43% for the total population and below 36% for high-risk individuals. The eligibility for ongoing vaccine trials against SARS-CoV-2 can reduce by half depending on interpretation and application of a single unspecified exclusion criterion. This exclusion criterion in our study would especially affect the elderly and those with pre-existing morbidities. These findings thus indicate the difficulty as well as importance of developing clinical recommendations for vaccination and applying these to the appropriate target populations. This becomes especially paramount considering the fact that many countries worldwide have initiated their vaccination programs by first targeting the elderly and most vulnerable persons.

Dynamic meta-analysis: a method of using global evidence for local decision making

Background

Meta-analysis is often used to make generalisations across all available evidence at the global scale. But how can these global generalisations be used for evidence-based decision making at the local scale, if the global evidence is not perceived to be relevant to local decisions? We show how an interactive method of meta-analysis—dynamic meta-analysis—can be used to assess the local relevance of global evidence.

Results

We developed Metadataset (www.metadataset.com) as a proof-of-concept for dynamic meta-analysis. Using Metadataset, we show how evidence can be filtered and weighted, and results can be recalculated, using dynamic methods of subgroup analysis, meta-regression, and recalibration. With an example from agroecology, we show how dynamic meta-analysis could lead to different conclusions for different subsets of the global evidence. Dynamic meta-analysis could also lead to a rebalancing of power and responsibility in evidence synthesis, since evidence users would be able to make decisions that are typically made by systematic reviewers—decisions about which studies to include (e.g. critical appraisal) and how to handle missing or poorly reported data (e.g. sensitivity analysis).

Conclusions

In this study, we show how dynamic meta-analysis can meet an important challenge in evidence-based decision making—the challenge of using global evidence for local decisions. We suggest that dynamic meta-analysis can be used for subject-wide evidence synthesis in several scientific disciplines, including agroecology and conservation biology. Future studies should develop standardised classification systems for the metadata that are used to filter and weight the evidence. Future studies should also develop standardised software packages, so that researchers can efficiently publish dynamic versions of their meta-analyses and keep them up-to-date as living systematic reviews. Metadataset is a proof-of-concept for this type of software, and it is open source. Future studies should improve the user experience, scale the software architecture, agree on standards for data and metadata storage and processing, and develop protocols for responsible evidence use.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12915-021-00974-w.

Barriers impacting the POINT pragmatic trial: the unavoidable overlap between research and intervention procedures in "real-world" research

Background

This manuscript provides a research update to the ongoing pragmatic trial of Project POINT (Planned Outreach, Intervention, Naloxone, and Treatment), an emergency department-based peer recovery coaching intervention for linking patients with opioid use disorder to evidence-based treatment. The research team has encountered a number of challenges related to the “real-world” study setting since the trial began. Using an implementation science lens, we sought to identify and describe barriers impacting both the intervention and research protocols of the POINT study, which are often intertwined in pragmatic trials due to the focus on external validity.

Method

Qualitative data were collected from 3 peer recovery coaches, 2 peer recovery coach supervisors, and 3 members of the research team. Questions and deductive qualitative analysis were guided by the Consolidated Framework for Implementation Research (CFIR).

Results

Nine unique barriers were noted, with 5 of these barriers impacting intervention and research protocol implementation simultaneously. These simultaneous barriers were timing of intervention delivery, ineffective communication with emergency department staff, lack of privacy in the emergency department, the fast-paced emergency department setting, and patient’s limited resources. Together, these barriers represent the intervention characteristics, inner setting, and outer setting domains of the CFIR.

Conclusion

Results highlight the utility of employing an implementation science framework to assess implementation issues in pragmatic trials and how this approach might be used as a quality assurance mechanism given the considerable overlap that exists between research and intervention protocols in real-world trial settings. Previously undocumented changes to the trial design that have been made as a result of the identified barriers are discussed.

No inexplicable disagreements between real-world data-based nonrandomized controlled studies and randomized controlled trials were found

OBJECTIVES

We assessed disagreements between nonrandomized controlled studies based on real-world data (NRCS-RWDs) and randomized controlled trials (RCTs).

STUDY DESIGN AND SETTING

We systematically searched for studies that compared treatment effect estimates from NRCS-RWDs and RCTs on the same clinical question. We assessed the potential difference between NRCS-RWDs and RCTs related to internal and external validity. We calculated various meta-epidemiological measures to assess agreement. In case of disagreements, we tried to identify the probable causes of disagreements.

RESULTS

We included 12 studies comparing 15 treatment effect estimates of NRCS-RWDs and RCTs. There were many potential causes of disagreement. Ninety-five percent confidence intervals overlapped for 12 of 15 treatment effect estimates. Our analysis on predicted vs. observed overlap showed that there were no more disagreements than expected by chance. We observed only two substantial differences between the 15 treatment effect estimates. In both cases, we identified risk of bias in the NRCS-RWDs as the most probable cause of disagreement.

CONCLUSION

Our findings suggest that there are clinical questions where the difference in risk of bias between a well-conducted NRCS-RWD and an RCT is negligible. In our analysis, threats to external validity appeared to have no or only a weak impact on the disagreements of treatment effect estimates.

Participant characteristics are poorly reported in exercise trials in tendinopathy: A systematic review

OBJECTIVE

To evaluate the reporting of eligibility criteria and baseline participant characteristics in randomised controlled trials investigating the effects of exercise interventions in tendinopathy.

METHODS

Randomised controlled trials investigating the effects of exercise therapy compared to a non-exercising intervention in upper and lower limb tendinopathy were included. Data extraction was categorised into the following domains: participant demographics, tendinopathy descriptors, general health, participant recruitment and eligibility criteria.

RESULTS

The review included the following tendinopathies: Achilles (n = 9), gluteal (n = 2), lateral elbow tendinopathy (n = 15), patellar (n = 3) plantar (n = 3), and rotator cuff (n = 13). Age, sex, duration of symptoms and symptom severity were commonly reported across the review, while prior history of tendinopathy was poorly reported (6/45). Variables such as physical activity level (17/45), sleep (0/45), psychological factors (2/45), medication at baseline (8/45), co morbid health complaints (10/45) and sociodemographic factors (11/45) were poorly reported across the included studies. Substantial variation existed between studies in the specific eligibility criteria used.

CONCLUSION

The findings of this systematic review demonstrate that participant characteristics are poorly reported in exercise trials in tendinopathy. To improve effectiveness of exercise interventions in tendinopathy, improved reporting of participant characteristics may allow better comparisons and targeted interventions for specific subgroups.

The findings of a surgical hip fracture trial were generalizable to the UK national hip fracture database

OBJECTIVE

To estimate the generalizability of treatment effects observed in a randomized trial of hip fracture surgery implants to a broader population of people undergoing hip surgery in the United Kingdom.

STUDY DESIGN AND SETTING

In 2018, the WHiTE-3 trial (n = 958) demonstrated that a modular hemiarthroplasty implant conferred no additional benefit over the traditional monoblock implant for quality of life and length of hospital stay. We compared and weighted the trial sample against two target populations: WHiTE-cohort (n = 2,457) and UK-National Hip Fracture Database (NHFD, n = 190,894), and re-estimate expected treatment effects for the target populations.

RESULTS

Despite differences in baseline characteristics of the trial sample and target populations, the re-estimated treatment effects were comparable. For quality of life, the differences between the trial estimate and WHiTE-cohort and NHFD estimates were 0.01 points on the EuroQol (EQ5D). For length of stay, the difference between the trial estimate and WHiTE-cohort was 0.50 days; and the difference between the trial estimate and NHFD estimate was -0.47 days.

CONCLUSION

This generalizability analysis of the WHiTE-3 trial found that the inferences from the trial can be generalized to a wider population of individuals in the UK NHFD and the WHiTE-cohort who met the inclusion criteria for WHiTE-3.

The weirdness of belief in free will

It has been argued that belief in free will is socially consequential and psychologically universal. In this paper we look at the folk concept of free will and its critical assessment in the context of recent psychological research. Is there a widespread consensus about the conceptual content of free will? We compared English "free will" with its lexical equivalents in Lithuanian, Hindi, Chinese and Mongolian languages and found that unlike Lithuanian, Chinese, Hindi and Mongolian lexical expressions of "free will" do not refer to the same concept free will. What kind people have been studied so far? A review of papers indicate that, overall, 91% of participants in studies on belief in free will were WEIRD. Thus, given that free will has no cross-culturally universal conceptual content and that most of the reviewed studies were based on WEIRD samples, belief in free will is not a psychological universal.

A systematic review of the implementation of obesity prevention interventions in early childcare and education settings using the RE-AIM framework

Previous systematic reviews have examined the efficacy of obesity prevention interventions within early childcare/education settings. Often lacking in these reviews is reporting on external validity, which continues to be underemphasized compared to internal validity. More attention to external validity would help better translate evidence-based interventions to real-world settings. This systematic review aimed to determine the availability of data on both internal and external validity across dimensions of the Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework in studies reporting on obesity prevention interventions in early childcare/education settings. Inclusion criteria included: randomized controlled trials, early childcare/education setting, targeted children 2-6 years old, addressed both diet and physical activity, collected measures of weight status and diet and/or physical activity, and published within the last 10 years. Searches were conducted in ERIC, PsychInfo, and PubMed; 23 studies met inclusion criteria. A validated RE-AIM abstraction tool was used to code studies. Most commonly reported dimensions were Reach (62.3%), Implementation (53.5%), and Efficacy/Effectiveness (48.7%). Adoption (21.7%) and Maintenance (11.6%) were less often reported. All studies reported on primary outcomes, but few reported on RE-AIM indicators of characteristics of participation and adoption, quality of life, methods used to identify staff, staff inclusion/exclusion criteria and adoption rates, implementation fidelity, measures of cost to start-up and deliver the intervention, and indicators of maintenance. This systematic review underscores the need for more focus on external validity to inform replication, dissemination, and implementation so that evidence-based early childcare/education obesity interventions can be generalized to real-world settings.

Patient distrust in pharmaceutical companies: an explanation for women under-representation in respiratory clinical trials?

BACKGROUND

Patient skepticism concerning medical innovations can have major consequences for current public health and may threaten future progress, which greatly relies on clinical research. The primary objective of this study is to determine the variables associated with patient acceptation or refusal to participate in clinical research. Specifically, we sought to evaluate if distrust in pharmaceutical companies and associated psychosocial factors could represent a recruitment bias in clinical trials and thus threaten the applicability of their results.

METHODS

This prospective, multicenter survey consisted in the administration of a self-questionnaire to patients during a pulmonology consultation. The 1025 questionnaires distributed collected demographics, socio-professional and basic health literacy characteristics. Patients were asked to rank their level of trust for pharmaceutical companies and indicate their willingness to participate in different categories of research (pre or post marketing, sponsored by an academic institution or pharmaceutical company). Logistic regression was used to determine factors contributing to "trust" versus "distrust" group membership and willingness to participate in each category of research.

RESULTS

One thousand patients completed the survey, corresponding to a response rate of 97.5%. Data from 838 patients were analyzed in this study. 48.3% of respondents declared that they trusted pharmaceutical companies, while 35.5% declared distrust. Being female (p = 0.042), inactive in the employment market(p = 0.007), and not-knowing the name of one's disease(p = 0.010) are factors related to declared distrust. Distrust-group membership is associated with unwillingness to participate in certain categories of trials such as pre-marketing and industry-sponsored trials.

CONCLUSION

Distrust in pharmaceutical companies is associated with a specific patient profile and with refusal to participate in certain subcategories of trials. This potential recruitment bias may explain the under-representation of certain categories of patients such as women in pre-marketing drug trials.

Glucocorticoid-trials in rheumatoid arthritis mostly study representative real-world patients: A systematic review and meta-analysis

OBJECTIVE

Randomized controlled trials (RCTs) are considered the gold standard in clinical research due to credible causality. Their results, however, may not be generalizable to real-world populations. While glucocorticoids (GCs) remain a mainstay of rheumatoid arthritis (RA) treatment, it is unclear whether the results of GC-RCTs are generalizable to current real-world RA patients.

METHODS

MEDLINE was searched for RCTs and, as comparators, cohort studies (CSs) in RA evaluating systemic GCs. Random-effects meta-analyses were performed for descriptive baseline characteristics (including general demographics, comorbidities, and disease activity) that have been shown to be able to modify the benefit-risk-ratio of various RA therapeutics. These meta-analyses were stratified by study type (RCT and CS). Stratified estimates were subsequently compared. Further sensitivity analyses were performed stratifying by disease duration.

RESULTS

56 RCTs (7053 participants) and 10 CSs (14,688 participants) were included. 12 characteristics were reported frequently enough to allow for comparative analysis. In 10/12 characteristics (83%), RCT estimates did not appear to differ from CS estimates. However, RCT participants were younger (-4.7 years [95% CI -7.2 to -2.1]; p < 0.001) and had higher erythrocyte sedimentation rates (11.8 mm/h [5.7 to 17.8]; p < 0.001) than CS participants. Comorbidities could not be assessed due to insufficient reporting.

CONCLUSION

Our findings suggest that evidence from GC trials in RA is of acceptable generalizability to current real-world patients - especially compared to findings from biologic agents in RA. However, RCT participants were younger than real-world patients, potentially limiting the generalizability of trial results to elderly patients.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO (CRD42019134675).