The Logic of Generalization From Systematic Reviews and Meta-Analyses of Impact Evaluations

Systematic reviews and meta-analyses are viewed as potent tools for generalized causal inference. These reviews are routinely used to inform decision makers about expected effects of interventions. However, the logic of generalization from research reviews to diverse policy and practice contexts is not well developed. Building on sampling theory, concerns about epistemic uncertainty, and principles of generalized causal inference, this article presents a pragmatic approach to generalizability assessment for use with systematic reviews and meta-analyses. This approach is applied to two systematic reviews and meta-analyses of effects of "evidence-based" psychosocial interventions for youth and families. Evaluations included in systematic reviews are not necessarily representative of populations and treatments of interest. Generalizability of results is limited by high risks of bias, uncertain estimates, and insufficient descriptive data from impact evaluations. Systematic reviews and meta-analyses can be used to test generalizability claims, explore heterogeneity, and identify potential moderators of effects. These reviews can also produce pooled estimates that are not representative of any larger sets of studies, programs, or people. Further work is needed to improve the conduct and reporting of impact evaluations and systematic reviews, and to develop practical approaches to generalizability assessment and guide applications of interventions in diverse policy and practice contexts.

How Mixed-Methods Research Can Improve the Policy Relevance of Impact Evaluations

This paper describes how mixed methods can improve the value and policy relevance of impact evaluations, paying particular attention to how mixed methods can be used to address external validity and generalization issues. We briefly review the literature on the rationales for using mixed methods; provide documentation of the extent to which mixed methods have been used in impact evaluations in recent years; describe how we developed a list of recent impact evaluations using mixed methods and the process used to conduct full-text reviews of these articles; summarize the findings from our analysis of the articles; discuss three exemplars of using mixed methods in impact evaluations; and discuss how mixed methods have been used for studying and improving external validity and potential improvements that could be made in this area. We find that mixed methods are rarely used in impact evaluations, and we believe that increased use of mixed methods would be useful because they can reinforce findings from the quantitative analysis (triangulation), and they can also help us understand the mechanism by which programs have their impacts and the reasons why programs fail.

The Myth of the Need for Diversity Among Subjects in Theory-Testing Research: Comments on "Racial Inequality in Psychological Research" by Roberts et al. (2020)

Roberts and colleagues focus on two aspects of racial inequality in psychological research, namely an alleged underrepresentation of racial minorities and the effects attributed to this state of affairs. My comment focuses only on one aspect, namely the assumed consequences of the lack of diversity in subject populations. Representativeness of samples is essential in survey research or applied research that examines whether a particular intervention will work for a particular population. Representativeness or diversity is not necessary in theory-testing research, where we attempt to establish laws of causality. Because theories typically apply to all of humanity, all members of humanity (even American undergraduates) are suitable for assessing the validity of theoretical hypotheses. Admittedly, the assumption that a theory applies to all of humanity is also a hypothesis that can be tested. However, to test it, we need theoretical hypotheses about specific moderating variables. Supporting a theory with a racially diverse sample does not make conclusions more valid than support from a nondiverse sample. In fact, cause-effect conclusions based on a diverse sample might not be valid for any member of that sample.

Consensus on the definition and assessment of external validity of randomized controlled trials: A Delphi study

External validity is an important parameter that needs to be considered for decision making in health research, but no widely accepted measurement tool for the assessment of external validity of randomized controlled trials (RCTs) exists. One of the most limiting factors for creating such a tool is probably the substantial heterogeneity and lack of consensus in this field. The objective of this study was to reach consensus on a definition of external validity and on criteria to assess the external validity of RCTs included in systematic reviews. A three-round online Delphi study was conducted. The development of the Delphi survey was based on findings from a previous systematic review. Potential panelists were identified through a comprehensive web search. Consensus was reached when at least 67% of the panelists agreed to a proposal. Eighty-four panelists from different countries and various disciplines participated in at least one round of this study. Consensus was reached on the definition of external validity ("External validity is the extent to which results of trials provide an acceptable basis for generalization to other circumstances such as variations in populations, settings, interventions, outcomes, or other relevant contextual factors"), and on 14 criteria to assess the external validity of RCTs in systematic reviews. The results of this Delphi study provide a consensus-based reference standard for future tool development. Future research should focus on adapting, pilot testing, and validating these criteria to develop measurement tools for the assessment of external validity.

Visualizing External Validity: Graphical Displays to Inform the Extension of Treatment Effects from Trials to Clinical Practice

BACKGROUND

In the presence of effect measure modification, estimates of treatment effects from randomized controlled trials may not be valid in clinical practice settings. The development and application of quantitative approaches for extending treatment effects from trials to clinical practice settings is an active area of research.

METHODS

In this article, we provide researchers with a practical roadmap and four visualizations to assist in variable selection for models to extend treatment effects observed in trials to clinical practice settings and to assess model specification and performance. We apply this roadmap and visualizations to an example extending the effects of adjuvant chemotherapy (5-fluorouracil vs. plus oxaliplatin) for colon cancer from a trial population to a population of individuals treated in community oncology practices in the United States.

RESULTS

The first visualization screens for potential effect measure modifiers to include in models extending trial treatment effects to clinical practice populations. The second visualization displays a measure of covariate overlap between the clinical practice populations and the trial population. The third and fourth visualizations highlight considerations for model specification and influential observations. The conceptual roadmap describes how the output from the visualizations helps interrogate the assumptions required to extend treatment effects from trials to target populations.

CONCLUSIONS

The roadmap and visualizations can inform practical decisions required for quantitatively extending treatment effects from trials to clinical practice settings.

Transporting results in an observational epidemiology setting: purposes, methods, and applied example

In the medical domain, substantial effort has been invested in generating internally valid estimates in experimental as well as observational studies, but limited effort has been made in testing generalizability, or external validity. Testing the external validity of scientific findings is nevertheless crucial for the application of knowledge across populations. In particular, transporting estimates obtained from observational studies requires the combination of methods for causal inference and methods to transport the effect estimates in order to minimize biases inherent to observational studies and to account for differences between the study and target populations. In this paper, the conceptual framework and assumptions behind transporting results from a population-based study population to a target population is described in an observational setting. An applied example to life-course epidemiology, where internal validity was constructed for illustrative purposes, is shown by using the targeted maximum likelihood estimator.

Standardizing to Specific Target Populations in Distributed Networks and Multi-Site Pharmacoepidemiologic Studies

Distributed networks and other multi-site studies assess drug safety and effectiveness in diverse populations by pooling information. Targeting groups of clinical or policy interest (including specific sites or site combinations) and applying weights based on effect measure modifiers (EMMs) prior to pooling estimates within multi-site studies may increase interpretability and improve precision. We simulated a four-site study, standardized each site using inverse odds weights (IOW) to resemble the three smallest sites or the smallest site, estimated IOW-weighted RDs, and combined estimates with inverse variance weights (IVW). We also created an artificial distributed network in the Clinical Practice Research Datalink (CPRD) Aurum consisting of one site for each geographic region. We compared metformin and sulfonylurea initiators with respect to mortality, targeting the smallest region. In the simulation, IOW reduced differences between estimates and increased precision when targeting the three smallest sites or the smallest site. In the CPRD study, the IOW + IVW estimate was also more precise (smallest region RD and 95% CI: 5.41%, 1.03%-9.79%), IOW+IVW RD and 95% CI: 3.25%, 3.07%-3.43%). When performing pharmacoepidemiologic research in distributed networks or multi-site studies in the presence of EMMs, designating target populations has the potential to improve estimate precision and interpretability.

Safety climate scale for vineyards: an external validity study

OBJECTIVES

Agriculture entered the discussions about safety climates late, despite being one of the most hazardous industries. We recently developed a safety climate scale in Bordeaux vineyards, for which we provided good evidence of reliability and construct validity (Grimbuhler and Viel 2019). In this study, we aimed to establish the external validity of this safety climate scale with the help of an independent national sample of vineyard professionals.

POPULATION AND METHODS

We approached vineyard managers and operators during compulsory training and certification procedures for pesticide-related activities. Trainees giving informed consent for participation in the study were asked to complete a safety climate questionnaire at the start and end of a training session. In total, 406 vineyard managers or operators completed the questionnaire at the start of the study, 37 of whom declined to complete the questionnaire at the end of the training session, leaving 369 subjects available for pretraining/post-training comparisons. Statistical comparisons were based on t-tests and mixed models for repeated measures.

RESULTS

A mean safety climate score of 82.91 (SD: 9.06) was obtained in the initial survey in the Bordeaux region, whereas the safety climate score was estimated at 83.78 (SD 10.39) in this nationwide survey (P = 0.23). A significant increase was observed after the training course, for the safety climate score (7.5%, P < 10-15) and for each of its 7 dimensions (P < 10-4 or less), in both univariate and multivariate analyses. However, the magnitude of these increases varied with dimension, ranging from 2.4% for rules and best practices to 15.5% for communication and feedback.

CONCLUSIONS

External validity was demonstrated by transferability and sensitivity to intervention. This safety climate scale can now be considered to provide a good inference of the safety culture, with a meaning generalizable across vineyards.

Binocular rivalry under naturalistic geometry: Evidence from worlds simulated in virtual reality

Binocular rivalry is a fascinating, widely studied visual phenomenon in which perception alternates between two competing images. This experience, however, is generally restricted to laboratory settings where two irreconcilable images are presented separately to the two eyes, an implausible geometry where two objects occupy the same physical location. Such laboratory experiences are in stark contrast to everyday visual behavior, where rivalry is almost never encountered, casting doubt on whether rivalry is relevant to our understanding of everyday binocular vision. To investigate the external validity of binocular rivalry, we manipulated the geometric plausibility of rival images using a naturalistic, cue-rich, 3D-corridor model created in virtual reality. Rival stimuli were presented in geometrically implausible, semi-plausible, or plausible layouts. Participants tracked rivalry fluctuations in each of these three layouts and for both static and moving rival stimuli. Results revealed significant and canonical binocular rivalry alternations regardless of geometrical plausibility and stimulus type. Rivalry occurred for layouts that mirrored the unnatural geometry used in laboratory studies and for layouts that mimicked real-world occlusion geometry. In a complementary 3D modeling analysis, we show that interocular conflict caused by geometrically plausible occlusion is a common outcome in a visual scene containing multiple objects. Together, our findings demonstrate that binocular rivalry can reliably occur for both geometrically implausible interocular conflicts and conflicts caused by a common form of naturalistic occlusion. Thus, key features of binocular rivalry are not simply laboratory artifacts but generalize to conditions that match the geometry of everyday binocular vision.

Applying Trial-Derived Treatment Effects to Real-World Populations: Generalizing Cost-Effectiveness Estimates When Modeling Complex Hazards

OBJECTIVES

Generalizability of trial-based cost-effectiveness estimates to real-world target populations is important for decision making. In the context of independent aggregate time-to-event baseline and relative effects data, complex hazards can make modeling of data for use in economic evaluation challenging. Our article provides an overview of methods that can be used to apply trial-derived relative treatment effects to external real-world baselines when faced with complex hazards and follows with a motivating example.

METHODS

Approaches for applying trial-derived relative effects to real-world baselines are presented in the context of complex hazards. Appropriate methods are applied in a cost-effectiveness analysis using data from a previously published study assessing the real-world cost-effectiveness of a treatment for carcinoma of the head and neck as a motivating example.

RESULTS

Lack of common hazards between the trial and target real-world population, a complex baseline hazard function, and nonproportional relative effects made the use of flexible models necessary to adequately estimate survival. Assuming common distributions between trial and real-world reference survival substantially affected survival and cost-effectiveness estimates. Modeling time-dependent vs proportional relative effects affected estimates to a lesser extent, dependent on assumptions used in cost-effectiveness modeling.

CONCLUSIONS

Appropriately capturing reference treatment survival when attempting to generalize trial-derived relative treatment effects to real-world target populations can have important impacts on cost-effectiveness estimates. A balance between model complexity and adequacy for decision making should be considered where multiple data sources with complex hazards are being evaluated.

Family-based pediatric weight management interventions in US primary care settings targeting children ages 6-12 years old: A systematic review guided by the RE-AIM framework

Obesity is a pandemic that disproportionately affects children from vulnerable populations in the USA. Current treatment approaches in primary care settings in the USA have been reported to be insufficient at managing pediatric obesity, primarily due to implementation challenges for healthcare systems and barriers for families. While the literature has examined the efficacy of pediatric obesity interventions focused on internal validity, it lacks sufficient reporting and analysis of external validity necessary for successful translation to primary care settings. We conducted a systematic review of the primary-care-setting literature from January 2007 to March 2020 on family-based pediatric weight management interventions in both English and/or Spanish for children ages 6-12 years in the USA using the Reach, Efficacy/Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework. A literature search, using PRISMA guidelines, was conducted in January 2022 using the following electronic databases: Medline Ovid, Embase, and Cochrane Library. 22 270 records were screened, and 376 articles were reviewed in full. 184 studies were included. The most commonly reported dimensions of the RE-AIM framework were Reach (65%), Efficacy/Effectiveness (64%), and Adoption (64%), while Implementation (47%) and Maintenance (42%) were less often reported. The prevalence of reporting RE-AIM construct indicators ranged greatly, from 1% to 100%. This systematic review underscores the need for more focus on external validity to guide the development, implementation, and dissemination of future pediatric obesity interventions based in primary care settings. It also suggests conducting additional research on sustainable financing for pediatric obesity interventions.

Conditional cross-design synthesis estimators for generalizability in Medicaid

While much of the causal inference literature has focused on addressing internal validity biases, both internal and external validity are necessary for unbiased estimates in a target population of interest. However, few generalizability approaches exist for estimating causal quantities in a target population that is not well-represented by a randomized study but is reflected when additionally incorporating observational data. To generalize to a target population represented by a union of these data, we propose a novel class of conditional cross-design synthesis estimators that combine randomized and observational data, while addressing their estimates' respective biases-lack of overlap and unmeasured confounding. These methods enable estimating the causal effect of managed care plans on health care spending among Medicaid beneficiaries in New York City, which requires obtaining estimates for the 7% of beneficiaries randomized to a plan and 93% who choose a plan, who do not resemble randomized beneficiaries. Our new estimators include outcome regression, propensity weighting, and double robust approaches. All use the covariate overlap between the randomized and observational data to remove potential unmeasured confounding bias. Applying these methods, we find substantial heterogeneity in spending effects across managed care plans. This has major implications for our understanding of Medicaid, where this heterogeneity has previously been hidden. Additionally, we demonstrate that unmeasured confounding rather than lack of overlap poses a larger concern in this setting.

Keeping track of reality: embedding visual memory in natural behaviour

Since immersive virtual reality (IVR) emerged as a research method in the 1980s, the focus has been on the similarities between IVR and actual reality. In this vein, it has been suggested that IVR methodology might fill the gap between laboratory studies and real life. IVR allows for high internal validity (i.e., a high degree of experimental control and experimental replicability), as well as high external validity by letting participants engage with the environment in an almost natural manner. Despite internal validity being crucial to experimental designs, external validity also matters in terms of the generalizability of results. In this paper, we first highlight and summarise the similarities and differences between IVR, desktop situations (both non-immersive VR and computer experiments), and reality. In the second step, we propose that IVR is a promising tool for visual memory research in terms of investigating the representation of visual information embedded in natural behaviour. We encourage researchers to carry out experiments on both two-dimensional computer screens and in immersive virtual environments to investigate visual memory and validate and replicate the findings. IVR is valuable because of its potential to improve theoretical understanding and increase the psychological relevance of the findings.

Real World Studies: What They Are and What They Are Not

Patients are filtered by rigorously defined study selection criteria for recruitment into research; this is necessary to improve signal detection, improve internal validity, reduce study-related risks, and meet ethical standards. Research patients are assessed and managed in ways that differ from usual practice. So, neither patients nor the treatment environment resembles everyday patients treated in everyday practice. This diminishes the generalizability of study findings; that is, their external validity. There is, therefore, an increasing trend to conduct "real-world studies." In this context, "real-world patients" are those who are not filtered by restrictive study selection criteria, and "real-world settings" are those in which patients are managed with few study-related guidelines and restrictions. The elephant in the room is that the glamour associated with such real-world studies is an illusion. This is because real-world patients in one real-world setting can differ widely from real-world patients in another real-world setting. So, even in real-world studies, we can only generalize study findings to the population from which the sample was drawn and the setting in which the sample was managed. As a final note, many assessments in research, such as computerized or pen-and-paper neuropsychological tests, are not real-world measures as are, for example, measures of activities of daily living or quality of life.

A New Way to Think About Internal and External Validity

Researchers have been concerned with internal and external validity for decades, and the discussion continues. The present proposal is that there are less important and more important senses in which one can interpret internal and external validity, and these can be integrated with a taxonomy that includes theoretical, auxiliary, statistical, and inferential assumptions. The integration sheds light on recent exchanges in the literature on validity and suggests that the vaunted internal-external validity trade-off is false for more important senses of internal and external validity; internal and external validity increase or decrease together when there is an emphasis on underlying theories. Finally, the integration implies the desirability of some changes in typical research advice and practice.

Mapping strategies towards improved external validity in preclinical translational research

INTRODUCTION

Translation is about successfully bringing findings from preclinical contexts into the clinic. This transfer is challenging as clinical trials frequently fail despite positive preclinical results. Limited robustness of preclinical research has been marked as one of the drivers of such failures. One suggested solution is to improve the external validity of in vitro and in vivo experiments via a suite of complementary strategies.

AREAS COVERED

In this review, the authors summarize the literature available on different strategies to improve external validity in in vivo, in vitro, or ex vivo experiments; systematic heterogenization; generalizability tests; and multi-batch and multicenter experiments. Articles that tested or discussed sources of variability in systematically heterogenized experiments were identified, and the most prevalent sources of variability are reviewed further. Special considerations in sample size planning, analysis options, and practical feasibility associated with each strategy are also reviewed.

EXPERT OPINION

The strategies reviewed differentially influence variation in experiments. Different research projects, with their unique goals, can leverage the strengths and limitations of each strategy. Applying a combination of these approaches in confirmatory stages of preclinical research putatively increases the chances of success in clinical studies.

Improving the Generalizability of Behavioral Science by Using Reality Checks: A Tool for Assessing Heterogeneity in Participants' Consumership of Study Stimuli

In attempting to draw bigger conclusions, researchers in psychology open their labs to more diverse groups of people. Yet even the most far-reaching theories must be tested with specific stimuli, materials, and methodology. To the extent that a study's stimuli are familiar beyond the lab to groups of people writ large, an experiment is said to have mundane realism-a type of external validity. We propose that an experiment's stimuli will vary in their relevance to each individual participant (such as how much they consume the stimuli outside the lab) and can be assessed using a tool: reality checks. We found that accounting for a study's mundane realism, at the individual level, significantly altered a study's results-which we found to be the case in testing well-established findings in psychology and behavioral economics. Our work suggests that measuring mundane realism (in addition to creating it) is a useful way of testing effects in psychology among the participants for whom the studies' scenarios and decisions will matter most outside of the lab.

Seminatural environments for rodent behavioral testing: a representative design improving animal welfare and enhancing replicability

The low replicability of scientific studies has become an important issue. One possible cause is low representativeness of the experimental design employed. Already in the 1950's, Egon Brunswick pointed out that experimental setups ideally should be based on a random sample of stimuli from the subjects' natural environment or at least include basic features of that environment. Only experimental designs satisfying this criterion, representative designs in Brunswikian terminology, can produce results generalizable beyond the procedure used and to situations outside the laboratory. Such external validity is crucial in preclinical drug studies, for example, and should be important for replicability in general. Popular experimental setups in rodent research on non-human animals, like the tail suspension test or the Geller-Seifter procedure, do not correspond to contexts likely to be encountered in the animals' habitat. Consequently, results obtained in this kind of procedures can be generalized neither to other procedures nor to contexts outside the laboratory. Furthermore, many traditional procedures are incompatible with current notions of animal welfare. An approximation to the natural social and physical context can be provided in the laboratory, in the form of a seminatural environment. In addition to satisfy the basic demands for a representative design, such environments offer a far higher level of animal welfare than the typical small cages. This perspective article will briefly discuss the basic principles of the generalizability of experimental results, the virtues of representative designs and the coincidence of enhanced scientific quality and animal welfare provided by this kind of design.

Generalizing treatment effects with incomplete covariates: Identifying assumptions and multiple imputation algorithms

We focus on the problem of generalizing a causal effect estimated on a randomized controlled trial (RCT) to a target population described by a set of covariates from observational data. Available methods such as inverse propensity sampling weighting are not designed to handle missing values, which are however common in both data sources. In addition to coupling the assumptions for causal effect identifiability and for the missing values mechanism and to defining appropriate estimation strategies, one difficulty is to consider the specific structure of the data with two sources and treatment and outcome only available in the RCT. We propose three multiple imputation strategies to handle missing values when generalizing treatment effects, each handling the multisource structure of the problem differently (separate imputation, joint imputation with fixed effect, joint imputation ignoring source information). As an alternative to multiple imputation, we also propose a direct estimation approach that treats incomplete covariates as semidiscrete variables. The multiple imputation strategies and the latter alternative rely on different sets of assumptions concerning the impact of missing values on identifiability. We discuss these assumptions and assess the methods through an extensive simulation study. This work is motivated by the analysis of a large registry of over 20,000 major trauma patients and an RCT studying the effect of tranexamic acid administration on mortality in major trauma patients admitted to intensive care units. The analysis illustrates how the missing values handling can impact the conclusion about the effect generalized from the RCT to the target population.

Using a Multi-Site RCT to Predict Impacts for a Single Site: Do Better Data and Methods Yield More Accurate Predictions?

Multi-site randomized controlled trials (RCTs) provide unbiased estimates of the average impact in the study sample. However, their ability to accurately predict the impact for individual sites outside the study sample, to inform local policy decisions, is largely unknown. To extend prior research on this question, we analyzed six multi-site RCTs and tested modern prediction methods-lasso regression and Bayesian Additive Regression Trees (BART)-using a wide range of moderator variables. The main study findings are that: (1) all of the methods yielded accurate impact predictions when the variation in impacts across sites was close to zero (as expected); (2) none of the methods yielded accurate impact predictions when the variation in impacts across sites was substantial; and (3) BART typically produced "less inaccurate" predictions than lasso regression or than the Sample Average Treatment Effect. These results raise concerns that when the impact of an intervention varies considerably across sites, statistical modelling using the data commonly collected by multi-site RCTs will be insufficient to explain the variation in impacts across sites and accurately predict impacts for individual sites.

Reducing bias in experimental ecology through directed acyclic graphs

Ecologists often rely on randomized control trials (RCTs) to quantify causal relationships in nature. Many of our foundational insights of ecological phenomena can be traced back to well‐designed experiments, and RCTs continue to provide valuable insights today. Although RCTs are often regarded as the “gold standard” for causal inference, it is important to recognize that they too rely on a set of causal assumptions that must be justified and met by the researcher to draw valid causal conclusions. We use key ecological examples to show how biases such as confounding, overcontrol, and collider bias can occur in experimental setups. In tandem, we highlight how such biases can be removed through the application of the structural causal model (SCM) framework. The SCM framework visualizes the causal structure of a system or process under study using directed acyclic graphs (DAGs) and subsequently applies a set of graphical rules to remove bias from both observational and experimental data. We show how DAGs can be applied across ecological experimental studies to ensure proper study design and statistical analysis, leading to more accurate causal estimates drawn from experimental data. Although causal conclusions drawn from RCTs are often taken at face value, ecologists are increasingly becoming aware that experimental approaches must be carefully designed and analyzed to avoid potential biases. By applying DAGs as a visual and conceptual tool, experimental ecologists can increasingly meet the causal assumptions required for valid causal inference.

Transporting monovalent rotavirus vaccine efficacy estimates to an external target population: a secondary analysis of data from a randomised controlled trial in Malawi

Oral rotavirus vaccine efficacy estimates from randomised controlled trials are highly variable across settings. Although the randomised study design increases the likelihood of internal validity of findings, results from trials may not always apply outside the context of the study due to differences between trial participants and the target population. Here, we used a weight-based method to transport results from a monovalent rotavirus vaccine clinical trial conducted in Malawi between 2005 and 2008 to a target population of all trial-eligible children in Malawi, represented by data from the 2015-2016 Malawi Demographic and Health Survey (DHS). We reweighted trial participants to reflect the population characteristics described by the Malawi DHS. Vaccine efficacy was estimated for 1008 trial participants after applying these weights such that they represented trial-eligible children in Malawi. We also conducted subgroup analyses to examine the heterogeneous treatment effects by stunting and tuberculosis vaccination status at enrolment. In the original trial, the estimates of one-year vaccine efficacy against severe rotavirus gastroenteritis and any-severity rotavirus gastroenteritis in Malawi were 49.2% (95% CI 15.6%-70.3%) and 32.1% (95% CI 2.5%-53.1%), respectively. After weighting trial participants to represent all trial-eligible children in Malawi, vaccine efficacy increased to 62.2% (95% CI 35.5%-79.0%) against severe rotavirus gastroenteritis and 38.9% (95% CI 11.4%-58.5%) against any-severity rotavirus gastroenteritis. Rotavirus vaccine efficacy may differ between trial participants and target populations when these two populations differ. Differences in tuberculosis vaccination status between the trial sample and DHS population contributed to varying trial and target population vaccine efficacy estimates.

Proxy Variables and the Generalizability of Study Results

When individuals self-select (or are selected) into a study based on factors that influence the outcome, conclusions may not generalize to the full population. To compensate for this, results may be adjusted, for example, by standardization on the set of common causes of participation and outcome. Although such standardization is useful in some contexts, the common causes of participation and outcome may in practice not be fully observed. Instead, the researcher may have access to one or several variables related to the common causes, that is, to proxies for the common causes. This article defines and examines different types of proxy variables and shows how these can be used to obtain generalizable study results. First of all, the researcher may exploit proxies that influence only participation or outcome but which still allow for perfect generalizability by rendering participation and outcome conditionally independent. Further, generalizability can be achieved by leveraging 2 proxies, one of which is allowed to influence participation and one of which is allowed to influence the outcome, even if participation and outcome do not become independent conditional on these. Finally, approximate generalizability may be obtained by exploiting a single proxy that does not itself influence participation or outcome.

Adherence to CONSORT Guidelines and Reporting of the Determinants of External Validity in Clinical Oncology Randomized Controlled Trials: A Review of Trials Published in Four Major Journals between 2013 and 2015

Our primary objective was to determine the proportion of trials that report the number of patients assessed for eligibility before randomization. We performed the systematic retrieval and analysis of all phase II, III, and IV RCTs published between 2013 and 2015 in four high-impact-factor journals in the field of clinical oncology. Among 456 RCTs reviewed, 236 trials (51.8%) reported the number of patients assessed for eligibility. Among the 236 trials that reported the entire enrollment process, the reasons for patient exclusion could be found in 184 trials (78%). A flow diagram was presented in 452 trials (99.1%), and 98 trials (21.5%) included a discussion on generalizability. Reporting the parameters of external validity in medical oncology RCTs is challenging. Improving adherence to the 2010 CONSORT guidelines concerning the enrollment process could help clinicians and health policymakers establish to whom trial results apply.

External validity of analogue sexual risk laboratory data

This article presents data on the external validity of an alcohol administration study of sexual decision-making in men who have sex with men (MSM) ages 21-50. Men (N = 135) randomized to alcohol (blood alcohol concentration [BAC] = .075%) or water control conditions reported intentions to engage in condomless anal intercourse (CAI) in response to video vignettes. Following the experiment participants provided 6 weeks of experience sampling method (ESM) data assessing intoxication, sexual arousal, partner relationship, and sexual behavior. Laboratory CAI intentions were hypothesized to predict future CAI behavior, and associations were hypothesized to be conditional upon sexual arousal and intoxication contextual factors as well as laboratory beverage condition. The hypotheses were partially supported. CAI intentions were correlated with subject proportions of days engaging in CAI (r = .29). A multilevel analysis indicated, on average, CAI intention predicted increased probability of CAI versus anal intercourse with a condom (relative risk ratio [RRR] = 1.43). There was mixed evidence of CAI intentions effects being conditional upon laboratory condition as well as arousal and intoxication contextual factors. Graphs of conditional marginal effects identified regions of significance. Effects of CAI intention for men in the alcohol condition on the CAI versus No Sex contrast were significant when sexual arousal was elevated. CAI intentions for men in the water control condition predicted a higher probability of CAI versus anal intercourse with a condom when intoxication was moderately elevated and/or arousal moderately low. The results support the external validity of alcohol administration experiments of sexual decision-making among MSM and, reciprocally, provide support for the validity of ESM assessment of sexual behavior. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Evaluative Conditioning: Past, Present, and Future

Evaluative conditioning (EC) research investigates changes in the evaluation of a stimulus after co-occurrence with an affective stimulus. To explain the motivation behind this research, this review begins with an overview of the history of EC research, followed by a summary of the state of the art with respect to three key questions. First, how should EC procedures be used to influence evaluation? We provide a guide based on evidence concerning the functional properties of EC effects. Second, how does the EC effect occur? We discuss the possible mediating cognitive processes and their automaticity. Third, are EC effects ubiquitous outside the lab? We discuss the evidence for the external validity of EC research. We conclude that the most important open questions pertain to the relevance of EC to everyday life and to the level of control that characterizes the processes that mediate the EC effect after people notice the stimulus co-occurrence.

Neuroimaging findings in preclinical amyotrophic lateral sclerosis models-How well do they mimic the clinical phenotype? A systematic review

Background and objectives

Animal models for motor neuron diseases (MND) such as amyotrophic lateral sclerosis (ALS) are commonly used in preclinical research. However, it is insufficiently understood how much findings from these model systems can be translated to humans. Thus, we aimed at systematically assessing the translational value of MND animal models to probe their external validity with regards to magnetic resonance imaging (MRI) features.

Methods

In a comprehensive literature search in PubMed and Embase, we retrieved 201 unique publications of which 34 were deemed eligible for qualitative synthesis including risk of bias assessment.

Results

ALS animal models can indeed present with human ALS neuroimaging features: Similar to the human paradigm, (regional) brain and spinal cord atrophy as well as signal changes in motor systems are commonly observed in ALS animal models. Blood-brain barrier breakdown seems to be more specific to ALS models, at least in the imaging domain. It is noteworthy that the G93A-SOD1 model, mimicking a rare clinical genotype, was the most frequently used ALS proxy.

Conclusions

Our systematic review provides high-grade evidence that preclinical ALS models indeed show imaging features highly reminiscent of human ALS assigning them a high external validity in this domain. This opposes the high attrition of drugs during bench-to-bedside translation and thus raises concerns that phenotypic reproducibility does not necessarily render an animal model appropriate for drug development. These findings emphasize a careful application of these model systems for ALS therapy development thereby benefiting refinement of animal experiments.

Systematic review registration

https://www.crd.york.ac.uk/PROSPERO/, identifier: CRD42022373146.

Lipid-Lowering Trials Are Not Representative of Patients Managed in Clinical Practice: A Systematic Review and Meta-Analysis of Exclusion Criteria

Background Randomized clinical trials (RCTs) might not be representative of the real-world population because of unreasonable exclusion criteria. We sought to determine which groups of patients are excluded from RCTs that included lipid-lowering therapy. Methods and Results We retrieved all trials from the Cholesterol Treatment Trialists Collaboration and systematically searched for large (≥1000 participants) lipid-lowering therapy RCTs, defined as statins, ezetimibe, and PCSK9 inhibitors. We predefined groups: older adults (>70 or >75 years), women, non-Whites, chronic kidney failure, heart failure, immunosuppression, cancer, dementia, treated thyroid disease, chronic obstructive pulmonary disease, mental illness, atrial fibrillation, multimorbidity (≥2 chronic diseases), and polypharmacy. We counted the number of RCTs excluding patients of the predefined groups and meta-analyzed the prevalence of included patients to obtain pooled estimates with a random-effects model. We included 42 RCTs (298 605 patients). Eighty-one percent of trials excluded patients with severe and 76% those with moderate kidney failure. Seventy-one percent of trials excluded groups of women, 64% excluded patients with moderate to severe heart failure, 64% those with immunosuppressant conditions, 48% those with cancer, 29% those with dementia, and 29% of trials excluded older adults. The pooled prevalence for patients >70 years of age was 25% (95% CI, 0%-49%), 11% (3%-18%) for >75 years of age, and 51% (38%-63%) for multimorbidity. Conclusions The majority of lipid-lowering therapy trials excluded patients with common diseases, such as moderate-to-severe kidney disease or heart failure or with immunosuppression. Underrepresenting certain populations, including women and older adults, might lead to limited transportability of study results and uncertainty on possible side-effects and efficacy in these groups. Future trials should promote diversity in the recruitment strategies and improve equity in cardiovascular research. Registration URL: ClinicalTrials.gov; Unique Identifier: CRD42021253909.

Evaluation of external validity of the OVHIPEC-1 trial in a real-world population

OBJECTIVE

The OVHIPEC-1 trial (Phase III randomised clinical trial for stage III ovarian carcinoma randomising between interval cytoreductive surgery with or without hyperthermic intraperitoneal chemotherapy) showed improved survival when interval cytoreductive surgery (CRS) was combined with hyperthermic intraperitoneal chemotherapy in patients with stage III epithelial ovarian cancer (EOC). The authors compared the control arm of the trial with a real-world population treated in the Netherlands during the same period to explore generalizability of the trial results.

METHODS

For this nationwide comparative cohort study, all patients with EOC undergoing interval CRS between 2007 and 2016 were identified from the Netherlands Cancer Registry if they fulfilled the eligibility criteria of OVHIPEC-1 (n = 1376). Patient and treatment characteristics, and overall survival (OS) were compared between trial and real-world populations.

RESULTS

Age, comorbidity, BRCA status, histologic subtype, and residual disease were similar in trial and real-world patients. Trial patients had a better performance status, higher socioeconomic status, and underwent bowel surgery more often. In a real-world setting, patients more often received more than six cycles. The difference in OS between the trial and the real-world populations was not statistically significant (unadjusted hazard ratio, 1.09 [95% confidence interval, 0.87-1.37]; P = 0.44).

CONCLUSION

Despite differences in patient characteristics, OS of patients treated in the control arm of OVHIPEC-1 was similar to patients treated outside the trial. The trial population accurately represents real-world patients with stage III EOC undergoing interval CRS in terms of outcome.

Asthma biologic trial eligibility and real-world outcomes in the United States

OBJECTIVE

To compare the outcomes of real-world patients who would have been eligible for asthma biologics to those who would not have been eligible.

METHODS

We used data from the OptumLabs Data Warehouse (OLDW) to categorize patients into eligible and ineligible groups based on clinical trials (n = 19 trials) used for Food and Drug Administration (FDA) approval. We then compared the change in the number of asthma exacerbations before and after biological initiation between the two groups.

RESULTS

The percentage of people who would have been eligible for asthma biologic clinical trials ranged from 0-10.2%. The eligible group had a greater reduction in number of asthma exacerbations compared to the ineligible group based on eligibility criteria from 1 omalizumab trial (1.52, 95% CI 1.25, 1.8 in eligible vs. 0.47, 95% CI 0.43, 0.52 in ineligible) and from 1 dupilumab trial (1.6, 95% CI 0.92, 2.28 in eligible vs. 0.52, 95% CI 0.38, 0.65 ineligible). Notably, 15 of the 19 trials had fewer than 11 eligible people, limiting additional comparisons.

CONCLUSIONS

Fewer than 1 in 10 people in the United States treated with asthma biologics would have been eligible to participate in the trial for the biologic they used. Where comparisons could be made, trial eligible people have a greater reduction in exacerbations.

Supplemental data for this article is available online at https://doi.org/10.1080/02770903.2021.2010749 .

Methodological issues when using face prototypes: A case study on the Faceaurus dataset

Prototype faces, created by averaging faces from several individuals sharing a common characteristic (for example a certain personality trait), can be used for highly informative experimental designs in face research. Although the facial prototype method is both ingenious and useful, we argue that its implementation is associated with three major issues: lack of external validity and non-independence of the units of information, both aggravated by a lack of transparency regarding the methods used and their limitations. Here, we describe these limitations and illustrate our claims with a systematic review of studies creating facial stimuli using the prototypes dataset 'Faceaurus'. We then propose some solutions that can eliminate or reduce these problems. We provide recommendations for future research employing this method on how to produce more generalisable and replicable results.

A guide for researchers seeking training in retrospective data harmonization for population neuroscience studies of Alzheimer's disease and related dementias

Due to needs surrounding rigor and reproducibility, subgroup specific disease knowledge, and questions of external validity, data harmonization is an essential tool in population neuroscience of Alzheimer's disease and related dementias (ADRD). Systematic harmonization of data elements is necessary to pool information from heterogeneous samples, and such pooling allows more expansive evaluations of health disparities, more precise effect estimates, and more opportunities to discover effective prevention or treatment strategies. The key goal of this Tutorial in Population Neuroimaging Curriculum, Instruction, and Pedagogy article is to guide researchers in creating a customized population neuroscience of ADRD harmonization training plan to fit their needs or those of their mentees. We provide brief guidance for retrospective data harmonization of multiple data types in this area, including: (1) clinical and demographic, (2) neuropsychological, and (3) neuroimaging data. Core competencies and skills are reviewed, and resources are provided to fill gaps in training as well as data needs. We close with an example study in which harmonization is a critical tool. While several aspects of this tutorial focus specifically on ADRD, the concepts and resources are likely to benefit population neuroscientists working in a range of research areas.

Using Pandemic Behavior to Test the External Validity of Laboratory Measurements of Risk Aversion and Guilt

We test whether laboratory measures of individual preferences for risk and guilt relate to risk-connected behaviors in a pandemic, such as socializing, dining in at restaurants, and hand washing. We utilize a survey administrated to a nationally representative subject pool in the United States in April, 2020 - the month following the declaration of a national state of emergency in response to the global outbreak of COVID-19. We find that higher levels of risk aversion are associated with risk-reducing behaviors during the COVID-19 pandemic. Meanwhile, we do not find strong evidence that guilt relates to the same behavior.

Variability in repeated economic games: comparing trust game decisions to other social trust measures

Economic games are well-established tools that offer a convenient approach to study social behaviour. Although widely used, recent evidence suggests that decisions made in the context of standard economic games are less predictive of real-world behaviour than previously assumed self-reported questionnaires. A possible explanation for this discrepancy is that economic games decisions in the laboratory are more likely to be influenced by the current situation, while questionnaires are specifically designed to measure people's average behaviour across a long period of time. To test this hypothesis, we performed a longitudinal study where 275 respondents played 16 Trust games every two days within a three-week period, and filled out a questionnaire that measures social trust. This study confirmed the instability of our measure of trust behaviour over time and the substantial stability of questionnaire responses. However, we found a significant association between self-reported social trust and participants' average behaviour in the trust game measured across sessions, but also with participants' behaviour measured only in Session 1. Nevertheless, analysis of behavioural changes in the Trust games over time revealed different behavioural profiles, highlighting how economic games and questionnaires can complement each other in the study of social trust.

Toward a Clearer Definition of Selection Bias When Estimating Causal Effects

Selection bias remains a subject of controversy. Existing definitions of selection bias are ambiguous. To improve communication and the conduct of epidemiologic research focused on estimating causal effects, we propose to unify the various existing definitions of selection bias in the literature by considering any bias away from the true causal effect in the referent population (the population before the selection process), due to selecting the sample from the referent population, as selection bias. Given this unified definition, selection bias can be further categorized into two broad types: type 1 selection bias owing to restricting to one or more level(s) of a collider (or a descendant of a collider) and type 2 selection bias owing to restricting to one or more level(s) of an effect measure modifier. To aid in explaining these two types-which can co-occur-we start by reviewing the concepts of the target population, the study sample, and the analytic sample. Then, we illustrate both types of selection bias using causal diagrams. In addition, we explore the differences between these two types of selection bias, and describe methods to minimize selection bias. Finally, we use an example of "M-bias" to demonstrate the advantage of classifying selection bias into these two types.

A Comparison of Individuals with Diabetes and EMPA-REG Trial Participants: Exploring Aspects of External Validity

BACKGROUND

There is increasing use of sodium glucose co-transporter 2 (SGLT2) inhibitors to treat diabetes. Since trials apply specific entry and exclusion criteria to ensure internal validity, comparisons of trial populations with nationally representative samples can inform the applicability of study findings to practice.

OBJECTIVE

To compare individuals with diabetes from a nationally representative sample to patients who underwent randomization in the EMPA-REG trial. A secondary aim was to characterize what proportion of individuals prescribed an SGLT2 inhibitor in a nationally representative sample would have been included in the EMPA-REG trial.

DESIGN

Retrospective cross-sectional study.

PARTICIPANTS

Adults with diabetes who took part in the National Health and Nutrition Examination Survey (NHANES) between 2011-2014 (primary analysis corresponding to EMPA-REG enrollment) and 2015-2018 (secondary analysis corresponding to contemporary sample).

MAIN MEASURES

The primary outcome was a comparison of demographic (age, sex, ethnicity, and pregnancy status), clinical (comorbidities and medication use), examination (weight, body mass index, and systolic and diastolic blood pressure), and laboratory (hgba1c, low- and high-density lipoprotein cholesterol, triglycerides, and estimated glomerular filtration rate) characteristics of NHANES respondents versus EMPA-REG trial participants. The secondary outcome was the proportion of NHANES respondents who had been prescribed an SGLT2 inhibitor that would have met inclusion criteria for the EMPA-REG trial.

KEY RESULTS

There were 655 and 48 respondents, representing a weighted sample of 21,849,775 and 1,062,573 individuals, included in the primary and secondary analyses, respectively. Overall, 7.6% (95% CI 4.8-10.6%) of 2011-2014 NHANES respondents would have met all EMPA-REG trial inclusion criteria. NHANES respondents and EMPA-REG participants differed across demographic, clinical, examination, and laboratory domains. Of NHANES respondents from 2015 to 2018 who were prescribed an SGLT2 inhibitor, 10.6% (95% CI <1-24.7%) would have met all inclusion criteria for the EMPA-REG trial.

CONCLUSIONS

The EMPA-REG population differed from a nationally representative sample, which could affect generalizability.

The challenges of assessing sample representativeness within community-based evaluations of parenting programmes delivered in England and Ireland

The findings from evaluations of parenting programmes can help inform policy and practice decisions, including how best to allocate scarce resources designed to support families. However, studies often fail to consider the extent to which the findings can be generalised to other settings or populations. One, yet unexplored solution, is to compare study findings and large-scale data sets including publicly available population data. The aims of this study were to assess the feasibility of assessing sample representativeness using publicly available data and to identify the challenges involved in considering the generalisability of study findings. Sociodemographic data from two community-based evaluations of parenting programmes conducted in England and Ireland between 2015 and 2018 were used in the study (N = 395 parent-infant dyads). The results indicated some differences between the trial samples and the wider population. However, it is difficult to reach definitive conclusions about these findings due to the limitations associated with using the comparative data sets. Our study revealed three key challenges, including: (1) how best to define and conceptualise representativeness; (2) the availability of comparative data sets; and (3) the quality of the available data. Our study suggests that there is a need for up-to-date, good-quality comparative data sets to allow for the assessment of representativeness. Further work is required to identify parameters for making claims about representativeness, specifically regarding the acceptable level of difference between the target population and the study sample. This is the first study to explore the feasibility of using publicly available population data in two jurisdictions, for the purpose of making judgements about the representativeness of the findings from parent programmes. It is hoped that our results will encourage further investigation around the reporting of trial external validity to enable effective decision-making at policy and practice level.

Psychometric Evaluation and External Validity of the Japanese Version of Lumbar Stiffness Disability Index

Introduction

Long fusion surgery for adult spinal deformity may restrict activities of daily living due to lumbar stiffness. While the Lumbar Stiffness Disability Index (LSDI) can help assess lumbar stiffness, in Asia the external validity of this questionnaire has not been sufficiently examined. We performed the psychometric evaluation and external validation of the Japanese version of the LSDI (LSDI-J).

Methods

Fifty consecutive patients (14 males and 36 females; mean age 70.6 years) who underwent lumbar fusion surgery at our institution a minimum of one year after surgery and who visited the outpatient clinic between April and May 2019, were surveyed using the LSDI-J. The mean number of fusion levels was 4.4. Cronbach's alpha coefficients were calculated for internal consistency, and the intraclass correlation coefficient (ICC) was calculated to evaluate reliability. External validity was assessed by comparisons with the Oswestry Disability Index (ODI), the Japanese Orthopaedic Association Back Pain Evaluation Questionnaire (JOABPEQ), and the lumbar range of motion (LROM) with LSDI-J scores.

Results

Cronbach's alpha coefficient was 0.652 overall, and 0.849 after excluding Question 10 due to a low response rate. The ICC was 0.824 overall and 0.851 after excluding Question 10. The correlation with the ODI was 0.684, and the correlation coefficients with each domain of the JOABPEQ ranged from -0.590 to -0.413, indicating moderate correlation. However, LROM and the LSDI-J were not correlated (r=-0.055, P=0.734).

Conclusions

The LSDI-J may not be suitable in Japan because there was no correlation with LROM, the most important factor for external validity. It may be necessary to investigate why the LSDI-J did not apply to the Japanese population in terms of lower limb function. Alternatively, a unique method may be needed to assess lumbar stiffness disability that is more suitable for actual clinical practice in Japan.

A Unified Framework on Generalizability of Clinical Prediction Models

To be useful, clinical prediction models (CPMs) must be generalizable to patients in new settings. Evaluating generalizability of CPMs helps identify spurious relationships in data, provides insights on when they fail, and thus, improves the explainability of the CPMs. There are discontinuities in concepts related to generalizability of CPMs in the clinical research and machine learning domains. Specifically, conventional statistical reasons to explain poor generalizability such as inadequate model development for the purposes of generalizability, differences in coding of predictors and outcome between development and external datasets, measurement error, inability to measure some predictors, and missing data, all have differing and often complementary treatments, in the two domains. Much of the current machine learning literature on generalizability of CPMs is in terms of dataset shift of which several types have been described. However, little research exists to synthesize concepts in the two domains. Bridging this conceptual discontinuity in the context of CPMs can facilitate systematic development of CPMs and evaluation of their sensitivity to factors that affect generalizability. We survey generalizability and dataset shift in CPMs from both the clinical research and machine learning perspectives, and describe a unifying framework to analyze generalizability of CPMs and to explain their sensitivity to factors affecting it. Our framework leads to a set of signaling statements that can be used to characterize differences between datasets in terms of factors that affect generalizability of the CPMs.

Validity of the Trait Emotional Intelligence Questionnaire (TEIQue) in a Brazilian Sample

The study of the relationship between reasoning and emotional processes is not new in Psychology. There are currently two main approaches to understanding the aspects related to these processes called emotional intelligence: the ability model and the trait model. This study focuses on the latter, analyzing the factor structure, reliability, and validity of the Trait Emotional Intelligence Questionnaire (TEIQue) in a Brazilian sample. 4314 adults with ages ranging from 18 to 60 years (M = 30.37; SD = 9.73) answered the TEIQue and other online instruments measuring emotional regulation, impulsiveness, alexithymia, loneliness, quality of life, positive and negative affect, personality traits, emotional perception, emotional understanding, and reasoning tests. The original four-factor structure of the TEIQue was replicated, Cronbach's alphas ranged from 0.60 to 0.89 for the facets, and from 0.76 to 0.90 for the factors and global score. The correlations followed theoretically expected directions, showing a stronger pattern for self-report measures than for performance tasks. Our results corroborated previous studies with the TEIQue, confirming the psychometric adequacy of the instrument in the Brazilian context. Future studies may focus on participants with lower levels of education and additional external criteria, such as career preferences, job performance, and health.

Subgroups in Late Adulthood Are Associated With Cognition and Wellbeing Later in Life

Objectives: In this study, we aim to discover whether there are valid subgroups in aging that are defined by modifiable factors and are determinant of clinically relevant outcomes regarding healthy aging.

Method: Data from interviews were collected in the Longitudinal Aging Study Amsterdam at two measurement occasions with a 3-year interval. Input for the analyses were seven well-known vulnerability and protective factors of healthy aging. By means of community detection, we tested whether we could distinguish subgroups in a sample of 1478 participants (T1-sample, aged 61–101 years). We tested both the external validity (T1) and predictive validity (T2) for wellbeing and subjective cognitive decline. Moreover, replicability and long-term stability were determined in 1186 participants (T2-sample, aged 61–101 years).

Results: Three similar subgroups were identified at T1 and T2. Subgroup A was characterized by high levels of education with personal vulnerabilities, subgroup B by being physically active with low support and low levels of education, and subgroup C by high levels of support with low levels of education. Subgroup C showed the lowest wellbeing and memory profile, both at T1 and T2. On most measures of wellbeing and memory, subgroups A and B did not differ from each other. At T2, the same number of subgroups was identified and subgroup profiles at T1 and T2 were practically identical. Per T1 subgroup 47–62% retained their membership at T2.

Discussion: We identified valid subgroups that replicate over time and differ on external variables at current and later measurement occasions. Individual change in subgroup membership over time shows that transitions to subgroups with better outcomes are possible.

Force-Velocity Profile of Competitive Kayakers: Evaluation of a Novel Single Kayak Stroke Test

The assessment of the force-velocity (F-V) profile in athletes may have important applications for training prescription, injury management, and fatigue monitoring. This study aimed to assess whether a novel single kayak stroke test (SKST) is able to provide the F-V relationship variables (maximum force, maximum velocity and maximum power) of competitive kayakers with acceptable reliability and external validity. Six female (age: 20.3 ± 3.7 years) and eight male (age: 20.8 ± 2.4 years) elite kayakers performed the SKST, bench press, bench pull, and short Wingate kayak test. The individual F-V relationships were highly linear [median r (range): left stroke = 0.986 (0.897 - 0.998); right stroke = 0.987 (0.971 - 0.999)]. The reliability of the F-V relationship parameters obtained during the SKST was high (within-session: CV ≤ 4.48% and ICC ≥ 0.93; between-session: CV ≤ 8.06% and ICC ≥ 0.65). The validity of the F-V relationship parameters obtained during the SKST was generally very high for maximum power (r range = 0.825 - 0.975), high for maximum force during both the bench press and the bench pull (r range = 0.751 - 0.831), and high or moderate for maximal velocity during the bench pull (r = 0.770 - 0.829) and the bench press (r = 0.355 - 0.471), respectively. The SKST can be considered a feasible procedure for testing the maximal upper-body muscle mechanical capacities of kayakers.

Effects of experience with access regimes on stewardship behaviors of small-scale fishers

Governance regimes that assign exclusive access to support collective action are increasingly promoted to manage common-pool resources under the premise that they foster environmental stewardship. However, experimental evidence linked to existing policies that support this premise is lacking. Overlapping access policies in small-scale fisheries provide a unique opportunity to test the effects of access regimes on users' stewardship behaviors. We performed a lab-in-the-field experiment to assess how fishers' previous experience with access regimes relates to compliance and peer enforcement (n = 120). Fishers' compliance and peer-enforcement decisions were compared in a common-pool-resource game. Treatments differed in framing to represent exclusive access and pseudo-open access regimes, both of which fishers face in real life. To contrast behavior in the game with real-life observations, we compared fishers' associations that have shown relatively high and low management performance under exclusive access policies. Compliance and peer enforcement were higher under the exclusive access treatment than under the pseudo-open access treatment only for fishers' associations with high management performance in real life. Behaviors in the game reflected differences between associations in real life. Our results support previous research on ocean governance by experimentally assessing the role of access regimes in determining users' stewardship and suggest potential mechanisms for stewardship internalization.

Continuous positive airway pressure and adverse cardiovascular events in obstructive sleep apnea: are participants of randomized trials representative of sleep clinic patients?

STUDY OBJECTIVES

Randomized controlled trials (RCTs) have shown no reduction in adverse cardiovascular (CV) events in patients randomized to continuous positive airway pressure (CPAP) therapy for obstructive sleep apnea (OSA). This study examined whether randomized study populations were representative of OSA patients attending a sleep clinic.

METHODS

Sleep clinic patients were 3,965 consecutive adults diagnosed with OSA by in-laboratory polysomnography from 2006 to 2010 at a tertiary hospital sleep clinic. Characteristics of these patients were compared with participants of five recent RCTs examining the effect of CPAP on adverse CV events in OSA. The percentage of patients with severe (apnea-hypopnea index, [AHI] ≥ 30 events/h) or any OSA (AHI ≥ 5 events/h) who met the eligibility criteria of each RCT was determined, and those criteria that excluded the most patients identified.

RESULTS

Compared to RCT participants, sleep clinic OSA patients were younger, sleepier, more likely to be female and less likely to have established CV disease. The percentage of patients with severe or any OSA who met the RCT eligibility criteria ranged from 1.2% to 20.9% and 0.8% to 21.9%, respectively. The eligibility criteria that excluded most patients were preexisting CV disease, symptoms of excessive sleepiness, nocturnal hypoxemia and co-morbidities.

CONCLUSIONS

A minority of sleep clinic patients diagnosed with OSA meet the eligibility criteria of RCTs of CPAP on adverse CV events in OSA. OSA populations in these RCTs differ considerably from typical sleep clinic OSA patients. This suggests that the findings of such OSA treatment-related RCTs are not generalizable to sleep clinic OSA patients.Randomized Intervention with Continuous Positive Airway Pressure in CAD and OSA (RICCADSA) trial, https://clinicaltrials.gov/ct2/show/NCT00519597, ClinicalTrials.gov number, NCT00519597.Usefulness of Nasal Continuous Positive Airway Pressure (CPAP) Treatment in Patients with a First Ever Stroke and Sleep Apnea Syndrome, https://clinicaltrials.gov/ct2/show/NCT00202501, ClinicalTrials.gov number, NCT00202501.Effect of Continuous Positive Airway Pressure (CPAP) on Hypertension and Cardiovascular Morbidity-Mortality in Patients with Sleep Apnea and no Daytime Sleepiness, https://clinicaltrials.gov/ct2/show/NCT00127348, ClinicalTrials.gov number, NCT00127348.Continuous Positive Airway Pressure (CPAP) in Patients with Acute Coronary Syndrome and Obstructive Sleep Apnea (OSA) (ISAACC), https://clinicaltrials.gov/ct2/show/NCT01335087, ClinicalTrials.gov number, NCT01335087.

Evaluating Psychometric Properties of the New Teachers' Perceptions of Collective Efficacy to Handle Bullying Scale (TCEB)

Collective efficacy is a promising theoretical construct that has been used to explain bullying rates in school. The development of school collective efficacy scales has increased in bullying research in recent years; however, gaps remain in measuring collective efficacy to handle bullying. This research assessed the psychometric properties of a new scale to evaluate collective efficacy against bullying. This first-order one-dimensional scale is called the teachers’ perceptions of collective efficacy to handle bullying (TCEB) scale. A sample of 804 Mexican primary teachers completed questionnaires. The sample was randomly split into two subsamples for calibration (n = 402) and cross-validation analysis (n = 402). The factor structure was supported by confirmatory factorial analysis. Measurement equivalence was confirmed by gender. The latent means differences showed no statistically significant differences by teachers’ gender. The TCEB correlation with school environment factors (e.g., principal support, school climate, and bullying) confirms the scale’s discriminant and concurrent validity. Our findings suggest that TCEB is a suitable instrument to assess teachers’ perceptions of collective efficacy to handle bullying, a construct that has proved to help predict a positive whole-school context and student bullying involvement.

Propensity score methods for merging observational and experimental datasets

We consider how to merge a limited amount of data from a randomized controlled trial (RCT) into a much larger set of data from an observational data base (ODB), to estimate an average causal treatment effect. Our methods are based on stratification. The strata are defined in terms of effect moderators as well as propensity scores estimated in the ODB. Data from the RCT are placed into the strata they would have occupied, had they been in the ODB instead. We assume that treatment differences are comparable in the two data sources. Our first "spiked-in" method simply inserts the RCT data into their corresponding ODB strata. We also consider a data-driven convex combination of the ODB and RCT treatment effect estimates within each stratum. Using the delta method and simulations, we identify a bias problem with the spiked-in estimator that is ameliorated by the convex combination estimator. We apply our methods to data from the Women's Health Initiative, a study of thousands of postmenopausal women which has both observational and experimental data on hormone therapy (HT). Using half of the RCT to define a gold standard, we find that a version of the spiked-in estimator yields lower-MSE estimates of the causal impact of HT on coronary heart disease than would be achieved using either a small RCT or the observational component on its own.

Impact of Amplification Efficiency Approaches on Telomere Length Measurement via Quantitative-Polymerase Chain Reaction

Background: Precise determination of amplification efficiency is critical for reliable conversion of within-sample changes in fluorescence occurring on a logarithmic scale to between-sample differences in DNA content occurring on a linear scale. This endeavor is especially challenging for the telomere length (TL) quantitative-PCR (qPCR) assay, where amplification efficiency can vary between reactions targeting telomeric repeats (T) and those targeting a single-copy gene (S) to calculate TL as the T/S ratio. Methods: We compared seven different approaches toward estimating amplification efficiency, including the standard-curve method utilized by the qPCR instrument software, and alternative approaches which estimate efficiency on a reaction-by-reaction basis using the stand-alone program LinRegPCR. After calculating T/S ratios using efficiency estimates from each approach (N = 363), we tested their relative performance on metrics of assay precision and correlates of external validity including chronological age (age range = 1-72 years), across tissues within-person (leukocyte-buccal), and between parents and offspring. Results: Estimated amplification efficiency for telomere reactions was significantly lower than estimates for single-copy gene reactions. Efficiency estimates for both reaction sets were significantly higher when estimated with the standard-curve method utilized by the qPCR instrument relative to estimates reconstructed during the log-linear phase with LinRegPCR. While estimates of single-copy gene efficiency reconstructed using LinRegPCR measured within 90% of perfect exponential doubling (E = 1.92), estimates generated using the standard-curve method were inflated beyond 100% (E = 2.10-2.12), indicating poor fidelity. Despite differences in raw value, TL measurements calculated with LinRegPCR efficiency estimates exhibited similar relationships with external validity correlates to measurements generated using the qPCR instrument software. Conclusion: Since methods to estimate amplification efficiency can vary across qPCR instruments, we suggest that future analyses empirically consider external methods of efficiency calculations such as LinRegPCR, and that already generated data be re-analyzed to glean possible improvements.

A Primer on Repeated Sitting Exposure and the Cardiovascular System: Considerations for Study Design, Analysis, Interpretation, and Translation

Sedentary behavior, particularly sitting, is ubiquitous in many contemporary societies. This is a major societal concern considering the evidence for a strong association between sitting behavior and cardiovascular disease morbidity and mortality. Unsurprisingly, leading public health agencies have begun to advocate “reduction” in sitting behavior. Though, the guidelines are typically vague and non-specific. The lack of specific guidelines for prolonged sitting is attributable to the absence of available evidence to facilitate guideline development. To inform policy, well-designed randomized controlled trials are required to test the efficacy of specific and translatable sitting reduction strategies. To guide the design of randomized controlled trials, this review postulates that several gaps in the literature first need to be filled. Following a general discussion of the importance of sitting behavior to contemporary societies, each of the following are discussed: (i) acute sitting exposure and systems physiology; (ii) recommendations for a systems physiology toolbox; (iii) study design considerations for acute sitting exposure; and (iv) translation of sitting-focused research.

External validity of randomized clinical trial studying preventing depressive symptoms following acute coronary syndrome

OBJECTIVE

The objective of the current study was to quantitatively explore aspects of external validity, both researcher's choices (eligibility) and patient's choices (consent), of a recently conducted clinical trial.

METHODS

A cohesive comparison between the MEDACIS trial (NCT02451293) database and a national quality and research database was conducted. Comparisons between both participants and nonconsenting patients (patient consent) and participants and noneligible patients (researcher selection) were performed. Comparisons of outcomes were depressive and anxiety symptoms, demographics, and somatic or psychiatric comorbidity.

RESULTS

Noneligible patients had significantly higher levels of depressive symptoms and anxiety and were older and more likely to suffer from unstable angina pectoris. Furthermore, noneligible patients were less likely to be married, had a lower educational level, used more medication, and had a higher frequency of comorbidity. Nonconsenting patients had significantly higher levels of depressive symptoms and anxiety and were older and more likely to be females compared to participants.

CONCLUSION

Significant differences were present between noneligible patients and participants; however, more troublingly significant differences were shown between nonconsenting patients and participants. The presence of depressive symptoms and anxiety has a significant impact on patients' willingness to give informed consent in clinical trials in cardiology with a focus on psychological outcomes.