The Kryptonite of Evidence-Based I–O Psychology

Study Registration for the Field of Prevention Science: Considering Options and Paths Forward

The practice of prospectively registering the details of intervention studies in a public database or registry is gaining momentum across disciplines as a strategy for increasing the transparency, credibility, and accessibility of study findings. In this article, we consider five registries that may be relevant for registration of intervention studies in the field of prevention science: ClinicalTrials.gov, the American Economic Association Registry of Randomized Controlled Trials (AEA RCT Registry), the Open Science Framework Preregistration (OSF Preregistration), the Registry for International Development Impact Evaluations (RIDIE), and the Registry of Efficacy and Effectiveness Studies (REES). We examine the five registries in terms of substantive focus, study designs, and contents of registry entries. We consider two paths forward for prospective registration of intervention studies in the field of prevention science: Path A: register all studies in ClinicalTrials.gov and Path B: allow individual researchers to select the registry with the "best fit." Lastly, we consider how the field might begin to establish norms around registration.

On reproducibility and replicability: Arguing for open science practices and methodological improvements at the South African Journal of Industrial Psychology

Problematisation: In recent years, psychology has been going through a crisis of sorts. Research methods and practices have come under increased scrutiny, with many issues identified as negatively contributing to low replicability and reproducibility of psychological research.Implications: As a consequence, researchers are increasingly called upon to overhaul and improve their research process. Various stakeholders within the scientific community are arguing for more openness and rigor within industrial and organisational (I-O) psychological research. A lack of transparency and openness further fuels criticisms as to the credibility and trustworthiness of I-O psychology which negatively affects the evidence-based practices which it supports. Furthermore, traditional gate-keepers such as grant agencies, professional societies and journals, are adapting their policies, reflecting an effort to curtail these trends.Purpose: The purpose of this opinion paper is, therefore, to stimulate an open dialogue with the South African Journal of Industrial Psychology (SAJIP) contributing authors, its editorial board and readership about the challenges associated with the replication crisis in psychology. Furthermore, it attempts to discuss how the identified issues affect I-O psychology and how these could be managed through open science practices and other structural improvements within the SAJIP.Recommendations: We enumerate several easily implementable open science practices, methodological improvements and editorial policy enhancements to enhance credibility and transparency within the SAJIP. Relying on these, we recommend changes to the current practices that can be taken up by researchers and the SAJIP to improve reproducibility and replicability in I-O psychological science.

Perceived Statistical Knowledge Level and Self-Reported Statistical Practice Among Academic Psychologists

Introduction: Publications arguing against the null hypothesis significance testing (NHST) procedure and in favor of good statistical practices have increased. The most frequently mentioned alternatives to NHST are effect size statistics (ES), confidence intervals (CIs), and meta-analyses. A recent survey conducted in Spain found that academic psychologists have poor knowledge about effect size statistics, confidence intervals, and graphic displays for meta-analyses, which might lead to a misinterpretation of the results. In addition, it also found that, although the use of ES is becoming generalized, the same thing is not true for CIs. Finally, academics with greater knowledge about ES statistics presented a profile closer to good statistical practice and research design. Our main purpose was to analyze the extension of these results to a different geographical area through a replication study.

Methods: For this purpose, we elaborated an on-line survey that included the same items as the original research, and we asked academic psychologists to indicate their level of knowledge about ES, their CIs, and meta-analyses, and how they use them. The sample consisted of 159 Italian academic psychologists (54.09% women, mean age of 47.65 years). The mean number of years in the position of professor was 12.90 (SD = 10.21).

Results: As in the original research, the results showed that, although the use of effect size estimates is becoming generalized, an under-reporting of CIs for ES persists. The most frequent ES statistics mentioned were Cohen's d and R²/η², which can have outliers or show non-normality or violate statistical assumptions. In addition, academics showed poor knowledge about meta-analytic displays (e.g., forest plot and funnel plot) and quality checklists for studies. Finally, academics with higher-level knowledge about ES statistics seem to have a profile closer to good statistical practices.

Conclusions: Changing statistical practice is not easy.This change requires statistical training programs for academics, both graduate and undergraduate.

The Influence of Judgment Calls on Meta-Analytic Findings

Previous research has suggested that judgment calls (i.e., methodological choices made in the process of conducting a meta-analysis) have a strong influence on meta-analytic findings and question their robustness. However, prior research applies case study comparison or reanalysis of a few meta-analyses with a focus on a few selected judgment calls. These studies neglect the fact that different judgment calls are related to each other and simultaneously influence the outcomes of a meta-analysis, and that meta-analytic findings can vary due to non-judgment call differences between meta-analyses (e.g., variations of effects over time). The current study analyzes the influence of 13 judgment calls in 176 meta-analyses in marketing research by applying a multivariate, multilevel meta-meta-analysis. The analysis considers simultaneous influences from different judgment calls on meta-analytic effect sizes and controls for alternative explanations based on non-judgment call differences between meta-analyses. The findings suggest that judgment calls have only a minor influence on meta-analytic findings, whereas non-judgment call differences between meta-analyses are more likely to explain differences in meta-analytic findings. The findings support the robustness of meta-analytic results and conclusions.

How Trustworthy Is the Scientific Literature in Industrial and Organizational Psychology?

The trustworthiness of research findings has been questioned in many domains of science. This article calls for a review of the trustworthiness of the scientific literature in industrial–organizational (I–O) psychology and a reconsideration of common practices that may harm the credibility of our literature. We note that most hypotheses in I–O psychology journals are confirmed. Thus, we are either approaching omniscience or our journals are publishing an unrepresentative sample of completed research. We view the latter explanation as more likely. We review structural problems in the publication process and in the conduct of research that is likely to promote a distortion of scientific knowledge. We then offer recommendations to make the I–O literature more accurate and trustworthy.

Evidence-Based I–O Psychology: Not There Yet but Now a Little Nearer?

Our focal article sought to promote discussion of evidence-based approaches to practice in industrial–organizational (I–O) psychology. It did so by describing the meanings and origins of evidence-based practice, evaluating the extent to which I–O psychology practice is currently evidence-based, and considering the role of systematic reviews in promoting evidence-based practice. The commentaries on our focal article raised many interesting and important points. In our response, we divide them into two broad categories. The first category consists of comments and objections that arise from what we believe to be misinterpretations of evidence-based practice and our focal article. The second category contains those comments that in various ways extend and elaborate the issues raised in our focal article. Although we are not there yet, we hope that these commentaries will take us a little nearer to an evidence-based approach to I–O psychology.

Top Management Team Diversity

Empirical research investigating the impact of top management team (TMT) diversity on executives’ decision making has produced inconclusive results. To synthesize and aggregate the results on the diversity-performance link, a meta-regression analysis (MRA) is conducted. It integrates more than 200 estimates from 53 empirical studies investigating TMT diversity and its impact on the quality of executives’ decision making as reflected in corporate performance. The analysis contributes to the literature by theoretically discussing and empirically examining the effects of TMT diversity on corporate performance. Our results do not show a link between TMT diversity and performance but provide evidence for publication bias. Thus, the findings raise doubts on the impact of TMT diversity on performance.

Publication bias and the failure of replication in experimental psychology

Replication of empirical findings plays a fundamental role in science. Among experimental psychologists, successful replication enhances belief in a finding, while a failure to replicate is often interpreted to mean that one of the experiments is flawed. This view is wrong. Because experimental psychology uses statistics, empirical findings should appear with predictable probabilities. In a misguided effort to demonstrate successful replication of empirical findings and avoid failures to replicate, experimental psychologists sometimes report too many positive results. Rather than strengthen confidence in an effect, too much successful replication actually indicates publication bias, which invalidates entire sets of experimental findings. Researchers cannot judge the validity of a set of biased experiments because the experiment set may consist entirely of type I errors. This article shows how an investigation of the effect sizes from reported experiments can test for publication bias by looking for too much successful replication. Simulated experiments demonstrate that the publication bias test is able to discriminate biased experiment sets from unbiased experiment sets, but it is conservative about reporting bias. The test is then applied to several studies of prominent phenomena that highlight how publication bias contaminates some findings in experimental psychology. Additional simulated experiments demonstrate that using Bayesian methods of data analysis can reduce (and in some cases, eliminate) the occurrence of publication bias. Such methods should be part of a systematic process to remove publication bias from experimental psychology and reinstate the important role of replication as a final arbiter of scientific findings.