Four Internal Inconsistencies in Tversky and Kahneman’s (1992) Cumulative Prospect Theory Article: A Case Study in Ambiguous Theoretical Scope and Ambiguous Parsimony

Scholars heavily rely on theoretical scope as a tool to challenge existing theory. We advocate that scientific discovery could be accelerated if far more effort were invested into also overtly specifying and painstakingly delineating the intended purview of any proposed new theory at the time of its inception. As a case study, we consider Tversky and Kahneman (1992). They motivated their Nobel-Prize-winning cumulative prospect theory with evidence that in each of two studies, roughly half of the participants violated independence, a property required by expected utility theory (EUT). Yet even at the time of inception, new theories may reveal signs of their own limited scope. For example, we show that Tversky and Kahneman’s findings in their own test of loss aversion provide evidence that at least half of their participants violated their theory, in turn, in that study. We highlight a combination of conflicting findings in the original article that make it ambiguous to evaluate both cumulative prospect theory’s scope and its parsimony on the authors’ own evidence. The Tversky and Kahneman article is illustrative of a social and behavioral research culture in which theoretical scope plays an extremely asymmetric role: to call existing theory into question and motivate surrogate proposals.

How Can Big Data Science Transform the Psychological Sciences?

Big data and related technologies are radically altering our society. In a similar way, these approaches can transform the psychological sciences. The goal of this commentary is to motivate psychologists to embrace big data science for the betterment of the field. Big data sources, algorithmic methods, and a culture that embraces prediction has the potential to advance our science, improve the robustness and replicability of our research, and allow us to focus more centrally on actual behaviors. We highlight these key transformations, acknowledge criticisms of big data approaches, and emphasize specific ways psychologists can contribute to the big data science revolution.

[Differences in the success rates of inpatient therapy for alcohol and illegal drug abusers. A Replication Study]

INTRODUCTION

Irrespective of the type of psychotherapy used, the abstinence-oriented treatment of drug abusers is less successful than that for alcohol abusers. If, on the other hand, the two groups are parallelized in such a way that the patients are identical with respect to the five characteristics of gender, age, schooling, work situation and partner situation, then there is no difference between the success rates of the drug and alcohol abusers. The aim of this study is to determine whether this result can be replicated in another therapeutic institution.

METHOD

Retrospective field study of 320 abusers of illegal drugs and 320 alcohol abusers who were treated with behaviour therapy. By combining the binary characteristics gender, work situation and age, the drug-dependent patients were divided into 2³=8 groups, and the same number of alcohol abusers were randomly selected for each group. The scheduled period of inpatient treatment was 90 days for the alcohol abusers and 120 days for the drug abusers. Every week the patients had one session of individual psychotherapy and four to five group therapy sessions. According to the indications, the certified behaviour therapists implemented the following interventions including behaviour analysis, relapse prevention, cognitive therapy, self-management and behavioural family therapy. Comparison of the success rates was carried out using the Chi² test, and changes in the psychological findings were tested with one-way variance analysis.

RESULTS

There was no difference between drug and alcohol abusers with respect to the rate of therapy termination according to plan (around 80%). A total of 48% of the drug abusers and 41 % of the alcohol abusers who could be followed up had been continuously abstinent at the one-year catamnesis without a single relapse. There were also no differences between the two groups when it was assumed that the patients who could not be followed up had relapsed. In the case of both the drug and alcohol abusers the abstinence rate was highest in over-29-year-old employed men (57.6%; 48.4%). The abstinence rate was lowest in employed female drug abusers (27.8%) and young, unemployed female drug abusers (0%, n=11).

DISCUSSION

What appears to influence the abstinence rate after inpatient treatment is not only the type of substance consumed but also sociodemographic characteristics. In addition to individually tailored therapy, our results confirm the importance of a highly differentiated presentation of the outcomes of therapy in the specialist literature. An average rate of abstinence (e.g. 30%) is insufficient to evaluate an intervention unless information is also provided about the patients for which the intervention is suitable and those for which it is not. In accordance with the Reproducibility Project, we consider replication studies essential in psychotherapy, even though in practice the considerable methodical requirements can only be partially fulfilled.

Quantitative Data From Rating Scales: An Epistemological and Methodological Enquiry

Rating scales are popular methods for generating quantitative data directly by persons rather than automated technologies. But scholars increasingly challenge their foundations. This article contributes epistemological and methodological analyses of the processes involved in person-generated quantification. They are crucial for measurement because data analyses can reveal information about study phenomena only if relevant properties were encoded systematically in the data. The Transdisciplinary Philosophy-of-Science Paradigm for Research on Individuals (TPS-Paradigm) is applied to explore psychological and social-science concepts of measurement and quantification, including representational measurement theory, psychometric theories and their precursors in psychophysics. These are compared to theories from metrology specifying object-dependence of measurement processes and subject-independence of outcomes as key criteria, which allow tracing data to the instances measured and the ways they were quantified. Separate histories notwithstanding, the article's basic premise is that general principles of scientific measurement and quantification should apply to all sciences. It elaborates principles by which these metrological criteria can be implemented also in psychology and social sciences, while considering their research objects' peculiarities. Application of these principles is illustrated by quantifications of individual-specific behaviors ('personality'). The demands rating methods impose on data-generating persons are deconstructed and compared with the demands involved in other quantitative methods (e.g., ethological observations). These analyses highlight problematic requirements for raters. Rating methods sufficiently specify neither the empirical study phenomena nor the symbolic systems used as data nor rules of assignment between them. Instead, pronounced individual differences in raters' interpretation and use of items and scales indicate considerable subjectivity in data generation. Together with recoding scale categories into numbers, this introduces a twofold break in the traceability of rating data, compromising interpretability of findings. These insights question common reliability and validity concepts for ratings and provide novel explanations for replicability problems. Specifically, rating methods standardize only data formats but not the actual data generation. Measurement requires data generation processes to be adapted to the study phenomena's properties and the measurement-executing persons' abilities and interpretations, rather than to numerical outcome formats facilitating statistical analyses. Researchers must finally investigate how people actually generate ratings to specify the representational systems underlying rating data.

Selbstwirksamkeitserwartung, psychische Belastung und Therapieerfahrung als Prädiktoren der Abstinenz nach stationärer Behandlung – eine Replikationsstudie

Zusammenfassung. Fragestellung: Gestatten die Merkmale „Anzahl vorheriger Entgiftungen“, „Depressivität“ und „Selbstwirksamkeitserwartung, SWE“ eine Prognose der Abstinenz zur Ein-Jahres Katamnese? Methode: Prospektive Replikations-Feldstudie in einer anderen Klinik, in der mittels binärer logistischer Regression und Chi Quadrat Tests Unterschiede in Patienten-Merkmalen zwischen zur Ein-Jahres-Katamnese durchgehend abstinenten (N = 285) vs. rückfällig gewordenen Alkoholabhängigen (N = 274) analysiert wurden. Ergebnisse: Ebenso wie in unserer vorherigen Studie waren Alter, Geschlecht, Schulbildung, Arbeitslosigkeit, Familienstand, Partnersituation, suchtspezifische und psychische Komorbidität prognostisch nicht relevant, Ausnahme Persönlichkeitsstörungen. Wiederum hatten Patienten mit weniger als zwei Entgiftungen und einer hohen SWE die höchste Wahrscheinlichkeit ein Jahr durchgehend abstinent zu leben (82 %). Ebenso bestätigt wurde, dass Verbesserungen in der psychischen Belastung nicht mit Abstinenz korrelieren. Depressivität und vorherige Entwöhnungsbehandlungen wurden nicht als Prädiktoren repliziert. Schlussfolgerungen: Vorherige Entgiftungen, SWE und Persönlichkeitsstörungen könnten für den hier untersuchten Kliniktyp allgemeinverbindliche Prädiktoren sein. Inwiefern der Ausschluss der beiden Prädiktoren Depressivität und vorherige Entwöhnungsbehandlungen eine Folge neu eingeführter Interventionen für Depressive und Therapieerfahrene ist, wäre zu prüfen. Wie in unserer vorherigen Studie gestattet das Regressionsmodell trotz geringer Varianzaufklärung und mittlerer Effektstärken die Ableitung kausaler Hypothesen zur klinikspezifischen Verbesserung der Behandlung. Replikationsstudien sollten ebenso wie die empirische Orientierung ein fester Bestandteil verhaltenstherapeutischer Behandlung sein.

The problem of false positives and false negatives in violent video game experiments

The problem of false positives and negatives has received considerable attention in behavioral research in recent years. The current paper uses video game violence research as an example of how such issues may develop in a field. Despite decades of research, evidence on whether violent video games (VVGs) contribute to aggression in players has remained mixed. Concerns have been raised in recent years that experiments regarding VVGs may suffer from both "false positives" and "false negatives." The current paper examines this issue in three sets of video game experiments, two sets of video game experiments on aggression and prosocial behaviors identified in meta-analysis, and a third group of recent null studies. Results indicated that studies of VVGs and aggression appear to be particularly prone to false positive results. Studies of VVGs and prosocial behavior, by contrast are heterogeneous and did not demonstrate any indication of false positive results. However, their heterogeneous nature made it difficult to base solid conclusions on them. By contrast, evidence for false negatives in null studies was limited, and little evidence emerged that null studies lacked power in comparison those highlighted in past meta-analyses as evidence for effects. These results are considered in light of issues related to false positives and negatives in behavioral science more broadly.

Optimizing Research Payoff

In this article, we present a model for determining how total research payoff depends on researchers' choices of sample sizes, α levels, and other parameters of the research process. The model can be used to quantify various trade-offs inherent in the research process and thus to balance competing goals, such as (a) maximizing both the number of studies carried out and also the statistical power of each study, (b) minimizing the rates of both false positive and false negative findings, and (c) maximizing both replicability and research efficiency. Given certain necessary information about a research area, the model can be used to determine the optimal values of sample size, statistical power, rate of false positives, rate of false negatives, and replicability, such that overall research payoff is maximized. More specifically, the model shows how the optimal values of these quantities depend upon the size and frequency of true effects within the area, as well as the individual payoffs associated with particular study outcomes. The model is particularly relevant within current discussions of how to optimize the productivity of scientific research, because it shows which aspects of a research area must be considered and how these aspects combine to determine total research payoff.

The what, why, and how of born-open data

Although many researchers agree that scientific data should be open to scrutiny to ferret out poor analyses and outright fraud, most raw data sets are not available on demand. There are many reasons researchers do not open their data, and one is technical. It is often time consuming to prepare and archive data. In response, my laboratory has automated the process such that our data are archived the night they are created without any human approval or action. All data are versioned, logged, time stamped, and uploaded including aborted runs and data from pilot subjects. The archive is GitHub, github.com, the world's largest collection of open-source materials. Data archived in this manner are called born open. In this paper, I discuss the benefits of born-open data and provide a brief technical overview of the process. I also address some of the common concerns about opening data before publication.

Interpreting confidence intervals: A comment on Hoekstra, Morey, Rouder, and Wagenmakers (2014)

Hoekstra, Morey, Rouder, and Wagenmakers (Psychonomic Bulletin & Review 21(5), 1157-1164 2014) reported the results of a questionnaire designed to assess students' and researchers' understanding of confidence intervals (CIs). They interpreted their results as evidence that these groups "have no reliable knowledge about the correct interpretation of CIs" (Hoekstra et al. Psychonomic Bulletin & Review 21(5), 1157-1164 2014, p. 1161). We argue that their data do not substantiate this conclusion and that their report includes misleading suggestions about the correct interpretations of confidence intervals.

Age and Employee Green Behaviors: A Meta-Analysis

Recent economic and societal developments have led to an increasing emphasis on organizational environmental performance. At the same time, demographic trends are resulting in increasingly aging labor forces in many industrialized nations. Commonly held stereotypes suggest that older workers are less likely to be environmentally responsible than younger workers. To evaluate the degree to which such age differences are present, we meta-analyzed 132 independent correlations and 336 d-values based on 4676 professional workers from 22 samples in 11 countries. Contrary to popular stereotypes, age showed small positive relationships with pro-environmental behaviors, suggesting that older adults engaged in these workplace behaviors slightly more frequently. Relationships with age appeared to be linear for overall, Conserving, Avoiding Harm, and Taking Initiative pro-environmental behaviors, but non-linear trends were observed for Transforming and Influencing Others behaviors.

Evaluating replicability of laboratory experiments in economics

The replicability of some scientific findings has recently been called into question. To contribute data about replicability in economics, we replicated 18 studies published in the American Economic Review and the Quarterly Journal of Economics between 2011 and 2014. All of these replications followed predefined analysis plans that were made publicly available beforehand, and they all have a statistical power of at least 90% to detect the original effect size at the 5% significance level. We found a significant effect in the same direction as in the original study for 11 replications (61%); on average, the replicated effect size is 66% of the original. The replicability rate varies between 67% and 78% for four additional replicability indicators, including a prediction market measure of peer beliefs.

Playing with Data--Or How to Discourage Questionable Research Practices and Stimulate Researchers to Do Things Right

Recent fraud cases in psychological and medical research have emphasized the need to pay attention to Questionable Research Practices (QRPs). Deliberate or not, QRPs usually have a deteriorating effect on the quality and the credibility of research results. QRPs must be revealed but prevention of QRPs is more important than detection. I suggest two policy measures that I expect to be effective in improving the quality of psychological research. First, the research data and the research materials should be made publicly available so as to allow verification. Second, researchers should more readily consider consulting a methodologist or a statistician. These two measures are simple but run against common practice to keep data to oneself and overestimate one's methodological and statistical skills, thus allowing secrecy and errors to enter research practice.

Using prediction markets to estimate the reproducibility of scientific research

Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications.

Replicating research in ecology and evolution: feasibility, incentives, and the cost-benefit conundrum

We believe that replicating studies in ecology and evolution is extremely valuable, but replication within species and systems is troublingly rare, and even ‘quasi-replications’ in different systems are often insufficient. We make a case for supporting multiple types of replications and point out that the current incentive structure needs to change if ecologists and evolutionary biologist are to value scientific replication sufficiently.

Randomization Does Not Help Much, Comparability Does

According to R.A. Fisher, randomization "relieves the experimenter from the anxiety of considering innumerable causes by which the data may be disturbed." Since, in particular, it is said to control for known and unknown nuisance factors that may considerably challenge the validity of a result, it has become very popular. This contribution challenges the received view. First, looking for quantitative support, we study a number of straightforward, mathematically simple models. They all demonstrate that the optimism surrounding randomization is questionable: In small to medium-sized samples, random allocation of units to treatments typically yields a considerable imbalance between the groups, i.e., confounding due to randomization is the rule rather than the exception. In the second part of this contribution, the reasoning is extended to a number of traditional arguments in favour of randomization. This discussion is rather non-technical, and sometimes touches on the rather fundamental Frequentist/Bayesian debate. However, the result of this analysis turns out to be quite similar: While the contribution of randomization remains doubtful, comparability contributes much to a compelling conclusion. Summing up, classical experimentation based on sound background theory and the systematic construction of exchangeable groups seems to be advisable.

Replications in Psychology Research: How Often Do They Really Occur?

Recent controversies in psychology have spurred conversations about the nature and quality of psychological research. One topic receiving substantial attention is the role of replication in psychological science. Using the complete publication history of the 100 psychology journals with the highest 5-year impact factors, the current article provides an overview of replications in psychological research since 1900. This investigation revealed that roughly 1.6% of all psychology publications used the term replication in text. A more thorough analysis of 500 randomly selected articles revealed that only 68% of articles using the term replication were actual replications, resulting in an overall replication rate of 1.07%. Contrary to previous findings in other fields, this study found that the majority of replications in psychology journals reported similar findings to their original studies (i.e., they were successful replications). However, replications were significantly less likely to be successful when there was no overlap in authorship between the original and replicating articles. Moreover, despite numerous systemic biases, the rate at which replications are being published has increased in recent decades.

Comparing Happiness and Hypomania Risk: A Study of Extraversion and Neuroticism Aspects

Positive affect has long been considered a hallmark of subjective happiness. Yet, high levels of positive affect have also been linked with hypomania risk: a set of cognitive, affective, and behavioral characteristics that constitute a dispositional risk for future episodes of hypomania and mania. At a personality level, two powerful predictors of affective experience are extraversion and neuroticism: extraversion has been linked to positive affect, and neuroticism to negative affect. As such, a single personality trait--extraversion--has been linked to both beneficial and harmful outcomes associated with positivity. It is clear that positive affect, in different forms, has divergent consequences for well-being, but previous research has struggled to articulate the nature of these differences. We suggest that the relationship between affect and well-being needs to be situated within the psychological context of the individual--both in terms of more specific forms of extraversion and neuroticism, but also in terms of interactions among personality aspects. Consistent with this idea, we found that two aspects of extraversion (enthusiasm and assertiveness) differentially predicted subjective happiness from hypomania risk and two aspects of neuroticism (volatility and withdrawal) interacted to predict hypomania risk: the highest levels of hypomania risk were associated with the combination of high volatility and low withdrawal. These findings underscore the importance of examining personality at the right level of resolution to understand well-being and dysfunction.

Negative affect mediates the relationship between interpersonal problems and binge-eating disorder symptoms and psychopathology in a clinical sample: a test of the interpersonal model

This study evaluated the validity of the interpersonal model of binge-eating disorder (BED) psychopathology in a clinical sample of women with BED. Data from a cross-sectional sample of 255 women with BED were examined for the direct effects of interpersonal problems on BED symptoms and psychopathology, and indirect effects mediated by negative affect. Structural equation modelling analyses demonstrated that higher levels of interpersonal problems were associated with greater negative affect, and greater negative affect was associated with higher frequency of BED symptoms and psychopathology. There was a significant indirect effect of interpersonal problems on BED symptoms and psychopathology mediated through negative affect. Interpersonal problems may lead to greater BED symptoms and psychopathology, and this relationship may be partially explained by elevated negative affect. The results of the study are the first to provide support for the interpersonal model of BED symptoms and psychopathology in a clinical sample of women.

Data management in astrobiology: challenges and opportunities for an interdisciplinary community

Data management and sharing are growing concerns for scientists and funding organizations throughout the world. Funding organizations are implementing requirements for data management plans, while scientists are establishing new infrastructures for data sharing. One of the difficulties is sharing data among a diverse set of research disciplines. Astrobiology is a unique community of researchers, containing over 110 different disciplines. The current study reports the results of a survey of data management practices among scientists involved in the astrobiology community and the NASA Astrobiology Institute (NAI) in particular. The survey was administered over a 2-month period in the first half of 2013. Fifteen percent of the NAI community responded (n=114), and additional (n=80) responses were collected from members of an astrobiology Listserv. The results of the survey show that the astrobiology community shares many of the same concerns for data sharing as other groups. The benefits of data sharing are acknowledged by many respondents, but barriers to data sharing remain, including lack of acknowledgement, citation, time, and institutional rewards. Overcoming technical, institutional, and social barriers to data sharing will be a challenge into the future.

The place of human psychophysics in modern neuroscience

Human psychophysics is the quantitative measurement of our own perceptions. In essence, it is simply a more sophisticated version of what humans have done since time immemorial: noticed and reflected upon what we can see, hear, and feel. In the 21st century, when hugely powerful techniques are available that enable us to probe the innermost structure and function of nervous systems, is human psychophysics still relevant? I argue that it is, and that in combination with other techniques, it will continue to be a key part of neuroscience for the foreseeable future. I discuss these points in detail using the example of binocular stereopsis, where human psychophysics in combination with physiology and computational vision, has made a substantial contribution.

Beat processing is pre-attentive for metrically simple rhythms with clear accents: an ERP study

The perception of a regular beat is fundamental to music processing. Here we examine whether the detection of a regular beat is pre-attentive for metrically simple, acoustically varying stimuli using the mismatch negativity (MMN), an ERP response elicited by violations of acoustic regularity irrespective of whether subjects are attending to the stimuli. Both musicians and non-musicians were presented with a varying rhythm with a clear accent structure in which occasionally a sound was omitted. We compared the MMN response to the omission of identical sounds in different metrical positions. Most importantly, we found that omissions in strong metrical positions, on the beat, elicited higher amplitude MMN responses than omissions in weak metrical positions, not on the beat. This suggests that the detection of a beat is pre-attentive when highly beat inducing stimuli are used. No effects of musical expertise were found. Our results suggest that for metrically simple rhythms with clear accents beat processing does not require attention or musical expertise. In addition, we discuss how the use of acoustically varying stimuli may influence ERP results when studying beat processing.

Continuously Cumulating Meta-Analysis and Replicability

The current crisis in scientific psychology about whether our findings are irreproducible was presaged years ago by Tversky and Kahneman (1971), who noted that even sophisticated researchers believe in the fallacious Law of Small Numbers—erroneous intuitions about how imprecisely sample data reflect population phenomena. Combined with the low power of most current work, this often leads to the use of misleading criteria about whether an effect has replicated. Rosenthal (1990) suggested more appropriate criteria, here labeled the continuously cumulating meta-analytic (CCMA) approach. For example, a CCMA analysis on a replication attempt that does not reach significance might nonetheless provide more, not less, evidence that the effect is real. Alternatively, measures of heterogeneity might show that two studies that differ in whether they are significant might have only trivially different effect sizes. We present a nontechnical introduction to the CCMA framework (referencing relevant software), and then explain how it can be used to address aspects of replicability or more generally to assess quantitative evidence from numerous studies. We then present some examples and simulation results using the CCMA approach that show how the combination of evidence can yield improved results over the consideration of single studies.

Why it is hard to find genes associated with social science traits: theoretical and empirical considerations

OBJECTIVES

We explain why traits of interest to behavioral scientists may have a genetic architecture featuring hundreds or thousands of loci with tiny individual effects rather than a few with large effects and why such an architecture makes it difficult to find robust associations between traits and genes.

METHODS

We conducted a genome-wide association study at 2 sites, Harvard University and Union College, measuring more than 100 physical and behavioral traits with a sample size typical of candidate gene studies. We evaluated predictions that alleles with large effect sizes would be rare and most traits of interest to social science are likely characterized by a lack of strong directional selection. We also carried out a theoretical analysis of the genetic architecture of traits based on R.A. Fisher's geometric model of natural selection and empirical analyses of the effects of selection bias and phenotype measurement stability on the results of genetic association studies.

RESULTS

Although we replicated several known genetic associations with physical traits, we found only 2 associations with behavioral traits that met the nominal genome-wide significance threshold, indicating that physical and behavioral traits are mainly affected by numerous genes with small effects.

CONCLUSIONS

The challenge for social science genomics is the likelihood that genes are connected to behavioral variation by lengthy, nonlinear, interactive causal chains, and unraveling these chains requires allying with personal genomics to take advantage of the potential for large sample sizes as well as continuing with traditional epidemiological studies.

PsychDisclosure.org

There is currently an unprecedented level of doubt regarding the reliability of research findings in psychology. Many recommendations have been made to improve the current situation. In this article, we report results from PsychDisclosure.org , a novel open-science initiative that provides a platform for authors of recently published articles to disclose four methodological design specification details that are not required to be disclosed under current reporting standards but that are critical for accurate interpretation and evaluation of reported findings. Grassroots sentiment—as manifested in the positive and appreciative response to our initiative—indicates that psychologists want to see changes made at the systemic level regarding disclosure of such methodological details. Almost 50% of contacted researchers disclosed the requested design specifications for the four methodological categories (excluded subjects, nonreported conditions and measures, and sample size determination). Disclosed information provided by participating authors also revealed several instances of questionable editorial practices, which need to be thoroughly examined and redressed. On the basis of these results, we argue that the time is now for mandatory methods disclosure statements for all psychology journals, which would be an important step forward in improving the reliability of findings in psychology.

Deep impact: unintended consequences of journal rank

Most researchers acknowledge an intrinsic hierarchy in the scholarly journals ("journal rank") that they submit their work to, and adjust not only their submission but also their reading strategies accordingly. On the other hand, much has been written about the negative effects of institutionalizing journal rank as an impact measure. So far, contributions to the debate concerning the limitations of journal rank as a scientific impact assessment tool have either lacked data, or relied on only a few studies. In this review, we present the most recent and pertinent data on the consequences of our current scholarly communication system with respect to various measures of scientific quality (such as utility/citations, methodological soundness, expert ratings or retractions). These data corroborate previous hypotheses: using journal rank as an assessment tool is bad scientific practice. Moreover, the data lead us to argue that any journal rank (not only the currently-favored Impact Factor) would have this negative impact. Therefore, we suggest that abandoning journals altogether, in favor of a library-based scholarly communication system, will ultimately be necessary. This new system will use modern information technology to vastly improve the filter, sort and discovery functions of the current journal system.

Deep impact: unintended consequences of journal rank

Most researchers acknowledge an intrinsic hierarchy in the scholarly journals (“journal rank”) that they submit their work to, and adjust not only their submission but also their reading strategies accordingly. On the other hand, much has been written about the negative effects of institutionalizing journal rank as an impact measure. So far, contributions to the debate concerning the limitations of journal rank as a scientific impact assessment tool have either lacked data, or relied on only a few studies. In this review, we present the most recent and pertinent data on the consequences of our current scholarly communication system with respect to various measures of scientific quality (such as utility/citations, methodological soundness, expert ratings or retractions). These data corroborate previous hypotheses: using journal rank as an assessment tool is bad scientific practice. Moreover, the data lead us to argue that any journal rank (not only the currently-favored Impact Factor) would have this negative impact. Therefore, we suggest that abandoning journals altogether, in favor of a library-based scholarly communication system, will ultimately be necessary. This new system will use modern information technology to vastly improve the filter, sort and discovery functions of the current journal system.

Magical thinking in predictions of negative events: Evidence for tempting fate but not for a protection effect

In this paper we test two hypotheses regarding magical thinking about the perceived likelihood of future events. The first is that people believe that those who “tempt fate” by failing to take necessary precautions are more likely to suffer negative outcomes. The second is the “protection effect”, where reminding people of precautions they have taken leads them to see related risks as less likely. To this end, we describe the results from three attempted direct replications of a protection effect experiment reported in Tykocinski (2008) and two replications of a tempting fate experiment reported in Risen and Gilovich (2008) in which we add a test of the protection effect. We did not replicate the protection effect but did replicate the tempting fate effect.

Why your new cancer biomarker may never work: recurrent patterns and remarkable diversity in biomarker failures

Less than 1% of published cancer biomarkers actually enter clinical practice. Although best practices for biomarker development are published, optimistic investigators may not appreciate the statistical near-certainty and diverse modes by which the other 99% (likely including your favorite new marker) do indeed fail. Here, patterns of failure were abstracted for classification from publications and an online database detailing marker failures. Failure patterns formed a hierarchical logical structure, or outline, of an emerging, deeply complex, and arguably fascinating science of biomarker failure. A new cancer biomarker under development is likely to have already encountered one or more of the following fatal features encountered by prior markers: lack of clinical significance, hidden structure in the source data, a technically inadequate assay, inappropriate statistical methods, unmanageable domination of the data by normal variation, implausibility, deficiencies in the studied population or in the investigator system, and its disproof or abandonment for cause by others. A greater recognition of the science of biomarker failure and its near-complete ubiquity is constructive and celebrates a seemingly perpetual richness of biologic, technical, and philosophical complexity, the full appreciation of which could improve the management of scarce research resources.

Rewarding Replications

Although replications are vital to scientific progress, psychologists rarely engage in systematic replication efforts. In this article, we consider psychologists’ narrative approach to scientific publications as an underlying reason for this neglect and propose an incentive structure for replications within psychology. First, researchers need accessible outlets for publishing replications. To accomplish this, psychology journals could publish replication reports in files that are electronically linked to reports of the original research. Second, replications should get cited. This can be achieved by cociting replications along with original research reports. Third, replications should become a valued collaborative effort. This can be realized by incorporating replications in teaching programs and by stimulating adversarial collaborations. The proposed incentive structure for replications can be developed in a relatively simple and cost-effective manner. By promoting replications, this incentive structure may greatly enhance the dependability of psychology’s knowledge base.

An Agenda for Purely Confirmatory Research

The veracity of substantive research claims hinges on the way experimental data are collected and analyzed. In this article, we discuss an uncomfortable fact that threatens the core of psychology’s academic enterprise: almost without exception, psychologists do not commit themselves to a method of data analysis before they see the actual data. It then becomes tempting to fine tune the analysis to the data in order to obtain a desired result—a procedure that invalidates the interpretation of the common statistical tests. The extent of the fine tuning varies widely across experiments and experimenters but is almost impossible for reviewers and readers to gauge. To remedy the situation, we propose that researchers preregister their studies and indicate in advance the analyses they intend to conduct. Only these analyses deserve the label “confirmatory,” and only for these analyses are the common statistical tests valid. Other analyses can be carried out but these should be labeled “exploratory.” We illustrate our proposal with a confirmatory replication attempt of a study on extrasensory perception.

Is the Replicability Crisis Overblown? Three Arguments Examined

We discuss three arguments voiced by scientists who view the current outpouring of concern about replicability as overblown. The first idea is that the adoption of a low alpha level (e.g., 5%) puts reasonable bounds on the rate at which errors can enter the published literature, making false-positive effects rare enough to be considered a minor issue. This, we point out, rests on statistical misunderstanding: The alpha level imposes no limit on the rate at which errors may arise in the literature (Ioannidis, 2005b). Second, some argue that whereas direct replication attempts are uncommon, conceptual replication attempts are common—providing an even better test of the validity of a phenomenon. We contend that performing conceptual rather than direct replication attempts interacts insidiously with publication bias, opening the door to literatures that appear to confirm the reality of phenomena that in fact do not exist. Finally, we discuss the argument that errors will eventually be pruned out of the literature if the field would just show a bit of patience. We contend that there are no plausible concrete scenarios to back up such forecasts and that what is needed is not patience, but rather systematic reforms in scientific practice.

Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability

An academic scientist's professional success depends on publishing. Publishing norms emphasize novel, positive results. As such, disciplinary incentives encourage design, analysis, and reporting decisions that elicit positive results and ignore negative results. Prior reports demonstrate how these incentives inflate the rate of false effects in published science. When incentives favor novelty over replication, false results persist in the literature unchallenged, reducing efficiency in knowledge accumulation. Previous suggestions to address this problem are unlikely to be effective. For example, a journal of negative results publishes otherwise unpublishable reports. This enshrines the low status of the journal and its content. The persistence of false findings can be meliorated with strategies that make the fundamental but abstract accuracy motive-getting it right-competitive with the more tangible and concrete incentive-getting it published. This article develops strategies for improving scientific practices and knowledge accumulation that account for ordinary human motivations and biases.