2
|
Ebersole CR, Mathur MB, Baranski E, Bart-Plange DJ, Buttrick NR, Chartier CR, Corker KS, Corley M, Hartshorne JK, IJzerman H, Lazarević LB, Rabagliati H, Ropovik I, Aczel B, Aeschbach LF, Andrighetto L, Arnal JD, Arrow H, Babincak P, Bakos BE, Baník G, Baskin E, Belopavlović R, Bernstein MH, Białek M, Bloxsom NG, Bodroža B, Bonfiglio DBV, Boucher L, Brühlmann F, Brumbaugh CC, Casini E, Chen Y, Chiorri C, Chopik WJ, Christ O, Ciunci AM, Claypool HM, Coary S, Čolić MV, Collins WM, Curran PG, Day CR, Dering B, Dreber A, Edlund JE, Falcão F, Fedor A, Feinberg L, Ferguson IR, Ford M, Frank MC, Fryberger E, Garinther A, Gawryluk K, Ashbaugh K, Giacomantonio M, Giessner SR, Grahe JE, Guadagno RE, Hałasa E, Hancock PJB, Hilliard RA, Hüffmeier J, Hughes S, Idzikowska K, Inzlicht M, Jern A, Jiménez-Leal W, Johannesson M, Joy-Gaba JA, Kauff M, Kellier DJ, Kessinger G, Kidwell MC, Kimbrough AM, King JPJ, Kolb VS, Kołodziej S, Kovacs M, Krasuska K, Kraus S, Krueger LE, Kuchno K, Lage CA, Langford EV, Levitan CA, de Lima TJS, Lin H, Lins S, Loy JE, Manfredi D, Markiewicz Ł, Menon M, Mercier B, Metzger M, Meyet V, Millen AE, Miller JK, Montealegre A, Moore DA, Muda R, Nave G, Nichols AL, Novak SA, Nunnally C, Orlić A, Palinkas A, Panno A, Parks KP, Pedović I, Pękala E, Penner MR, Pessers S, Petrović B, Pfeiffer T, Pieńkosz D, Preti E, Purić D, Ramos T, Ravid J, Razza TS, Rentzsch K, Richetin J, Rife SC, Rosa AD, Rudy KH, Salamon J, Saunders B, Sawicki P, Schmidt K, Schuepfer K, Schultze T, Schulz-Hardt S, Schütz A, Shabazian AN, Shubella RL, Siegel A, Silva R, Sioma B, Skorb L, de Souza LEC, Steegen S, Stein LAR, Sternglanz RW, Stojilović D, Storage D, Sullivan GB, Szaszi B, Szecsi P, Szöke O, Szuts A, Thomae M, Tidwell ND, Tocco C, Torka AK, Tuerlinckx F, Vanpaemel W, Vaughn LA, Vianello M, Viganola D, Vlachou M, Walker RJ, Weissgerber SC, Wichman AL, Wiggins BJ, Wolf D, Wood MJ, Zealley D, Žeželj I, Zrubka M, Nosek BA. Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability. ADVANCES IN METHODS AND PRACTICES IN PSYCHOLOGICAL SCIENCE 2020. [DOI: 10.1177/2515245920958687] [Citation(s) in RCA: 21] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect ( p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δ r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols ( r = .05) was similar to that of the RP:P protocols ( r = .04) and the original RP:P replications ( r = .11), and smaller than that of the original studies ( r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00–.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19–.50).
Collapse
Affiliation(s)
| | | | | | | | | | | | | | - Martin Corley
- Psychology, School of Philosophy, Psychology & Language Sciences, University of Edinburgh
| | | | - Hans IJzerman
- LIP/PC2S, Université Grenoble Alpes
- Institut Universitaire de France
| | - Ljiljana B. Lazarević
- Institute of Psychology, Faculty of Philosophy, University of Belgrade
- Laboratory for Research of Individual Differences, Faculty of Philosophy, University of Belgrade
| | - Hugh Rabagliati
- Psychology, School of Philosophy, Psychology & Language Sciences, University of Edinburgh
| | - Ivan Ropovik
- Institute for Research and Development of Education, Faculty of Education, Charles University
- Faculty of Education, University of Presov
| | - Balazs Aczel
- Institute of Psychology, ELTE Eötvös Loránd University
| | | | | | | | - Holly Arrow
- Department of Psychology, University of Oregon
| | - Peter Babincak
- Institute of Psychology, Faculty of Arts, University of Presov
| | | | - Gabriel Baník
- Institute of Psychology, Faculty of Arts, University of Presov
| | - Ernest Baskin
- Department of Food Marketing, Haub School of Business, Saint Joseph’s University
| | | | - Michael H. Bernstein
- Department of Behavioral and Social Sciences, School of Public Health, Brown University
- Department of Psychology, University of Rhode Island
| | | | | | - Bojana Bodroža
- Department of Psychology, Faculty of Philosophy, University of Novi Sad
| | | | - Leanne Boucher
- Department of Psychology and Neuroscience, Nova Southeastern University
| | | | | | - Erica Casini
- Department of Psychology, University of Milano-Bicocca
| | - Yiling Chen
- John A. Paulson School of Engineering and Applied Sciences, Harvard University
| | - Carlo Chiorri
- Department of Educational Science, University of Genova
| | | | | | | | | | - Sean Coary
- Quinlan School of Business, Loyola University Chicago
| | - Marija V. Čolić
- Faculty of Sport and Physical Education, University of Belgrade
| | | | | | - Chris R. Day
- Centre for Trust, Peace and Social Relations, Coventry University
| | | | - Anna Dreber
- Department of Economics, Stockholm School of Economics
- Department of Economics, University of Innsbruck
| | - John E. Edlund
- Department of Psychology, Rochester Institute of Technology
| | | | - Anna Fedor
- Institute of Evolution, Centre for Ecological Research, Budapest, Hungary
| | - Lily Feinberg
- Psychology and Neuroscience Department, Boston College
| | - Ian R. Ferguson
- Department of Psychology, Virginia Commonwealth University
- Department of Psychology, New York University
| | - Máire Ford
- Department of Psychology, Loyola Marymount University
| | | | | | | | | | - Kayla Ashbaugh
- Department of Biology and Biomedical Engineering, Rose-Hulman Institute of Technology
| | - Mauro Giacomantonio
- Department of Social & Developmental Psychology, Sapienza University of Rome
| | | | - Jon E. Grahe
- Department of Psychology, Pacific Lutheran University
| | | | - Ewa Hałasa
- Faculty of Economics, Maria Curie-Skłodowska University
| | | | - Rias A. Hilliard
- Department of Humanities, Social Sciences, and the Arts, Rose-Hulman Institute of Technology
| | | | - Sean Hughes
- Department of Experimental Clinical and Health Psychology, Ghent University
| | | | | | - Alan Jern
- Department of Humanities, Social Sciences, and the Arts, Rose-Hulman Institute of Technology
| | | | | | | | | | | | | | | | - Amanda M. Kimbrough
- School of Arts, Technology, Emerging Media, & Communication, University of Texas at Dallas
| | - Josiah P. J. King
- Psychology, School of Philosophy, Psychology & Language Sciences, University of Edinburgh
| | | | | | - Marton Kovacs
- Institute of Psychology, ELTE Eötvös Loránd University
| | | | | | - Lacy E. Krueger
- Department of Psychology & Special Education, Texas A&M University-Commerce
| | | | - Caio Ambrosio Lage
- Department of Psychology, Pontifical Catholic University of Rio de Janeiro
| | | | | | | | - Hause Lin
- Department of Psychology, University of Toronto
| | - Samuel Lins
- Department of Psychology, University of Porto
| | - Jia E. Loy
- Linguistics & English Language, School of Philosophy, Psychology & Language Sciences, University of Edinburgh
| | - Dylan Manfredi
- Marketing Department, The Wharton School of the University of Pennsylvania
| | | | - Madhavi Menon
- Department of Psychology and Neuroscience, Nova Southeastern University
| | - Brett Mercier
- Department of Psychological Science, University of California, Irvine
| | | | - Venus Meyet
- Department of Psychology, Brigham Young University–Idaho
| | | | | | | | - Don A. Moore
- Haas School of Business, University of California at Berkeley
| | - Rafał Muda
- Faculty of Economics, Maria Curie-Skłodowska University
| | - Gideon Nave
- Marketing Department, The Wharton School of the University of Pennsylvania
| | | | | | - Christian Nunnally
- Department of Computer Science and Software Engineering, Rose-Hulman Institute of Technology
| | - Ana Orlić
- Faculty of Sport and Physical Education, University of Belgrade
| | - Anna Palinkas
- Institute of Psychology, ELTE Eötvös Loránd University
| | - Angelo Panno
- Department of Human Science, European University of Rome
| | | | - Ivana Pedović
- Department of Psychology, Faculty of Philosophy, University of Niš
| | | | | | | | - Boban Petrović
- Laboratory for Research of Individual Differences, Faculty of Philosophy, University of Belgrade
- Institute of Criminological and Sociological Research, Belgrade, Serbia
| | | | | | | | - Danka Purić
- Laboratory for Research of Individual Differences, Faculty of Philosophy, University of Belgrade
- Department of Psychology, Faculty of Philosophy, University of Belgrade
| | - Tiago Ramos
- Department of Psychology, University of Porto
| | | | - Timothy S. Razza
- Department of Psychology and Neuroscience, Nova Southeastern University
| | | | | | - Sean C. Rife
- Department of Psychology, Murray State University
| | - Anna Dalla Rosa
- Department of Philosophy, Sociology, Education and Applied Psychology, University of Padova
| | | | - Janos Salamon
- Institute of Psychology, ELTE Eötvös Loránd University
- Doctoral School of Psychology, ELTE Eötvös Loránd University
| | | | | | - Kathleen Schmidt
- School of Psychological and Behavioral Sciences, Southern Illinois University Carbondale
| | | | - Thomas Schultze
- Institute of Psychology, University of Göttingen
- Leibniz Science Campus Primate Cognition, Göttingen, Germany
| | - Stefan Schulz-Hardt
- Institute of Psychology, University of Göttingen
- Leibniz Science Campus Primate Cognition, Göttingen, Germany
| | | | | | - Rachel L. Shubella
- Department of Humanities, Social Sciences, and the Arts, Rose-Hulman Institute of Technology
| | | | - Rúben Silva
- Department of Psychology, University of Porto
| | - Barbara Sioma
- Faculty of Economics, Maria Curie-Skłodowska University
| | - Lauren Skorb
- Psychology and Neuroscience Department, Boston College
| | | | - Sara Steegen
- Psychology and Educational Sciences, University of Leuven
| | - L. A. R. Stein
- Department of Behavioral and Social Sciences, School of Public Health, Brown University
- Department of Psychology, University of Rhode Island
- Rhode Island Training School, Rhode Island Department of Children, Youth and Families
| | | | | | | | | | | | - Peter Szecsi
- Institute of Psychology, ELTE Eötvös Loránd University
| | - Orsolya Szöke
- Institute of Psychology, ELTE Eötvös Loránd University
| | - Attila Szuts
- Institute of Psychology, ELTE Eötvös Loránd University
| | - Manuela Thomae
- MEU - Die Multiversität
- Diploma University of Applied Sciences
| | | | - Carly Tocco
- Department of Psychology, Queens College, City University of New York
| | | | | | - Wolf Vanpaemel
- Psychology and Educational Sciences, University of Leuven
| | | | - Michelangelo Vianello
- Department of Philosophy, Sociology, Education and Applied Psychology, University of Padova
| | | | - Maria Vlachou
- Psychology and Educational Sciences, University of Leuven
| | | | | | - Aaron L. Wichman
- Department of Psychological Sciences, Western Kentucky University
| | | | - Daniel Wolf
- Department of Psychology, University of Bamberg
| | | | - David Zealley
- Department of Psychology, Brigham Young University–Idaho
| | - Iris Žeželj
- Laboratory for Research of Individual Differences, Faculty of Philosophy, University of Belgrade
- Department of Psychology, Faculty of Philosophy, University of Belgrade
| | - Mark Zrubka
- Department of Psychology, University of Amsterdam
| | - Brian A. Nosek
- Department of Psychology, University of Virginia
- Center for Open Science, Charlottesville, Virginia
| |
Collapse
|