2
|
Breznau N, Rinke EM, Wuttke A, Nguyen HHV, Adem M, Adriaans J, Alvarez-Benjumea A, Andersen HK, Auer D, Azevedo F, Bahnsen O, Balzer D, Bauer G, Bauer PC, Baumann M, Baute S, Benoit V, Bernauer J, Berning C, Berthold A, Bethke FS, Biegert T, Blinzler K, Blumenberg JN, Bobzien L, Bohman A, Bol T, Bostic A, Brzozowska Z, Burgdorf K, Burger K, Busch KB, Carlos-Castillo J, Chan N, Christmann P, Connelly R, Czymara CS, Damian E, Ecker A, Edelmann A, Eger MA, Ellerbrock S, Forke A, Forster A, Gaasendam C, Gavras K, Gayle V, Gessler T, Gnambs T, Godefroidt A, Grömping M, Groß M, Gruber S, Gummer T, Hadjar A, Heisig JP, Hellmeier S, Heyne S, Hirsch M, Hjerm M, Hochman O, Hövermann A, Hunger S, Hunkler C, Huth N, Ignácz ZS, Jacobs L, Jacobsen J, Jaeger B, Jungkunz S, Jungmann N, Kauff M, Kleinert M, Klinger J, Kolb JP, Kołczyńska M, Kuk J, Kunißen K, Kurti Sinatra D, Langenkamp A, Lersch PM, Löbel LM, Lutscher P, Mader M, Madia JE, Malancu N, Maldonado L, Marahrens H, Martin N, Martinez P, Mayerl J, Mayorga OJ, McManus P, McWagner K, Meeusen C, Meierrieks D, Mellon J, Merhout F, Merk S, Meyer D, Micheli L, Mijs J, Moya C, Neunhoeffer M, Nüst D, Nygård O, Ochsenfeld F, Otte G, Pechenkina AO, Prosser C, Raes L, Ralston K, Ramos MR, Roets A, Rogers J, Ropers G, Samuel R, Sand G, Schachter A, Schaeffer M, Schieferdecker D, Schlueter E, Schmidt R, Schmidt KM, Schmidt-Catran A, Schmiedeberg C, Schneider J, Schoonvelde M, Schulte-Cloos J, Schumann S, Schunck R, Schupp J, Seuring J, Silber H, Sleegers W, Sonntag N, Staudt A, Steiber N, Steiner N, Sternberg S, Stiers D, Stojmenovska D, Storz N, Striessnig E, Stroppe AK, Teltemann J, Tibajev A, Tung B, Vagni G, Van Assche J, van der Linden M, van der Noll J, Van Hootegem A, Vogtenhuber S, Voicu B, Wagemans F, Wehl N, Werner H, Wiernik BM, Winter F, Wolf C, Yamada Y, Zhang N, Ziller C, Zins S, Żółtak T. Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. Proc Natl Acad Sci U S A 2022; 119:e2203150119. [PMID: 36306328 PMCID: PMC9636921 DOI: 10.1073/pnas.2203150119] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2022] [Accepted: 08/22/2022] [Indexed: 11/25/2022] Open
Abstract
This study explores how researchers' analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers' expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers' results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.
Collapse
Affiliation(s)
- Nate Breznau
- Research Center on Inequality and Social Policy (SOCIUM), University of Bremen, Bremen, 28359, Germany
| | - Eike Mark Rinke
- School of Politics and International Studies, University of Leeds, Leeds, LS2 9JT, United Kingdom
| | - Alexander Wuttke
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
- Department of Political Science, Ludwig Maximilian University, 80539 Munich, Germany
| | - Hung H. V. Nguyen
- Research Center on Inequality and Social Policy (SOCIUM), University of Bremen, Bremen, 28359, Germany
- Bremen International Graduate School of Social Sciences, 28359 Bremen, Germany
| | - Muna Adem
- Department of Sociology, Indiana University, Bloomington, IN 47405
| | - Jule Adriaans
- Socio-Economic Panel Study (SOEP), German Institute for Economic Research (DIW), 10117 Berlin, Germany
| | - Amalia Alvarez-Benjumea
- Mechanisms of Normative Change, Max Planck Institute for Research on Collective Goods, 53113 Bonn, Germany
| | - Henrik K. Andersen
- Institute of Sociology, Chemnitz University of Technology, 09126 Chemnitz, Germany
| | - Daniel Auer
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
| | - Flavio Azevedo
- Department of Psychology, University of Cambridge, Cambridge, CB23RQ, United Kingdom
| | - Oke Bahnsen
- School of Social Sciences, University of Mannheim, 68159 Mannheim, Germany
| | - Dave Balzer
- Institute of Sociology, Johannes Gutenberg University Mainz, 55128 Mainz, Germany
| | - Gerrit Bauer
- Department of Sociology, Ludwig Maximilian University, 80801 Munich, Germany
| | - Paul C. Bauer
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
| | - Markus Baumann
- Heidelberg University, 69117 Heidelberg, Germany
- Institute for Political Science, Goethe University Frankfurt, 60323 Frankfurt, Germany
| | - Sharon Baute
- Comparative Political Economy, University of Konstanz, 78457 Konstanz, Germany
| | - Verena Benoit
- Department of Political Science, Ludwig Maximilian University, 80539 Munich, Germany
- Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, 96052 Bamberg, Germany
| | - Julian Bernauer
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
| | - Carl Berning
- Institute for Political Science, Johannes Gutenberg University Mainz, 55099 Mainz, Germany
| | - Anna Berthold
- Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, 96052 Bamberg, Germany
| | - Felix S. Bethke
- Research Department on Intrastate Conflict, Peace Research Institute Frankfurt, 60329 Frankfurt, Germany
| | - Thomas Biegert
- Department of Social Policy, London School of Economics and Political Science, London, WC2A 2AE, United Kingdom
| | - Katharina Blinzler
- Survey Data Curation, Leibniz Institute for the Social Sciences (GESIS), 50667 Cologne, Germany
| | - Johannes N. Blumenberg
- Knowledge Exchange and Outreach, Leibniz Institute for the Social Sciences (GESIS), 68159 Mannheim, Germany
| | - Licia Bobzien
- Jacques Delors Centre, Hertie School, 10117 Berlin, Germany
| | - Andrea Bohman
- Department of Sociology, Umeå University, 90187 Umeå, Sweden
| | - Thijs Bol
- Social Research Institute, Institute of Education, University College London, London, WC1H 0AL, United Kingdom
- Department of Sociology, University of Amsterdam, 1001 Amsterdam, The Netherlands
| | - Amie Bostic
- Department of Sociology, The University of Texas Rio Grande Valley, Brownsville, TX 78520
| | - Zuzanna Brzozowska
- Vienna Institute of Demography, Austrian Academy of Sciences, 1030 Vienna, Austria
- Austrian National Public Health Institute, Gesundheit Österreich (GÖG), 1030 Vienna, Austria
| | - Katharina Burgdorf
- School of Social Sciences, University of Mannheim, 68159 Mannheim, Germany
| | - Kaspar Burger
- Social Research Institute, Institute of Education, University College London, London, WC1H 0AL, United Kingdom
- Department of Sociology, University of Zurich, 8050 Zurich, Switzerland
- Jacobs Center for Productive Youth, University of Zurich, 8050 Zurich, Switzerland
| | | | - Juan Carlos-Castillo
- Department of Sociology, University of Chile, Santiago, 7800284, Chile
- Center for Social Conflict and Cohesion Studies (COES), Pontificia Universidad Católica de Chile, Santiago, 8331150, Chile
| | - Nathan Chan
- Department of Political Science and International Relations, Loyola Marymount University, Los Angeles, CA 90045
| | - Pablo Christmann
- Data and Research on Society, Leibniz Institute for the Social Sciences, 68159 Mannheim, Germany
| | - Roxanne Connelly
- School of Social and Political Science, University of Edinburgh, Edinburgh, EH8 9LD, United Kingdom
| | | | - Elena Damian
- Lifestyle and Chronic Diseases, Epidemiology and Public Health, Sciensano, 1000 Brussels, Belgium
| | - Alejandro Ecker
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
| | | | - Maureen A. Eger
- Department of Sociology, Umeå University, 90187 Umeå, Sweden
| | - Simon Ellerbrock
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
- School of Social Sciences, University of Mannheim, 68159 Mannheim, Germany
| | | | - Andrea Forster
- Empirical Educational and Higher Education Research, Freie Universität Berlin, 14195 Berlin, Germany
| | - Chris Gaasendam
- Department of Sociology, Center for Sociological Research, KU Leuven, 3000 Leuven, Belgium
| | - Konstantin Gavras
- School of Social Sciences, University of Mannheim, 68159 Mannheim, Germany
| | - Vernon Gayle
- School of Social and Political Science, University of Edinburgh, Edinburgh, EH8 9LD, United Kingdom
| | - Theresa Gessler
- Kulturwissenschaftliche Fakultät, European University Viadrina, 15230 Frankfurt (Oder), Germany
| | - Timo Gnambs
- Educational Measurement, Leibniz Institute for Educational Trajectories, 96047 Bamberg, Germany
| | - Amélie Godefroidt
- Centre for Research on Peace and Development, KU Leuven, 3000 Leuven, Belgium
| | - Max Grömping
- School of Government and International Relations, Griffith University, Nathan, QLD, 4111, Australia
| | - Martin Groß
- Department of Sociology, University of Tübingen, 72074 Tübingen, Germany
| | - Stefan Gruber
- Max Planck Institute for Social Law and Social Policy, 80799 Munich, Germany
| | - Tobias Gummer
- Data and Research on Society, Leibniz Institute for the Social Sciences, 68159 Mannheim, Germany
| | - Andreas Hadjar
- University of Luxembourg, 4365 Esch-sur-Alzette, Luxembourg
- Wirtschafts- und Sozialwissenschaftliches Institut (WSI), Hans Böckler Foundation, 40474 Düsseldorf, Germany
- University of Fribourg, 1700 Fribourg, Switzerland
- Department of Social Sciences, University of Luxembourg, 4366 Esch-sur-Alzette, Luxembourg
| | - Jan Paul Heisig
- University of Groningen, 9712 CP Groningen,The Netherlands
- Research Group "Health and Social Inequality", Berlin Social Science Center (WZB), 10785 Berlin, Germany
| | - Sebastian Hellmeier
- Transformations of Democracy Unit, Berlin Social Science Center (WZB), 10785 Berlin, Germany
| | - Stefanie Heyne
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
| | - Magdalena Hirsch
- Research Unit Migration, Integration, Transnationalization, Berlin Social Science Center (WZB), 10785 Berlin, Germany
| | - Mikael Hjerm
- Department of Sociology, Umeå University, 90187 Umeå, Sweden
| | - Oshrat Hochman
- Data and Research on Society, Leibniz Institute for the Social Sciences, 68159 Mannheim, Germany
| | - Andreas Hövermann
- Wirtschafts- und Sozialwissenschaftliches Institut (WSI), Hans Böckler Foundation, 40474 Düsseldorf, Germany
- German Socio-Economic Panel Survey, 10117 Berlin, Germany
| | - Sophia Hunger
- Center for Civil Society Research, Berlin Social Science Center, 10785 Berlin, Germany
| | - Christian Hunkler
- Berlin Institute for Integration and Migration Research (BIM), Humboldt University Berlin, 10099 Berlin, Germany
| | - Nora Huth
- School of Human and Social Sciences, University of Wuppertal, 42119 Wuppertal, Germany
| | - Zsófia S. Ignácz
- Institute of Sociology, Goethe University Frankfurt, 60323 Frankfurt, Germany
| | - Laura Jacobs
- Department of Political Science, Université Libre de Bruxelles, 1050 Bruxelles, Belgium
| | - Jannes Jacobsen
- Zeppelin University, 88045 Friedrichshafen, Germany
- Cluster "Data-Methods-Monitoring", German Center for Integration and Migration Research (DeZIM),10117 Berlin, Germany
| | - Bastian Jaeger
- Department of Social Psychology, Tilburg University, 5037AB Tilburg, The Netherlands
| | - Sebastian Jungkunz
- Institute for Socio-Economics, University of Duisburg-Essen, 47057 Duisburg, Germany
- Institute of Political Science, University of Münster, 48149 Münster, Germany
- Chair of Political Sociology, University of Bamberg, 96052 Bamberg, Germany
| | - Nils Jungmann
- Survey Data Curation, Leibniz Institute for the Social Sciences (GESIS), 50667 Cologne, Germany
| | - Mathias Kauff
- Department of Psychology, Medical School Hamburg, 20457 Hamburg, Germany
| | - Manuel Kleinert
- Institute of Sociology, Justus Liebig University of Giessen, 35394 Giessen, Germany
| | - Julia Klinger
- Institute of Sociology and Social Psychology, University of Cologne, 50931 Cologne, Germany
| | - Jan-Philipp Kolb
- Federal Statistics Office Germany, Destatis, 65189 Wiesbaden, Germany
| | - Marta Kołczyńska
- Department of Research on Social and Institutional Transformations, Institute of Political Studies of the Polish Academy of Sciences, 00-625 Warsaw, Poland
| | - John Kuk
- Department of Political Science, University of Oklahoma, Norman, OK 73019
| | - Katharina Kunißen
- Institute of Sociology, Johannes Gutenberg University Mainz, 55128 Mainz, Germany
| | | | | | - Philipp M. Lersch
- Socio-Economic Panel Study (SOEP), German Institute for Economic Research (DIW), 10117 Berlin, Germany
- Department of Social Sciences, Humboldt University Berlin, 10099 Berlin, Germany
| | - Lea-Maria Löbel
- Socio-Economic Panel Study (SOEP), German Institute for Economic Research (DIW), 10117 Berlin, Germany
| | - Philipp Lutscher
- Department of Political Science, University of Oslo, 0851 Oslo, Norway
| | - Matthias Mader
- Department of Politics and Public Administration, University of Konstanz, 78457 Konstanz, Germany
| | - Joan E. Madia
- Department of Sociology, Nuffield College, University of Oxford, Oxford, OX1 1JD, United Kingdom
- Institute for the Evaluation of Public Policies, Fondazione Bruno Kessler, 38122 Trento, Italy
| | - Natalia Malancu
- The Institute of Citizenship Studies (InCite), University of Geneva, 1205 Geneva, Switzerland
| | - Luis Maldonado
- Instituto de Sociologia, Pontifical Catholic University of Chile, Santiago, 7820436, Chile
| | - Helge Marahrens
- Department of Sociology, Indiana University, Bloomington, IN 47405
| | - Nicole Martin
- Department of Politics, University of Manchester, Manchester, M19 2JS, United Kingdom
| | - Paul Martinez
- Department of Institutional Research, Western Governors University, Salt Lake City, UT 84107
| | - Jochen Mayerl
- Institute of Sociology, Chemnitz University of Technology, 09126 Chemnitz, Germany
| | - Oscar J. Mayorga
- Department of Sociology, University of California, Los Angeles, CA 90095
| | - Patricia McManus
- Department of Sociology, Indiana University, Bloomington, IN 47405
| | - Kyle McWagner
- Department of Political Science, The University of California, Irvine, CA 92617
| | - Cecil Meeusen
- Department of Sociology, Center for Sociological Research, KU Leuven, 3000 Leuven, Belgium
| | - Daniel Meierrieks
- Research Unit Migration, Integration, Transnationalization, Berlin Social Science Center (WZB), 10785 Berlin, Germany
| | - Jonathan Mellon
- Department of Politics, University of Manchester, Manchester, M19 2JS, United Kingdom
| | - Friedolin Merhout
- Department of Sociology and Centre for Social Data Science, University of Copenhagen, 1353 Copenhagen, Denmark
| | - Samuel Merk
- Department of School Development, University of Education Karlsruhe, 76133 Karlsruhe, Germany
| | - Daniel Meyer
- Department of Education and Social Sciences, University of Cologne, 50931 Cologne, Germany
| | - Leticia Micheli
- Department of Psychology III, Julius-Maximilians University Würzburg, 97070 Würzburg, Germany
| | - Jonathan Mijs
- Department of Sociology, Boston University, Boston, MA 02215
| | - Cristóbal Moya
- Faculty of Sociology, Bielefeld University, 33615 Bielefeld, Germany
| | - Marcel Neunhoeffer
- School of Social Sciences, University of Mannheim, 68159 Mannheim, Germany
| | - Daniel Nüst
- Department of Geosciences, University of Münster, 49149 Münster, Germany
| | - Olav Nygård
- Division of Migration, Ethnicity and Society (REMESO), Linköping University, 60174 Linköping, Sweden
| | - Fabian Ochsenfeld
- Administrative Headquarters, Max Planck Society, 80539 Berlin, Germany
| | - Gunnar Otte
- Institute of Sociology, Johannes Gutenberg University Mainz, 55128 Mainz, Germany
| | | | - Christopher Prosser
- Department of Politics, International Relations and Philosophy, Royal Holloway University of London, London, TW20 0EX, United Kingdom
| | - Louis Raes
- Department of Economics, Tilburg University, 5037AB Tilburg, The Netherlands
| | - Kevin Ralston
- School of Social and Political Science, University of Edinburgh, Edinburgh, EH8 9LD, United Kingdom
| | - Miguel R. Ramos
- Department of Social Policy, Sociology and Criminology, University of Birmingham, Birmingham, B15 2TT, United Kingdom
| | - Arne Roets
- Department of Developmental, Personality and Social Psychology, Ghent University, 9000 Ghent, Belgium
| | - Jonathan Rogers
- Division of Social Science, New York University Abu Dhabi, Abu Dhabi, 10276, United Arab Emirates
| | - Guido Ropers
- School of Social Sciences, University of Mannheim, 68159 Mannheim, Germany
| | - Robin Samuel
- University of Luxembourg, 4365 Esch-sur-Alzette, Luxembourg
- Department of Social Sciences, University of Luxembourg, 4366 Esch-sur-Alzette, Luxembourg
| | - Gregor Sand
- Max Planck Institute for Social Law and Social Policy, 80799 Munich, Germany
| | - Ariela Schachter
- Department of Sociology, Washington University in St. Louis, St. Louis, MO 63130
| | - Merlin Schaeffer
- Department of Sociology, University of Copenhagen, 1353 Copenhagen, Denmark
| | - David Schieferdecker
- Institute for Media and Communication Studies, Freie Universität Berlin, 14195 Berlin, Germany
| | - Elmar Schlueter
- Institute of Sociology, Justus Liebig University of Giessen, 35394 Giessen, Germany
| | - Regine Schmidt
- Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, 96052 Bamberg, Germany
| | - Katja M. Schmidt
- Socio-Economic Panel Study (SOEP), German Institute for Economic Research (DIW), 10117 Berlin, Germany
| | | | | | - Jürgen Schneider
- Tübingen School of Education, University of Tübingen, 72074 Tübingen, Germany
| | - Martijn Schoonvelde
- University College Dublin, Dublin 4, Ireland
- Department of European Languages and Cultures, University of Groningen, 9712 EK Groningen, The Netherlands
| | - Julia Schulte-Cloos
- Robert Schuman Center for Advanced Studies, European University Institute, 50133 Florence, Italy
| | - Sandy Schumann
- Department of Security and Crime Science, University College London, London,WC1E 6BT, United Kingdom
| | - Reinhard Schunck
- School of Human and Social Sciences, University of Wuppertal, 42119 Wuppertal, Germany
| | - Jürgen Schupp
- Socio-Economic Panel Study (SOEP), German Institute for Economic Research (DIW), 10117 Berlin, Germany
| | - Julian Seuring
- Department of Migration, Leibniz Institute for Educational Trajectories, 96047 Bamberg, Germany
| | - Henning Silber
- Department of Survey Design and Methodology, Leibniz Institute for the Social Sciences (GESIS), 68159 Mannheim, Germany
| | - Willem Sleegers
- Department of Social Psychology, Tilburg University, 5037AB Tilburg, The Netherlands
| | - Nico Sonntag
- Institute of Sociology, Johannes Gutenberg University Mainz, 55128 Mainz, Germany
| | | | - Nadia Steiber
- Department of Sociology, University of Vienna, 1090 Vienna, Austria
| | - Nils Steiner
- Institute for Political Science, Johannes Gutenberg University Mainz, 55099 Mainz, Germany
| | | | - Dieter Stiers
- Center for Political Science Research, KU Leuven, 3000 Leuven, Belgium
| | - Dragana Stojmenovska
- Department of Sociology, University of Amsterdam, 1001 Amsterdam, The Netherlands
| | - Nora Storz
- Interdisciplinary Social Science, Utrecht University, 3584 Utrecht, The Netherlands
| | - Erich Striessnig
- Department of Demography, University of Vienna, 1010 Vienna, Austria
| | - Anne-Kathrin Stroppe
- Survey Data Curation, Leibniz Institute for the Social Sciences (GESIS), 50667 Cologne, Germany
| | - Janna Teltemann
- Institute for Social Sciences, University of Hildesheim, 31141 Hildesheim, Germany
| | - Andrey Tibajev
- Division of Migration, Ethnicity and Society (REMESO), Linköping University, 60174 Linköping, Sweden
| | - Brian Tung
- Department of Sociology, Washington University in St. Louis, St. Louis, MO 63130
| | - Giacomo Vagni
- Social Research Institute, Institute of Education, University College London, London, WC1H 0AL, United Kingdom
| | - Jasper Van Assche
- Department of Developmental, Personality and Social Psychology, Ghent University, 9000 Ghent, Belgium
- Center for Social and Cultural Psychology, Université Libre de Bruxelles, 1050 Brussels, Belgium
| | - Meta van der Linden
- Interdisciplinary Social Science, Utrecht University, 3584 Utrecht, The Netherlands
| | | | - Arno Van Hootegem
- Department of Sociology, Center for Sociological Research, KU Leuven, 3000 Leuven, Belgium
| | - Stefan Vogtenhuber
- Education and Employment, Institute for Advanced Studies, University of Vienna, Vienna, 1080 Austria
| | - Bogdan Voicu
- Research Institute for Quality of Life, Romanian Academy, 010071 Bucharest, Romania
- Department of Sociology, Lucian Blaga University of Sibiu, 550024 Sibiu, Romania
| | - Fieke Wagemans
- Netherlands Institute for Social Research, 2500 BD The Hague, the Netherlands
- Policy Perspectives, Citizen Perspectives, and Behaviors, Netherlands Institute for Social Research, 2594 The Hague, The Netherlands
| | - Nadja Wehl
- Research Cluster "The Politics of Inequality", University of Konstanz, 78464 Konstanz, Germany
| | - Hannah Werner
- Center for Political Science Research, KU Leuven, 3000 Leuven, Belgium
| | | | - Fabian Winter
- Mechanisms of Normative Change, Max Planck Institute for Research on Collective Goods, 53113 Bonn, Germany
| | - Christof Wolf
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
- School of Social Sciences, University of Mannheim, 68159 Mannheim, Germany
- President, Leibniz Institute for the Social Sciences (GESIS), 68159 Mannheim, Germany
| | - Yuki Yamada
- Faculty of Arts and Science, Kyushu University, Fukuoka, 819-0395, Japan
| | - Nan Zhang
- Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany
| | - Conrad Ziller
- Institute for Socio-Economics, University of Duisburg-Essen, 47057 Duisburg, Germany
- Department of Political Science, University of Duisburg-Essen, 47057 Duisburg, Germany
| | - Stefan Zins
- Institute for Employment Research, Federal Employment Agency, 90478 Nuremberg, Germany
| | - Tomasz Żółtak
- Department of Research on Social and Institutional Transformations, Institute of Political Studies of the Polish Academy of Sciences, 00-625 Warsaw, Poland
| |
Collapse
|
4
|
Salganik MJ, Lundberg I, Kindel AT, Ahearn CE, Al-Ghoneim K, Almaatouq A, Altschul DM, Brand JE, Carnegie NB, Compton RJ, Datta D, Davidson T, Filippova A, Gilroy C, Goode BJ, Jahani E, Kashyap R, Kirchner A, McKay S, Morgan AC, Pentland A, Polimis K, Raes L, Rigobon DE, Roberts CV, Stanescu DM, Suhara Y, Usmani A, Wang EH, Adem M, Alhajri A, AlShebli B, Amin R, Amos RB, Argyle LP, Baer-Bositis L, Büchi M, Chung BR, Eggert W, Faletto G, Fan Z, Freese J, Gadgil T, Gagné J, Gao Y, Halpern-Manners A, Hashim SP, Hausen S, He G, Higuera K, Hogan B, Horwitz IM, Hummel LM, Jain N, Jin K, Jurgens D, Kaminski P, Karapetyan A, Kim EH, Leizman B, Liu N, Möser M, Mack AE, Mahajan M, Mandell N, Marahrens H, Mercado-Garcia D, Mocz V, Mueller-Gastell K, Musse A, Niu Q, Nowak W, Omidvar H, Or A, Ouyang K, Pinto KM, Porter E, Porter KE, Qian C, Rauf T, Sargsyan A, Schaffner T, Schnabel L, Schonfeld B, Sender B, Tang JD, Tsurkov E, van Loon A, Varol O, Wang X, Wang Z, Wang J, Wang F, Weissman S, Whitaker K, Wolters MK, Woon WL, Wu J, Wu C, Yang K, Yin J, Zhao B, Zhu C, Brooks-Gunn J, Engelhardt BE, Hardt M, Knox D, Levy K, Narayanan A, Stewart BM, Watts DJ, McLanahan S. Measuring the predictability of life outcomes with a scientific mass collaboration. Proc Natl Acad Sci U S A 2020; 117:8398-8403. [PMID: 32229555 PMCID: PMC7165437 DOI: 10.1073/pnas.1915006117] [Citation(s) in RCA: 58] [Impact Index Per Article: 14.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022] Open
Abstract
How predictable are life trajectories? We investigated this question with a scientific mass collaboration using the common task method; 160 teams built predictive models for six life outcomes using data from the Fragile Families and Child Wellbeing Study, a high-quality birth cohort study. Despite using a rich dataset and applying machine-learning methods optimized for prediction, the best predictions were not very accurate and were only slightly better than those from a simple benchmark model. Within each outcome, prediction error was strongly associated with the family being predicted and weakly associated with the technique used to generate the prediction. Overall, these results suggest practical limits to the predictability of life outcomes in some settings and illustrate the value of mass collaborations in the social sciences.
Collapse
Affiliation(s)
| | - Ian Lundberg
- Department of Sociology, Princeton University, Princeton, NJ 08544
| | | | - Caitlin E Ahearn
- Department of Sociology, University of California, Los Angeles, CA 90095
| | | | - Abdullah Almaatouq
- Sloan School of Management, Massachusetts Institute of Technology, Cambridge, MA 02142
- Media Lab, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Drew M Altschul
- Mental Health Data Science Scotland, Department of Psychology, The University of Edinburgh, Edinburgh EH8 9JZ, United Kingdom
| | - Jennie E Brand
- Department of Sociology, University of California, Los Angeles, CA 90095
- Department of Statistics, University of California, Los Angeles, CA 90095
| | | | - Ryan James Compton
- Human Computer Interaction Lab, University of California, Santa Cruz, CA 95064
| | - Debanjan Datta
- Discovery Analytics Center, Virginia Polytechnic Institute and State University, Arlington, VA 22203
| | - Thomas Davidson
- Department of Sociology, Cornell University, Ithaca, NY 14853
| | | | - Connor Gilroy
- Department of Sociology, University of Washington, Seattle, WA 98105
| | - Brian J Goode
- Social and Decision Analytics Laboratory, Fralin Life Sciences Institute, Virginia Polytechnic Institute and State University, Arlington, VA 22203
| | - Eaman Jahani
- Institute for Data, Systems and Society, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Ridhi Kashyap
- Department of Sociology, University of Oxford, Oxford OX1 1JD, United Kingdom
- Nuffield College, University of Oxford, Oxford OX1 1NF, United Kingdom
- School of Anthropology and Museum Ethnography, University of Oxford, Oxford OX2 6PE, United Kingdom
| | - Antje Kirchner
- Program for Research in Survey Methodology, Survey Research Division, RTI International, Research Triangle Park, NC 27709
| | - Stephen McKay
- School of Social and Political Sciences, University of Lincoln, Brayford Pool, Lincoln LN6 7TS, United Kingdom
| | - Allison C Morgan
- Department of Computer Science, University of Colorado, Boulder, CO 80309
| | - Alex Pentland
- Media Lab, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Kivan Polimis
- Center for the Study of Demography and Ecology, University of Washington, Seattle, WA 98105
| | - Louis Raes
- Department of Economics, Tilburg School of Economics and Management, Tilburg University, 5037 AB Tilburg, The Netherlands
| | - Daniel E Rigobon
- Department of Operations Research and Financial Engineering, Princeton University, Princeton, NJ 08544
| | - Claudia V Roberts
- Department of Computer Science, Princeton University, Princeton, NJ 08544
| | - Diana M Stanescu
- Department of Politics, Princeton University,Princeton, NJ, 08544
| | - Yoshihiko Suhara
- Media Lab, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Adaner Usmani
- Department of Sociology, Harvard University, Cambridge, MA 02138
| | - Erik H Wang
- Department of Politics, Princeton University,Princeton, NJ, 08544
| | - Muna Adem
- Department of Sociology, Indiana University, Bloomington, IN 47405
| | - Abdulla Alhajri
- Department of Nuclear Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Bedoor AlShebli
- Computational Social Science Lab, Social Science Division, New York University Abu Dhabi, 129188 Abu Dhabi, United Arab Emirates
| | - Redwane Amin
- Bendheim Center for Finance, Princeton University, Princeton, NJ 08544
| | - Ryan B Amos
- Department of Computer Science, Princeton University, Princeton, NJ 08544
| | - Lisa P Argyle
- Department of Political Science, Brigham Young University, Provo, UT 84602
| | | | - Moritz Büchi
- Department of Communication and Media Research, University of Zurich, Zurich, Switzerland, ZH-8050
| | - Bo-Ryehn Chung
- Center for Statistics & Machine Learning, Princeton University, Princeton, NJ 08544
| | - William Eggert
- Department of Mechanical and Aerospace Engineering, Princeton University, Princeton, NJ 08544
| | - Gregory Faletto
- Statistics Group, Department of Data Sciences and Operations, Marshall School of Business, University of Southern California, Los Angeles, CA 90089
| | - Zhilin Fan
- Department of Statistics, Columbia University, New York, NY 10027
| | - Jeremy Freese
- Department of Sociology, Stanford University, Stanford, CA 94305
| | - Tejomay Gadgil
- Center for Data Science, New York University, New York, NY 10011
| | - Josh Gagné
- Department of Sociology, Stanford University, Stanford, CA 94305
| | - Yue Gao
- Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027
| | | | - Sonia P Hashim
- Department of Computer Science, Princeton University, Princeton, NJ 08544
| | - Sonia Hausen
- Department of Sociology, Stanford University, Stanford, CA 94305
| | - Guanhua He
- Department of Molecular Biology, Princeton University, Princeton, NJ 08544
| | - Kimberly Higuera
- Department of Sociology, Stanford University, Stanford, CA 94305
| | - Bernie Hogan
- Oxford Internet Institute, University of Oxford, Oxford OX1 3JS, United Kingdom
| | - Ilana M Horwitz
- Graduate School of Education, Stanford University, Stanford, CA, 94305
| | - Lisa M Hummel
- Department of Sociology, Stanford University, Stanford, CA 94305
| | - Naman Jain
- Department of Operations Research and Financial Engineering, Princeton University, Princeton, NJ 08544
| | - Kun Jin
- Department of Computer Science, Ohio State University, Columbus, OH 43210
| | - David Jurgens
- School of Information, University of Michigan, Ann Arbor, MI 48104
| | - Patrick Kaminski
- Department of Sociology, Indiana University, Bloomington, IN 47405
- Center for Complex Networks and Systems Research, Indiana University, Bloomington, IN 47405
| | - Areg Karapetyan
- Department of Computer Science, Masdar Institute, Khalifa University, 127788 Abu Dhabi, United Arab Emirates
- Research Institute for Mathematical Sciences, Kyoto University, Kyoto 606-8502, Japan
| | - E H Kim
- Department of Sociology, Stanford University, Stanford, CA 94305
| | - Ben Leizman
- Department of Computer Science, Princeton University, Princeton, NJ 08544
| | - Naijia Liu
- Department of Politics, Princeton University,Princeton, NJ, 08544
| | - Malte Möser
- Department of Computer Science, Princeton University, Princeton, NJ 08544
| | - Andrew E Mack
- Department of Politics, Princeton University,Princeton, NJ, 08544
| | - Mayank Mahajan
- Department of Computer Science, Princeton University, Princeton, NJ 08544
| | - Noah Mandell
- Department of Astrophysical Sciences, Princeton University, Princeton, NJ 08544
| | - Helge Marahrens
- Department of Sociology, Indiana University, Bloomington, IN 47405
| | | | - Viola Mocz
- Department of Neuroscience, Princeton University, Princeton, NJ 08544
| | | | - Ahmed Musse
- Department of Electrical Engineering, Princeton University, Princeton, NJ, 08544
| | - Qiankun Niu
- Bendheim Center for Finance, Princeton University, Princeton, NJ 08544
| | | | - Hamidreza Omidvar
- Department of Civil and Environmental Engineering, Princeton University, Princeton, NJ 08544
| | - Andrew Or
- Department of Computer Science, Princeton University, Princeton, NJ 08544
| | - Karen Ouyang
- Department of Computer Science, Princeton University, Princeton, NJ 08544
| | - Katy M Pinto
- Department of Sociology, California State University, Dominguez Hills, Carson, CA 90747
| | - Ethan Porter
- School of Media and Public Affairs, George Washington University, Washington, DC 20052
| | | | - Crystal Qian
- Department of Computer Science, Princeton University, Princeton, NJ 08544
| | - Tamkinat Rauf
- Department of Sociology, Stanford University, Stanford, CA 94305
| | - Anahit Sargsyan
- Social Science Division, New York University Abu Dhabi, 129188 Abu Dhabi, United Arab Emirates
| | - Thomas Schaffner
- Department of Computer Science, Princeton University, Princeton, NJ 08544
| | - Landon Schnabel
- Department of Sociology, Stanford University, Stanford, CA 94305
| | - Bryan Schonfeld
- Department of Politics, Princeton University,Princeton, NJ, 08544
| | - Ben Sender
- Department of Economics, Princeton University, Princeton, NJ 08544
| | - Jonathan D Tang
- Department of Computer Science, Princeton University, Princeton, NJ 08544
| | - Emma Tsurkov
- Department of Sociology, Stanford University, Stanford, CA 94305
| | - Austin van Loon
- Department of Sociology, Stanford University, Stanford, CA 94305
| | - Onur Varol
- Center for Complex Network Research, Northeastern University Networks Science Institute, Boston, MA 02115
- Luddy School of Informatics, Computing, & Engineering, Indiana University, Bloomington, IN 47408
| | - Xiafei Wang
- School of Social Work, David B. Falk College of Sport and Human Dynamics, Syracuse University, NY 13244
| | - Zhi Wang
- Luddy School of Informatics, Computing, & Engineering, Indiana University, Bloomington, IN 47408
- School of Public Health, Indiana University, Bloomington, IN 47408
| | - Julia Wang
- Department of Computer Science, Princeton University, Princeton, NJ 08544
| | - Flora Wang
- Department of Economics, Princeton University, Princeton, NJ 08544
| | - Samantha Weissman
- Department of Computer Science, Princeton University, Princeton, NJ 08544
| | - Kirstie Whitaker
- The Alan Turing Institute, London NW1 2DB, United Kingdom
- Department of Psychiatry, University of Cambridge, Cambridge CB2 0SZ, United Kingdom
| | - Maria K Wolters
- School of Informatics, University of Edinburgh, Edinburgh EH8 9AB, United Kingdom
| | - Wei Lee Woon
- Department of Marketplaces & Yield Data Science, Expedia Group, Seattle, WA 98119
| | - James Wu
- Department of the Applied Statistics, Social Science, and Humanities, New York University, New York, NY 10003
| | - Catherine Wu
- Department of Computer Science, Princeton University, Princeton, NJ 08544
| | - Kengran Yang
- Department of Civil and Environmental Engineering, Princeton University, Princeton, NJ 08544
| | - Jingwen Yin
- Department of Statistics, Columbia University, New York, NY 10027
| | - Bingyu Zhao
- Department of Engineering, University of Cambridge, Cambridge CB2 1PZ, United Kingdom
| | - Chenyun Zhu
- Department of Statistics, Columbia University, New York, NY 10027
| | - Jeanne Brooks-Gunn
- Department of Human Development, Teachers College, Columbia University, New York, NY 10027
- Department of Pediatrics, Vagelos College of Physicians and Surgeons, Columbia University, New York, NY 10032
| | - Barbara E Engelhardt
- Department of Computer Science, Princeton University, Princeton, NJ 08544
- Center for Statistics & Machine Learning, Princeton University, Princeton, NJ 08544
| | - Moritz Hardt
- Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, CA 94720
| | - Dean Knox
- Department of Politics, Princeton University,Princeton, NJ, 08544
| | - Karen Levy
- Department of Information Science, Cornell University, Ithaca, NY 14853
| | - Arvind Narayanan
- Department of Computer Science, Princeton University, Princeton, NJ 08544
| | | | - Duncan J Watts
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104
- Annenberg School of Communication, University of Pennsylvania, Philadelphia, PA 19104
- Operations, Information and Decisions Department, University of Pennsylvania, Philadelphia, PA 19104
| | - Sara McLanahan
- Department of Sociology, Princeton University, Princeton, NJ 08544;
| |
Collapse
|