1
|
Schippers EE, Smid W, Hoogsteder L, de Vogel V. Factor Analysis With Unusual Sexual Interests: A Replication Study in a Representative Population Sample. SEXUAL ABUSE : A JOURNAL OF RESEARCH AND TREATMENT 2024; 36:464-485. [PMID: 37729612 DOI: 10.1177/10790632231200841] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/22/2023]
Abstract
A previous study found a variety of unusual sexual interests to cluster in a five-factor structure, namely submission/masochism, forbidden sexual activities, dominance/sadism, mysophilia, and fetishism (Schippers et al., 2021). The current study was an empirical replication to examine whether these findings generalized to a representative population sample. An online, anonymous sample (N = 256) representative of the Dutch adult male population rated 32 unusual sexual interests on a scale from 1 (very unappealing) to 7 (very appealing). An exploratory factor analysis assessed whether similar factors would emerge as in the original study. A subsequent confirmatory factor analysis served to confirm the factor structure. Four slightly different factors of sexual interest were found: extreme, illegal and mysophilic sexual activities; light BDSM without real pain or suffering; heavy BDSM that may include pain or suffering; and illegal but lower-sentenced and fetishistic sexual activities. The model fit was acceptable. The representative replication sample was more sexually conservative and showed less sexual engagement than the original convenience sample. On a fundamental level, sexual interest in light BDSM activities and extreme, forbidden, and mysophilic activities seem to be relatively separate constructs.
Collapse
Affiliation(s)
- Eveline E Schippers
- Forensic Care Specialists, Utrecht, Netherlands
- Faculty of Psychology and Neuroscience, section Forensic Psychology, Maastricht University, Maastricht, Netherlands
| | - Wineke Smid
- Forensic Care Specialists, Utrecht, Netherlands
| | - Larissa Hoogsteder
- Forensic Care Specialists, Utrecht, Netherlands
- Faculty of Social and Behavioral Sciences, University of Amsterdam, Amsterdam, Netherlands
| | - Vivienne de Vogel
- Forensic Care Specialists, Utrecht, Netherlands
- Faculty of Psychology and Neuroscience, section Forensic Psychology, Maastricht University, Maastricht, Netherlands
| |
Collapse
|
2
|
Krefeld-Schwalb A, Sugerman ER, Johnson EJ. Exposing omitted moderators: Explaining why effect sizes differ in the social sciences. Proc Natl Acad Sci U S A 2024; 121:e2306281121. [PMID: 38466835 PMCID: PMC10962994 DOI: 10.1073/pnas.2306281121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2023] [Accepted: 01/26/2024] [Indexed: 03/13/2024] Open
Abstract
Policymakers increasingly rely on behavioral science in response to global challenges, such as climate change or global health crises. But applications of behavioral science face an important problem: Interventions often exert substantially different effects across contexts and individuals. We examine this heterogeneity for different paradigms that underlie many behavioral interventions. We study the paradigms in a series of five preregistered studies across one in-person and 10 online panels, with over 11,000 respondents in total. We find substantial heterogeneity across settings and paradigms, apply techniques for modeling the heterogeneity, and introduce a framework that measures typically omitted moderators. The framework's factors (Fluid Intelligence, Attentiveness, Crystallized Intelligence, and Experience) affect the effectiveness of many text-based interventions, producing different observed effect sizes and explaining variations across samples. Moderators are associated with effect sizes through two paths, with the intensity of the manipulation and with the effect of the manipulation directly. Our results motivate observing these moderators and provide a theoretical and empirical framework for understanding and predicting varying effect sizes in the social sciences.
Collapse
Affiliation(s)
- Antonia Krefeld-Schwalb
- Rotterdam School of Management, Department of Marketing Management, Erasmus University, Rotterdam3011LC, Netherlands
| | - Eli Rosen Sugerman
- Columbia Business School, Marketing Division, Columbia University, New York City, NY10027
| | - Eric J. Johnson
- Columbia Business School, Marketing Division, Columbia University, New York City, NY10027
| |
Collapse
|
3
|
Sanbonmatsu DM, Cooley EH, Posavac SS. The institutional impact of research challenges and constraints on psychology and other social and behavioral sciences. NEW IDEAS IN PSYCHOLOGY 2023. [DOI: 10.1016/j.newideapsych.2023.101014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/17/2023]
|
4
|
Walton GM, Murphy MC, Logel C, Yeager DS, Goyer JP, Brady ST, Emerson KTU, Paunesku D, Fotuhi O, Blodorn A, Boucher KL, Carter ER, Gopalan M, Henderson A, Kroeper KM, Murdock-Perriera LA, Reeves SL, Ablorh TT, Ansari S, Chen S, Fisher P, Galvan M, Gilbertson MK, Hulleman CS, Le Forestier JM, Lok C, Mathias K, Muragishi GA, Netter M, Ozier E, Smith EN, Thoman DB, Williams HE, Wilmot MO, Hartzog C, Li XA, Krol N. Where and with whom does a brief social-belonging intervention promote progress in college? Science 2023; 380:499-505. [PMID: 37141344 DOI: 10.1126/science.ade4420] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/06/2023]
Abstract
A promising way to mitigate inequality is by addressing students' worries about belonging. But where and with whom is this social-belonging intervention effective? Here we report a team-science randomized controlled experiment with 26,911 students at 22 diverse institutions. Results showed that the social-belonging intervention, administered online before college (in under 30 minutes), increased the rate at which students completed the first year as full-time students, especially among students in groups that had historically progressed at lower rates. The college context also mattered: The intervention was effective only when students' groups were afforded opportunities to belong. This study develops methods for understanding how student identities and contexts interact with interventions. It also shows that a low-cost, scalable intervention generalizes its effects to 749 4-year institutions in the United States.
Collapse
Affiliation(s)
- Gregory M Walton
- Department of Psychology, Stanford University, Stanford, CA, USA
| | - Mary C Murphy
- Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN, USA
| | - Christine Logel
- Department of Social Development Studies, Renison University College, University of Waterloo, Waterloo, ON, Canada
| | - David S Yeager
- Department of Psychology, University of Texas at Austin, Austin, TX, USA
| | - J Parker Goyer
- Department of Psychology, Stanford University, Stanford, CA, USA
| | - Shannon T Brady
- Department of Psychology, Wake Forest University, Winston-Salem, NC, USA
| | - Katherine T U Emerson
- Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN, USA
| | - David Paunesku
- Department of Psychology, Stanford University, Stanford, CA, USA
- The Project for Education Research that Scales (PERTS), San Francisco, CA, USA
| | - Omid Fotuhi
- Learning Research and Development Center, University of Pittsburgh, Pittsburgh, PA, USA
| | - Alison Blodorn
- Department of Psychology, Stanford University, Stanford, CA, USA
| | - Kathryn L Boucher
- Department of Psychological Sciences, University of Indianapolis, Indianapolis, IN, USA
| | | | - Maithreyi Gopalan
- Department of Education Policy Studies, The Pennsylvania State University, State College, PA, USA
| | - Amy Henderson
- Department of Psychology, Stanford University, Stanford, CA, USA
| | - Kathryn M Kroeper
- Department of Psychology, Sacred Heart University, Fairfield, CT, USA
| | | | | | - Tsotso T Ablorh
- Department of Clinical Psychology, University of Massachusetts Boston, Boston, MA, USA
| | - Shahana Ansari
- Department of Psychology, University of Hawai'i at Mānoa, Honolulu, HI, USA
| | | | - Peter Fisher
- Department of Psychology, University of California, Los Angeles, CA, USA
| | - Manuel Galvan
- Department of Psychology and Neuroscience, University of North Carolina, Chapel Hill, NC, USA
| | | | - Chris S Hulleman
- School of Education and Human Development, University of Virginia, Charlottesville, VA, USA
| | | | - Christopher Lok
- Department of Psychology, University of Waterloo, Waterloo, ON, Canada
| | - Katie Mathias
- Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN, USA
| | - Gregg A Muragishi
- Department of Psychology, University of Washington, Seattle, WA, USA
| | - Melanie Netter
- Department of Psychology, University of Texas at Austin, Austin, TX, USA
| | - Elise Ozier
- Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN, USA
| | - Eric N Smith
- Population Research Center, University of Texas at Austin, Austin, TX, USA
| | - Dustin B Thoman
- Department of Psychology, San Diego State University, San Diego, CA, USA
| | - Heidi E Williams
- Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN, USA
| | - Matthew O Wilmot
- Department of Psychology, The Ohio State University, Columbus, OH, USA
| | - Cassie Hartzog
- Department of Psychology, Stanford University, Stanford, CA, USA
| | - X Alice Li
- Department of Psychology, Stanford University, Stanford, CA, USA
| | - Natasha Krol
- Department of Psychology, Stanford University, Stanford, CA, USA
| |
Collapse
|
5
|
Mathur MB, Fox MP. Toward Open and Reproducible Epidemiology. Am J Epidemiol 2023; 192:658-664. [PMID: 36627249 PMCID: PMC10089067 DOI: 10.1093/aje/kwad007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2022] [Revised: 12/08/2022] [Accepted: 01/09/2023] [Indexed: 01/12/2023] Open
Abstract
Starting in the 2010s, researchers in the experimental social sciences rapidly began to adopt increasingly open and reproducible scientific practices. These practices include publicly sharing deidentified data when possible, sharing analytical code, and preregistering study protocols. Empirical evidence from the social sciences suggests such practices are feasible, can improve analytical reproducibility, and can reduce selective reporting. In academic epidemiology, adoption of open-science practices has been slower than in the social sciences (with some notable exceptions, such as registering clinical trials). Epidemiologic studies are often large, complex, conceived after data have already been collected, and difficult to replicate directly by collecting new data. These characteristics make it especially important to ensure their integrity and analytical reproducibility. Open-science practices can also pay immediate dividends to researchers' own work by clarifying scientific reasoning and encouraging well-documented, organized workflows. We consider how established epidemiologists and early-career researchers alike can help midwife a culture of open science in epidemiology through their research practices, mentorship, and editorial activities.
Collapse
Affiliation(s)
- Maya B Mathur
- Correspondence to Dr. Maya B. Mathur, Quantitative Sciences Unit, 3180 Porter Drive, Palo Alto, CA 94304 (e-mail: )
| | | |
Collapse
|
6
|
Hallsworth M. A manifesto for applying behavioural science. Nat Hum Behav 2023; 7:310-322. [PMID: 36941468 DOI: 10.1038/s41562-023-01555-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Accepted: 02/10/2023] [Indexed: 03/23/2023]
Abstract
Recent years have seen a rapid increase in the use of behavioural science to address the priorities of public and private sector actors. There is now a vibrant ecosystem of practitioners, teams and academics building on each other's findings across the globe. Their focus on robust evaluation means we know that this work has had an impact on important issues such as antimicrobial resistance, educational attainment and climate change. However, several critiques have also emerged; taken together, they suggest that applied behavioural science needs to evolve further over its next decade. This manifesto for the future of applied behavioural science looks at the challenges facing the field and sets out ten proposals to address them. Meeting these challenges will mean that behavioural science is better equipped to help to build policies, products and services on stronger empirical foundations-and thereby address the world's crucial challenges.
Collapse
|
7
|
Shek DTL, Dou D, Zhu X, Li X. Law abidance leadership education for university students in Hong Kong: Post-lecture evaluation. Front Psychol 2022; 13:994448. [PMID: 36389585 PMCID: PMC9650994 DOI: 10.3389/fpsyg.2022.994448] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2022] [Accepted: 10/12/2022] [Indexed: 11/03/2023] Open
Abstract
Law abidance is very important for effective leaders. Without law abidance, abuse of power and corruption would easily happen, which would eventually erode organizational health. To promote law abidance leadership in university students in Hong Kong, we developed a law abidance leadership program with 3 h of face-to-face lecture and 7 h of self-study of materials disturbed to students. To understand students' perception of the 3-h lecture, we conducted a post-lecture evaluation study using a 26-item measure (N = 914). Results showed that the scale possessed good reliability and validity. Confirmatory factor analyses showed that the assessment tool has three internally consistent and factorial invariant dimensions: program attributes, appreciation of law abidance, and teacher attributes. Regarding students' perception of the program, students were generally satisfied with the lecture attributes, including design, content, lecture atmosphere, teaching quality, and benefits to students. In particular, students agreed that the lecture helped them understand the importance and value of law abidance and national security; over 95% of the students indicated that they would try their best to serve as law-abiding citizens and socially responsible leaders. Multiple regression analyses showed that program attributes, appreciation of law abidance, and teacher attributes predicted overall satisfaction with the lecture. Qualitative comments of the student echoed the quantitative findings, with most of the comments being positive in nature. The present study replicated the findings reported previously. Local and international contextual factors relevant to the interpretations of the findings are highlighted.
Collapse
Affiliation(s)
- Daniel T. L. Shek
- Department of Applied Social Sciences, The Hong Kong Polytechnic University, Hung Hom, Hong Kong SAR, China
| | | | | | | |
Collapse
|
8
|
Hatch SG, Knopp K, Le Y, Allen MOT, Rothman K, Rhoades GK, Doss BD. Online relationship education for help-seeking low-income couples: A Bayesian replication and extension of the OurRelationship and ePREP programs. FAMILY PROCESS 2022; 61:1045-1061. [PMID: 34383314 DOI: 10.1111/famp.12710] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/20/2021] [Revised: 06/22/2021] [Accepted: 06/23/2021] [Indexed: 06/13/2023]
Abstract
Low-income couples are at increased risk for relationship instability and divorce. In response, online relationship education programs such as ePREP and OurRelationship have been developed to more easily reach this population. A previous trial indicated that these programs promote relationship functioning (Doss et al., 2020) and individual well-being (Roddy et al., 2020a). However, given that these effects were notably larger than previous studies of in-person relationship education and approached effect sizes observed in couple therapy, it is possible that the magnitude of these effects was somewhat spurious; therefore, these findings need replication. The current manuscript seeks to replicate these programs' previous effects on relationship functioning and determine whether these effects are stable. Using a sample of 671 low-income couples seeking relationship help (N = 1337 individuals) and Bayesian estimation, the current study replicated previous findings that the OurRelationship and ePREP programs offered with four coaching calls produced reliable improvements in relationship functioning relative to a 6-month waitlist control group. There were no statistically reliable differences between the two active interventions. Bayesian analyses indicated that the effects of the two online programs were larger than the average effects of in-person relationship education for low-income couples reported in previous studies, roughly equivalent to efficacy studies of in-person relationship education reported in previous studies, smaller than those that resulted from the OurRelationship program delivered to distressed couples without an income requirement and smaller than couple therapy.
Collapse
Affiliation(s)
- S Gabe Hatch
- Department of Psychology, University of Miami, Coral Gables, FL, USA
| | - Kayla Knopp
- Veteran Affairs San Diego Healthcare System, San Diego, CA, USA
- The University of California, San Diego, CA, USA
| | - Yunying Le
- Department of Human Development and Family Studies, The Pennsylvania State University, State College, PA, USA
| | | | - Karen Rothman
- Department of Psychology, University of Miami, Coral Gables, FL, USA
| | | | - Brian D Doss
- Department of Psychology, University of Miami, Coral Gables, FL, USA
| |
Collapse
|
9
|
DeKay ML, Rubinchik N, Li Z, De Boeck P. Accelerating Psychological Science With Metastudies: A Demonstration Using the Risky-Choice Framing Effect. PERSPECTIVES ON PSYCHOLOGICAL SCIENCE 2022; 17:1704-1736. [PMID: 35834353 DOI: 10.1177/17456916221079611] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
A metastudy is a set of many tiny studies (microstudies) created from a much larger collection of possibilities. Metastudies can yield many of the benefits of time-consuming replications and meta-analyses but more efficiently and with greater attention to generalizability and the causal effects of moderators. Statistical precision and power are higher than in studies with the same total sample size but with fewer conditions and more participants per condition. In this article, we describe metastudies and their benefits, demonstrate how to conduct a metastudy using the well-known risky-choice framing effect as an example, and report on the generalizability of that effect. In three metastudies (total N = 2,338), the framing effect generalized well across most of the potential moderators tested, as was expected. Surprisingly, however, the effect was up to twice as large when the certain option was replaced with a slightly risky option; prospect theory predicts the opposite, and fuzzy-trace theory predicts no difference. Metastudies provide a relatively quick and not-so-painful way of examining an effect's generalizability without waiting for a meta-analysis. Both individual labs and multilab networks are encouraged to shift from traditional studies to metastudies.
Collapse
Affiliation(s)
| | | | - Zhaojun Li
- Department of Psychology, The Ohio State University
| | | |
Collapse
|
10
|
Yeager DS, Bryan CJ, Gross JJ, Murray JS, Krettek Cobb D, H F Santos P, Gravelding H, Johnson M, Jamieson JP. A synergistic mindsets intervention protects adolescents from stress. Nature 2022; 607:512-520. [PMID: 35794485 PMCID: PMC9258473 DOI: 10.1038/s41586-022-04907-7] [Citation(s) in RCA: 27] [Impact Index Per Article: 13.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/22/2021] [Accepted: 05/25/2022] [Indexed: 11/09/2022]
Abstract
Social-evaluative stressors-experiences in which people feel they could be judged negatively-pose a major threat to adolescent mental health1-3 and can cause young people to disengage from stressful pursuits, resulting in missed opportunities to acquire valuable skills. Here we show that replicable benefits for the stress responses of adolescents can be achieved with a short (around 30-min), scalable 'synergistic mindsets' intervention. This intervention, which is a self-administered online training module, synergistically targets both growth mindsets4 (the idea that intelligence can be developed) and stress-can-be-enhancing mindsets5 (the idea that one's physiological stress response can fuel optimal performance). In six double-blind, randomized, controlled experiments that were conducted with secondary and post-secondary students in the United States, the synergistic mindsets intervention improved stress-related cognitions (study 1, n = 2,717; study 2, n = 755), cardiovascular reactivity (study 3, n = 160; study 4, n = 200), daily cortisol levels (study 5, n = 118 students, n = 1,213 observations), psychological well-being (studies 4 and 5), academic success (study 5) and anxiety symptoms during the 2020 COVID-19 lockdowns (study 6, n = 341). Heterogeneity analyses (studies 3, 5 and 6) and a four-cell experiment (study 4) showed that the benefits of the intervention depended on addressing both mindsets-growth and stress-synergistically. Confidence in these conclusions comes from a conservative, Bayesian machine-learning statistical method for detecting heterogeneous effects6. Thus, our research has identified a treatment for adolescent stress that could, in principle, be scaled nationally at low cost.
Collapse
Affiliation(s)
- David S Yeager
- Department of Psychology and Behavioral Science and Policy Institute, University of Texas at Austin, Austin, TX, USA.
| | - Christopher J Bryan
- Department of Business, Government, and Society and Behavioral Science and Policy Institute, University of Texas at Austin, Austin, TX, USA.
| | - James J Gross
- Department of Psychology, Stanford University, Stanford, CA, USA
| | - Jared S Murray
- Department of Information, Risk and Operations Management, University of Texas at Austin, Austin, TX, USA
- Department of Statistics and Data Sciences, University of Texas at Austin, Austin, TX, USA
| | | | - Pedro H F Santos
- Department of Information, Risk and Operations Management, University of Texas at Austin, Austin, TX, USA
| | - Hannah Gravelding
- Department of Psychology, University of Rochester, Rochester, NY, USA
| | - Meghann Johnson
- Department of Psychology and Behavioral Science and Policy Institute, University of Texas at Austin, Austin, TX, USA
| | - Jeremy P Jamieson
- Department of Psychology, University of Rochester, Rochester, NY, USA.
| |
Collapse
|
11
|
McShane BB, Gelman A. Selecting on statistical significance and practical importance is wrong. JOURNAL OF INFORMATION TECHNOLOGY 2022. [DOI: 10.1177/02683962221086297] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
|
12
|
Shek DTL, Dou D, Zhu X, Li X, Tan L. Materialism, Egocentrism and Delinquent Behavior in Chinese Adolescents in Mainland China: A Short-Term Longitudinal Study. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2022; 19:ijerph19084912. [PMID: 35457779 PMCID: PMC9026470 DOI: 10.3390/ijerph19084912] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Received: 03/15/2022] [Revised: 04/15/2022] [Accepted: 04/16/2022] [Indexed: 02/01/2023]
Abstract
Although research generally showed that holding materialistic beliefs would lead to poor developmental outcomes, few studies have used adolescent delinquency as an outcome measure. In addition, the intervening processes between materialism and adolescent developmental outcomes are unclear. In particular, it is not clear how materialistic beliefs influence egocentrism and adolescent delinquency. Methodologically, the existing studies have several weaknesses, including small samples, cross-sectional research designs, and being limited to people living in Western cultures. Using two waves of data collected from Sichuan, China (N = 4981), we studied the predictive effect of adolescent materialism on delinquency and the mediating role of egocentrism. Over two occasions separated by six months, students aged 11 and above responded to a questionnaire evaluating adolescent materialism, egocentrism, and delinquency (mean Wave 1 age = 13.15, range between 11 and 20.38). Results of multiple regression analyses suggested that materialism at Time 1 positively predicted Time 2 egocentrism. Additionally, Time 1 materialism positively predicted the level and change in Time 2 delinquency. Finally, based on 5000 bootstrap samples with gender, age, ethnic group, and Time 1 delinquent behavior as covariates, PROCESS analyses showed that egocentrism partially mediated the influence of Time 1 materialism delinquency and its change at Time 2. This study suggests that materialistic beliefs shape egocentrism, which further strengthens adolescent delinquent behavior. This study also replicates the findings of a pioneer study in China reported previously.
Collapse
|
13
|
McShane BB, Böckenholt U, Hansen KT. Variation and Covariation in Large-scale Replication Projects: An Evaluation of Replicability. J Am Stat Assoc 2022. [DOI: 10.1080/01621459.2022.2054816] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
|
14
|
Abstract
It is often claimed that only experiments can support strong causal inferences and therefore they should be privileged in the behavioral sciences. We disagree. Overvaluing experiments results in their overuse both by researchers and decision makers and in an underappreciation of their shortcomings. Neglect of other methods often follows. Experiments can suggest whether X causes Y in a specific experimental setting; however, they often fail to elucidate either the mechanisms responsible for an effect or the strength of an effect in everyday natural settings. In this article, we consider two overarching issues. First, experiments have important limitations. We highlight problems with external, construct, statistical-conclusion, and internal validity; replicability; and conceptual issues associated with simple X causes Y thinking. Second, quasi-experimental and nonexperimental methods are absolutely essential. As well as themselves estimating causal effects, these other methods can provide information and understanding that goes beyond that provided by experiments. A research program progresses best when experiments are not treated as privileged but instead are combined with these other methods.
Collapse
Affiliation(s)
- Ed Diener
- Department of Psychology, University of Utah.,Department of Psychology, University of Virginia.,Gallup, Washington, D.C
| | - Robert Northcott
- Department of Philosophy, Birkbeck College, University of London
| | | | | |
Collapse
|
15
|
Habiger J, Liang Y. Publication Policies for Replicable Research and the Community-Wide False Discovery Rate. AM STAT 2022. [DOI: 10.1080/00031305.2021.1999857] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/01/2022]
Affiliation(s)
- Joshua Habiger
- Department of Statistics, Oklahoma State University, Stillwater, OK
| | - Ye Liang
- Department of Statistics, Oklahoma State University, Stillwater, OK
| |
Collapse
|
16
|
Yeager DS, Carroll JM, Buontempo J, Cimpian A, Woody S, Crosnoe R, Muller C, Murray J, Mhatre P, Kersting N, Hulleman C, Kudym M, Murphy M, Duckworth AL, Walton GM, Dweck CS. Teacher Mindsets Help Explain Where a Growth-Mindset Intervention Does and Doesn't Work. Psychol Sci 2022; 33:18-32. [PMID: 34936529 PMCID: PMC8985222 DOI: 10.1177/09567976211028984] [Citation(s) in RCA: 32] [Impact Index Per Article: 16.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2020] [Accepted: 05/18/2021] [Indexed: 12/24/2022] Open
Abstract
A growth-mindset intervention teaches the belief that intellectual abilities can be developed. Where does the intervention work best? Prior research examined school-level moderators using data from the National Study of Learning Mindsets (NSLM), which delivered a short growth-mindset intervention during the first year of high school. In the present research, we used data from the NSLM to examine moderation by teachers' mindsets and answer a new question: Can students independently implement their growth mindsets in virtually any classroom culture, or must students' growth mindsets be supported by their teacher's own growth mindsets (i.e., the mindset-plus-supportive-context hypothesis)? The present analysis (9,167 student records matched with 223 math teachers) supported the latter hypothesis. This result stood up to potentially confounding teacher factors and to a conservative Bayesian analysis. Thus, sustaining growth-mindset effects may require contextual supports that allow the proffered beliefs to take root and flourish.
Collapse
Affiliation(s)
- David S. Yeager
- Department of Psychology, The
University of Texas at Austin
- Population Research Center, The
University of Texas at Austin
| | - Jamie M. Carroll
- Population Research Center, The
University of Texas at Austin
- Department of Sociology, The University
of Texas at Austin
| | - Jenny Buontempo
- Population Research Center, The
University of Texas at Austin
| | | | - Spencer Woody
- Department of Integrative Biology, The
University of Texas at Austin
| | - Robert Crosnoe
- Population Research Center, The
University of Texas at Austin
- Department of Sociology, The University
of Texas at Austin
| | - Chandra Muller
- Population Research Center, The
University of Texas at Austin
- Department of Sociology, The University
of Texas at Austin
| | - Jared Murray
- Department of Information, Risk, and
Operations Management, The University of Texas at Austin
| | - Pratik Mhatre
- Population Research Center, The
University of Texas at Austin
| | - Nicole Kersting
- Department of Teaching, Learning and
Sociocultural Studies, The University of Arizona
| | - Christopher Hulleman
- Department of Educational Leadership,
Policy, and Foundations, University of Virginia
| | - Molly Kudym
- Population Research Center, The
University of Texas at Austin
- Department of Sociology, The University
of Texas at Austin
| | - Mary Murphy
- Department of Psychological and Brain
Sciences, Indiana University Bloomington
| | | | | | | |
Collapse
|
17
|
Moving to a world beyond p-value < 0.05: a guide for business researchers. REVIEW OF MANAGERIAL SCIENCE 2021. [DOI: 10.1007/s11846-021-00504-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
|
18
|
Highhouse S, Freier LM, Stevenor BA, Shea MA, Childers M, Melick SR. Failure to replicate the basic dilution effect in performance prediction. INTERNATIONAL JOURNAL OF SELECTION AND ASSESSMENT 2021. [DOI: 10.1111/ijsa.12344] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
Affiliation(s)
- Scott Highhouse
- Department of Psychology Bowling Green State University Bowling Green Ohio USA
| | - Lindsey M. Freier
- Department of Psychology Bowling Green State University Bowling Green Ohio USA
| | - Brent A. Stevenor
- Department of Psychology Bowling Green State University Bowling Green Ohio USA
| | - Michael A. Shea
- Department of Psychology Bowling Green State University Bowling Green Ohio USA
| | - Marie Childers
- Department of Psychology Bowling Green State University Bowling Green Ohio USA
| | - Sarah R. Melick
- Department of Psychology Bowling Green State University Bowling Green Ohio USA
| |
Collapse
|
19
|
Behavioural science is unlikely to change the world without a heterogeneity revolution. Nat Hum Behav 2021; 5:980-989. [PMID: 34294901 PMCID: PMC8928154 DOI: 10.1038/s41562-021-01143-3] [Citation(s) in RCA: 79] [Impact Index Per Article: 26.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2020] [Accepted: 05/21/2021] [Indexed: 02/06/2023]
Abstract
In the past decade, behavioural science has gained influence in policymaking but suffered a crisis of confidence in the replicability of its findings. Here, we describe a nascent heterogeneity revolution that we believe these twin historical trends have triggered. This revolution will be defined by the recognition that most treatment effects are heterogeneous, so the variation in effect estimates across studies that defines the replication crisis is to be expected as long as heterogeneous effects are studied without a systematic approach to sampling and moderation. When studied systematically, heterogeneity can be leveraged to build more complete theories of causal mechanism that could inform nuanced and dependable guidance to policymakers. We recommend investment in shared research infrastructure to make it feasible to study behavioural interventions in heterogeneous and generalizable samples, and suggest low-cost steps researchers can take immediately to avoid being misled by heterogeneity and begin to learn from it instead.
Collapse
|
20
|
Gollwitzer M, Schwabe J. Context Dependency as a Predictor of Replicability. REVIEW OF GENERAL PSYCHOLOGY 2021. [DOI: 10.1177/10892680211015635] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
We scrutinize the argument that unsuccessful replications—and heterogeneous effect sizes more generally—may reflect an underappreciated influence of context characteristics. Notably, while some of these context characteristics may be conceptually irrelevant (as they merely affect psychometric properties of the measured/manipulated variables), others are conceptually relevant as they qualify a theory. Here, we present a conceptual and analytical framework that allows researchers to empirically estimate the extent to which effect size heterogeneity is due to conceptually relevant versus irrelevant context characteristics. According to this framework, contextual characteristics are conceptually relevant when the observed heterogeneity of effect sizes cannot be attributed to psychometric properties. As an illustrative example, we demonstrate that the observed heterogeneity of the “moral typecasting” effect, which had been included in the ManyLabs 2 replication project, is more likely attributable to conceptually relevant rather than irrelevant context characteristics, which suggests that the psychological theory behind this effect may need to be specified. In general, we argue that context dependency should be taken more seriously and treated more carefully by replication research.
Collapse
Affiliation(s)
- Mario Gollwitzer
- Department of Psychology, Ludwig-Maximilians-Universität, Munich, Germany
| | - Johannes Schwabe
- Department of Psychology, Ludwig-Maximilians-Universität, Munich, Germany
| |
Collapse
|
21
|
Sanbonmatsu DM, Cooley EH, Butner JE. The Impact of Complexity on Methods and Findings in Psychological Science. Front Psychol 2021; 11:580111. [PMID: 33551904 PMCID: PMC7859482 DOI: 10.3389/fpsyg.2020.580111] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/04/2020] [Accepted: 12/22/2020] [Indexed: 11/13/2022] Open
Abstract
The study of human behavior is severely hampered by logistical problems, ethical and legal constraints, and funding shortfalls. However, the biggest difficulty of conducting social and behavioral research is the extraordinary complexity of the study phenomena. In this article, we review the impact of complexity on research design, hypothesis testing, measurement, data analyses, reproducibility, and the communication of findings in psychological science. The systematic investigation of the world often requires different approaches because of the variability in complexity. Confirmatory testing, multi-factorial designs, survey methods, large samples, and modeling are frequently needed to study complex social and behavioral topics. Complexity impedes the measurement of general constructs, the reproducibility of results and scientific reporting, and the general rigor of research. Many of the benchmarks established by classic work in physical science are not attainable in studies of more complex phenomena. Consequently, the standards used to evaluate scientific research should be tethered to the complexity of the study topic.
Collapse
Affiliation(s)
- David M Sanbonmatsu
- Department of Psychology, University of Utah, Salt Lake City, UT, United States
| | - Emily H Cooley
- Department of Psychology, University of Utah, Salt Lake City, UT, United States
| | - Jonathan E Butner
- Department of Psychology, University of Utah, Salt Lake City, UT, United States
| |
Collapse
|
22
|
Greenland S. Analysis goals, error-cost sensitivity, and analysis hacking: Essential considerations in hypothesis testing and multiple comparisons. Paediatr Perinat Epidemiol 2021; 35:8-23. [PMID: 33269490 DOI: 10.1111/ppe.12711] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/24/2020] [Accepted: 06/28/2020] [Indexed: 01/03/2023]
Abstract
The "replication crisis" has been attributed to perverse incentives that lead to selective reporting and misinterpretations of P-values and confidence intervals. A crude fix offered for this problem is to lower testing cut-offs (α levels), either directly or in the form of null-biased multiple comparisons procedures such as naïve Bonferroni adjustments. Methodologists and statisticians have expressed positions that range from condemning all such procedures to demanding their application in almost all analyses. Navigating between these unjustifiable extremes requires defining analysis goals precisely enough to separate inappropriate from appropriate adjustments. To meet this need, I here review issues arising in single-parameter inference (such as error costs and loss functions) that are often skipped in basic statistics, yet are crucial to understanding controversies in testing and multiple comparisons. I also review considerations that should be made when examining arguments for and against modifications of decision cut-offs and adjustments for multiple comparisons. The goal is to provide researchers a better understanding of what is assumed by each side and to enable recognition of hidden assumptions. Basic issues of goal specification and error costs are illustrated with simple fixed cut-off hypothesis testing scenarios. These illustrations show how adjustment choices are extremely sensitive to implicit decision costs, making it inevitable that different stakeholders will vehemently disagree about what is necessary or appropriate. Because decisions cannot be justified without explicit costs, resolution of inference controversies is impossible without recognising this sensitivity. Pre-analysis statements of funding, scientific goals, and analysis plans can help counter demands for inappropriate adjustments, and can provide guidance as to what adjustments are advisable. Hierarchical (multilevel) regression methods (including Bayesian, semi-Bayes, and empirical-Bayes methods) provide preferable alternatives to conventional adjustments, insofar as they facilitate use of background information in the analysis model, and thus can provide better-informed estimates on which to base inferences and decisions.
Collapse
Affiliation(s)
- Sander Greenland
- Department of Epidemiology and Department of Statistics, University of California, Los Angeles, CA, USA
| |
Collapse
|
23
|
Howe ML, Otgaar H. Theoretically important failures to reject the null hypothesis: introduction to a special section of Memory. Memory 2020; 28:837-838. [PMID: 32840465 DOI: 10.1080/09658211.2020.1794669] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
Affiliation(s)
- Mark L Howe
- Department of Psychology, City, University of London, UK
| | - Henry Otgaar
- Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, The Netherlands
| |
Collapse
|
24
|
Brandt MJ, Kuppens T, Spears R, Andrighetto L, Autin F, Babincak P, Badea C, Bae J, Batruch A, Becker JC, Bocian K, Bodroža B, Bourguignon D, Bukowski M, Butera F, Butler SE, Chryssochoou X, Conway P, Crawford JT, Croizet J, de Lemus S, Degner J, Dragon P, Durante F, Easterbrook MJ, Essien I, Forgas JP, González R, Graf S, Halama P, Han G, Hong RY, Houdek P, Igou ER, Inbar Y, Jetten J, Jimenez Leal W, Jiménez‐Moya G, Karunagharan JK, Kende A, Korzh M, Laham SM, Lammers J, Lim L, Manstead ASR, Međedović J, Melton ZJ, Motyl M, Ntani S, Owuamalam CK, Peker M, Platow MJ, Prims JP, Reyna C, Rubin M, Saab R, Sankaran S, Shepherd L, Sibley CG, Sobkow A, Spruyt B, Stroebaek P, Sümer N, Sweetman J, Teixeira CP, Toma C, Ujhelyi A, van der Toorn J, van Hiel A, Vásquez‐Echeverría A, Vazquez A, Vianello M, Vranka M, Yzerbyt V, Zimmerman JL. Subjective status and perceived legitimacy across countries. EUROPEAN JOURNAL OF SOCIAL PSYCHOLOGY 2020; 50:921-942. [PMID: 32999511 PMCID: PMC7507836 DOI: 10.1002/ejsp.2694] [Citation(s) in RCA: 21] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/10/2019] [Revised: 02/14/2020] [Accepted: 05/21/2020] [Indexed: 11/28/2022]
Abstract
The relationships between subjective status and perceived legitimacy are important for understanding the extent to which people with low status are complicit in their oppression. We use novel data from 66 samples and 30 countries (N = 12,788) and find that people with higher status see the social system as more legitimate than those with lower status, but there is variation across people and countries. The association between subjective status and perceived legitimacy was never negative at any levels of eight moderator variables, although the positive association was sometimes reduced. Although not always consistent with hypotheses, group identification, self-esteem, and beliefs in social mobility were all associated with perceived legitimacy among people who have low subjective status. These findings enrich our understanding of the relationship between social status and legitimacy.
Collapse
Affiliation(s)
| | | | | | | | | | | | | | | | | | | | - Konrad Bocian
- Sopot Faculty of PsychologySWPS University of Social Sciences and Humanities
| | - Bojana Bodroža
- Department of PsychologyFaculty of PhilosophyUniversity of Novi SadNovi SadSerbia
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Anna Kende
- ELTE Eötvös Loránd UniversityBudapestHungary
| | | | | | | | | | | | - Janko Međedović
- Institute of Criminological and Sociological ResearchBelgradeSerbia
| | | | | | | | | | | | | | | | | | | | | | | | | | | | - Agata Sobkow
- Wroclaw Faculty of PsychologySWPS University of Social Sciences and Humanities
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
Collapse
|
25
|
|
26
|
Knight EL, McShane BB, Kutlikova HH, Morales PJ, Christian CB, Harbaugh WT, Mayr U, Ortiz TL, Gilbert K, Ma-Kellams C, Riečanský I, Watson NV, Eisenegger C, Lamm C, Mehta PH, Carré JM. Weak and Variable Effects of Exogenous Testosterone on Cognitive Reflection Test Performance in Three Experiments: Commentary on Nave, Nadler, Zava, and Camerer (2017). Psychol Sci 2020; 31:890-897. [PMID: 32586198 DOI: 10.1177/0956797619885607] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022] Open
Affiliation(s)
- Erik L Knight
- Department of Psychology, University of Oregon.,Center for Healthy Aging, The Pennsylvania State University
| | | | - Hana H Kutlikova
- Department of Cognition, Emotion, and Methods in Psychology, University of Vienna
| | | | | | | | - Ulrich Mayr
- Department of Psychology, University of Oregon
| | | | | | | | - Igor Riečanský
- Department of Cognition, Emotion, and Methods in Psychology, University of Vienna.,Centre of Experimental Medicine, Slovak Academy of Sciences
| | | | - Christoph Eisenegger
- Department of Cognition, Emotion, and Methods in Psychology, University of Vienna
| | - Claus Lamm
- Department of Cognition, Emotion, and Methods in Psychology, University of Vienna
| | - Pranjal H Mehta
- Department of Psychology, University of Oregon.,Department of Experimental Psychology, University College London
| | | |
Collapse
|
27
|
Colling LJ, Szűcs D, De Marco D, Cipora K, Ulrich R, Nuerk HC, Soltanlou M, Bryce D, Chen SC, Schroeder PA, Henare DT, Chrystall CK, Corballis PM, Ansari D, Goffin C, Sokolowski HM, Hancock PJB, Millen AE, Langton SRH, Holmes KJ, Saviano MS, Tummino TA, Lindemann O, Zwaan RA, Lukavský J, Becková A, Vranka MA, Cutini S, Mammarella IC, Mulatti C, Bell R, Buchner A, Mieth L, Röer JP, Klein E, Huber S, Moeller K, Ocampo B, Lupiáñez J, Ortiz-Tudela J, de la Fuente J, Santiago J, Ouellet M, Hubbard EM, Toomarian EY, Job R, Treccani B, McShane BB. Registered Replication Report on Fischer, Castel, Dodd, and Pratt (2003). ADVANCES IN METHODS AND PRACTICES IN PSYCHOLOGICAL SCIENCE 2020. [DOI: 10.1177/2515245920903079] [Citation(s) in RCA: 21] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/11/2022]
Abstract
The attentional spatial-numerical association of response codes (Att-SNARC) effect (Fischer, Castel, Dodd, & Pratt, 2003)—the finding that participants are quicker to detect left-side targets when the targets are preceded by small numbers and quicker to detect right-side targets when they are preceded by large numbers—has been used as evidence for embodied number representations and to support strong claims about the link between number and space (e.g., a mental number line). We attempted to replicate Experiment 2 of Fischer et al. by collecting data from 1,105 participants at 17 labs. Across all 1,105 participants and four interstimulus-interval conditions, the proportion of times the effect we observed was positive (i.e., directionally consistent with the original effect) was .50. Further, the effects we observed both within and across labs were minuscule and incompatible with those observed by Fischer et al. Given this, we conclude that we failed to replicate the effect reported by Fischer et al. In addition, our analysis of several participant-level moderators (finger-counting habits, reading and writing direction, handedness, and mathematics fluency and mathematics anxiety) revealed no substantial moderating effects. Our results indicate that the Att-SNARC effect cannot be used as evidence to support strong claims about the link between number and space.
Collapse
|
28
|
Abstract
Replication is an important contemporary issue in psychological research, and there is great interest in ways of assessing replicability, in particular, retrospectively via prior studies. The average power of a set of prior studies is a quantity that has attracted considerable attention for this purpose, and techniques to estimate this quantity via a meta-analytic approach have recently been proposed. In this article, we have two aims. First, we clarify the nature of average power and its implications for replicability. We explain that average power is not relevant to the replicability of actual prospective replication studies. Instead, it relates to efforts in the history of science to catalogue the power of prior studies. Second, we evaluate the statistical properties of point estimates and interval estimates of average power obtained via the meta-analytic approach. We find that point estimates of average power are too variable and inaccurate for use in application. We also find that the width of interval estimates of average power depends on the corresponding point estimates; consequently, the width of an interval estimate of average power cannot serve as an independent measure of the precision of the point estimate. Our findings resolve a seeming puzzle posed by three estimates of the average power of the power-posing literature obtained via the meta-analytic approach.
Collapse
|
29
|
Abstract
Throughout the last decade, the so-called replication crisis has stimulated many researchers to conduct large-scale replication projects. With data from four of these projects, we computed probabilistic forecasts of the replication outcomes, which we then evaluated regarding discrimination, calibration and sharpness. A novel model, which can take into account both inflation and heterogeneity of effects, was used and predicted the effect estimate of the replication study with good performance in two of the four data sets. In the other two data sets, predictive performance was still substantially improved compared to the naive model which does not consider inflation and heterogeneity of effects. The results suggest that many of the estimates from the original studies were inflated, possibly caused by publication bias or questionable research practices, and also that some degree of heterogeneity between original and replication effects should be expected. Moreover, the results indicate that the use of statistical significance as the only criterion for replication success may be questionable, since from a predictive viewpoint, non-significant replication results are often compatible with significant results from the original study. The developed statistical methods as well as the data sets are available in the R package ReplicationSuccess.
Collapse
Affiliation(s)
- Samuel Pawel
- Epidemiology, Biostatistics and Prevention Institute (EBPI), Center for Reproducible Science (CRS), University of Zurich, Zurich, Switzerland
- * E-mail:
| | - Leonhard Held
- Epidemiology, Biostatistics and Prevention Institute (EBPI), Center for Reproducible Science (CRS), University of Zurich, Zurich, Switzerland
| |
Collapse
|
30
|
Abstract
Abstract. Replication studies, pre-registration, and increases in statistical power will likely improve the reliability of scientific evidence. However, these measures face critical limitations in populations that are inherently difficult to study. Members of difficult-to-study populations (e.g., patients, children, non-human animals) are less accessible to researchers, which typically results in small-sample studies that are infeasible to replicate. Nevertheless, meta-analyses on clinical neuropsychological data suggest that difficult-to-study populations can be studied in a reliable way. These analyses often produce unbiased effect-size estimates despite aggregating across severely underpowered original studies. This finding can be attributed to a neuropsychological research culture involving the non-selective reporting of results from standardized and validated test procedures. Consensus guidelines, test manuals, and psychometric evidence constrain the methodological choices made by neuropsychologists, who regularly report the results from neuropsychological test batteries irrespective of their statistical significance or novelty. Comparable shifts toward more standardization and validation, complete result reports, and between-lab collaborations can allow for a meaningful and reliable study of psychological phenomena in other difficult-to-study populations.
Collapse
Affiliation(s)
- Florian Lange
- Behavioral Engineering Research Group, KU Leuven, Belgium
| |
Collapse
|
31
|
Maindonald J. Book Review. AUST NZ J STAT 2019. [DOI: 10.1111/anzs.12281] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
32
|
Panagopoulos C, van der Linden S. Political identity moderates the effect of watchful eyes on voter mobilization: A reply to Matland and Murray (2019). SOCIAL INFLUENCE 2019. [DOI: 10.1080/15534510.2019.1698123] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
Affiliation(s)
| | - Sander van der Linden
- Department of Psychology, School of Biological Sciences, University of Cambridge, Cambridge, UK
| |
Collapse
|
33
|
Mukherjee S. Revise the Belief in Loss Aversion. Front Psychol 2019; 10:2723. [PMID: 31849797 PMCID: PMC6902077 DOI: 10.3389/fpsyg.2019.02723] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2019] [Accepted: 11/18/2019] [Indexed: 11/16/2022] Open
|
34
|
McShane BB, Böckenholt U. Enriching Meta-Analytic Models of Summary Data: A Thought Experiment and Case Study. ADVANCES IN METHODS AND PRACTICES IN PSYCHOLOGICAL SCIENCE 2019. [DOI: 10.1177/2515245919884304] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
Meta-analysis typically involves the analysis of summary data (e.g., means, standard deviations, and sample sizes) from a set of studies via a statistical model that is a special case of a hierarchical (or multilevel) model. Unfortunately, the common summary-data approach to meta-analysis used in psychological research is often employed in settings where the complexity of the data warrants alternative approaches. In this article, we propose a thought experiment that can lead meta-analysts to move away from the common summary-data approach to meta-analysis and toward richer and more appropriate summary-data approaches when the complexity of the data warrants it. Specifically, we propose that it can be extremely fruitful for meta-analysts to act as if they possess the individual-level data from the studies and consider what model specifications they might fit even when they possess only summary data. This thought experiment is justified because (a) the analysis of the individual-level data from the studies via a hierarchical model is considered the “gold standard” for meta-analysis and (b) for a wide variety of cases common in meta-analysis, the summary-data and individual-level-data approaches are, by a principle known as statistical sufficiency, equivalent when the underlying models are appropriately specified. We illustrate the value of our thought experiment via a case study that evolves across five parts that cover a wide variety of data settings common in meta-analysis.
Collapse
Affiliation(s)
| | - Ulf Böckenholt
- Marketing Department, Kellogg School of Management, Northwestern University
| |
Collapse
|
35
|
Segal BD. Toward Replicability With Confidence Intervals for the Exceedance Probability. AM STAT 2019. [DOI: 10.1080/00031305.2019.1678521] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
|
36
|
Staggs VS. Why statisticians are abandoning statistical significance. Res Nurs Health 2019; 42:159-160. [PMID: 30963599 DOI: 10.1002/nur.21947] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2019] [Accepted: 04/01/2019] [Indexed: 11/05/2022]
Affiliation(s)
- Vincent S Staggs
- Research in Nursing & Health, Faculty Biostatistician, Children's Mercy Kansas City; Associate Professor of Pediatrics, University of Missouri-Kansas City, Kansas City, Missouri
| |
Collapse
|
37
|
|
38
|
Tackett JL, Brandes CM, King KM, Markon KE. Psychology's Replication Crisis and Clinical Psychological Science. Annu Rev Clin Psychol 2019; 15:579-604. [PMID: 30673512 DOI: 10.1146/annurev-clinpsy-050718-095710] [Citation(s) in RCA: 61] [Impact Index Per Article: 12.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Despite psychological scientists' increasing interest in replicability, open science, research transparency, and the improvement of methods and practices, the clinical psychology community has been slow to engage. This has been shifting more recently, and with this review, we hope to facilitate this emerging dialogue. We begin by examining some potential areas of weakness in clinical psychology in terms of methods, practices, and evidentiary base. We then discuss a select overview of solutions, tools, and current concerns of the reform movement from a clinical psychological science perspective. We examine areas of clinical science expertise (e.g., implementation science) that should be leveraged to inform open science and reform efforts. Finally, we reiterate the call to clinical psychologists to increase their efforts toward reform that can further improve the credibility of clinical psychological science.
Collapse
Affiliation(s)
- Jennifer L Tackett
- Department of Psychology, Northwestern University, Evanston, Illinois 60208, USA;
| | - Cassandra M Brandes
- Department of Psychology, Northwestern University, Evanston, Illinois 60208, USA;
| | - Kevin M King
- Department of Psychology, University of Washington, Seattle, Washington 98195, USA
| | - Kristian E Markon
- Department of Psychological and Brain Sciences, University of Iowa, Iowa City, Iowa 52242, USA
| |
Collapse
|