HARKing: Hypothesizing After the Results are Known

Best Practice for MRI Diagnostic Accuracy Research With Lessons and Examples from the LI-RADS Individual Participant Data Group

Medical imaging diagnostic test accuracy research is strengthened by adhering to best practices for study design, data collection, data documentation, and study reporting. In this review, key elements of such research are discussed, and specific recommendations provided for optimizing diagnostic accuracy study execution to improve uniformity, minimize common sources of bias and avoid potential pitfalls. Examples are provided regarding study methodology and data collection practices based on insights gained by the liver imaging reporting and data system (LI-RADS) individual participant data group, who have evaluated raw data from numerous MRI diagnostic accuracy studies for risk of bias and data integrity. The goal of this review is to outline strategies for investigators to improve research practices, and to help reviewers and readers better contextualize a study's findings while understanding its limitations. LEVEL OF EVIDENCE: 5 TECHNICAL EFFICACY: Stage 3.

Beyond 'statistical significance': A nontechnical primer of Bayesian statistics and Bayes factors for health researchers

RATIONALE

Hypothesis testing is integral to health research and is commonly completed through frequentist statistics focused on computing p values. p Values have been long criticized for offering limited information about the relationship of variables and strength of evidence concerning the plausibility, presence and certainty of associations among variables. Bayesian statistics is a potential alternative for inference-making. Despite emerging discussion on Bayesian statistics across various disciplines, the uptake of Bayesian statistics in health research is still limited.

AIM

To offer a primer on Bayesian statistics and Bayes factors for health researchers to gain preliminary knowledge of its use, application and interpretation in health research.

METHODS

Theoretical and empirical literature on Bayesian statistics and methods were used to develop this methodological primer.

CONCLUSIONS

Using Bayesian statistics in health research without a careful and complete understanding of its underlying philosophy and differences from frequentist testing, estimation and interpretation methods can result in similar ritualistic use as done for p values.

IMPLICATIONS

Health researchers should supplement frequentists statistics with Bayesian statistics when analysing research data. The overreliance on p values for clinical decisions making should be avoided. Bayes factors offer a more intuitive measure of assessing the strength of evidence for null and alternative hypothesis.

Campbell's Law Explains the Replication Crisis: Pre-Registration Badges Are History Repeating

Campbell's Law explains the replication crisis. In brief, useful tools such as hypotheses, p-values, and multi-study designs came to be viewed as indicators of strong science, and thus goals in and of themselves. Consequently, their use became distorted in unanticipated ways (e.g., hypothesizing after results were known [HARKing], p-Hacking, misuses of researcher degrees of freedom), and fragile findings proliferated. Pre-registration mandates are positioned as an antidote. However, I argue that such efforts, perhaps best exemplified by pre-registration badges (PRBs), are history repeating: Another useful tool has been converted into an indicator of strong science and a goal in and of itself. This, too, will distort its use and harm psychological science in unanticipated ways. For example, there is already evidence that papers seeking PRBs routinely violate the rules and spirit of pre-registration. I suggest that pre-registration mandates will (a) discourage optimal scientific practice, (b) exacerbate the file drawer problem, (c) encourage pre-registering after results are known (PRARKing), and (d) create false trust in fragile findings. I conclude that multiple design features can help support replicability (e.g., adequate sample size, valid measurement, robustness checks, pre-registration), none should be canonized, replication is the only arbiter of replicability, and the most important solution is sociocultural: to foster a field that reveres and reinforces robust science-just as we once revered and reinforced flashy but fragile science.

Cognitive Processing Speed and Loneliness in Stroke Survivors: Insights from a Large-Scale Cohort Study

OBJECTIVE

Loneliness, when prolonged, is associated with many deleterious effects and has been shown to be highly prevalent in those with a history of stroke, yet the cognitive mechanisms underpinning this phenomenon remain unclear. Therefore, the current study aims to investigate the extent to which cognitive factors, with specific focus on processing speed, are associated with loneliness in those with a history of stroke.

METHOD

Utilizing data from the British Cohort Study, a nationally representative dataset, we conducted secondary data analysis. A total of 7,752 participants completed relevant questions related to health, social interactions, demographics, loneliness, and cognitive assessments. Among them, 47 had experienced a stroke ("stroke," n = 47), 5,545 reported other health conditions ("ill," n = 5,545), and 2,857 were deemed healthy ("healthy," n = 2,857).

RESULTS

Consistent with previous research, our findings confirmed a positive correlation between stroke history and heightened loneliness. However, inferential analysis revealed that processing speed, alongside other cognitive factors, had a minimal impact on loneliness, with correlations too small to draw definitive conclusions.

CONCLUSION

This study suggests that cognitive processing speed alone is not a robust predictor of loneliness in stroke survivors. Consequently, when developing interventions to combat loneliness in this population, it is crucial to consider a broader spectrum of factors, such as social engagement, emotional wellbeing, and interpersonal relationships. This underscores the imperative need for comprehensive assessments to better comprehend the multifaceted nature of loneliness and inform more effective intervention strategies.

Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model

In recent years, concern has grown about the inappropriate application and interpretation of P values, especially the use of P<0.05 to denote "statistical significance" and the practice of P-hacking to produce results below this threshold and selectively reporting these in publications. Such behavior is said to be a major contributor to the large number of false and non-reproducible discoveries found in academic journals. In response, it has been proposed that the threshold for statistical significance be changed from 0.05 to 0.005. The aim of the current study was to use an evolutionary agent-based model comprised of researchers who test hypotheses and strive to increase their publication rates in order to explore the impact of a 0.005 P value threshold on P-hacking and published false positive rates. Three scenarios were examined, one in which researchers tested a single hypothesis, one in which they tested multiple hypotheses using a P<0.05 threshold, and one in which they tested multiple hypotheses using a P<0.005 threshold. Effects sizes were varied across models and output assessed in terms of researcher effort, number of hypotheses tested and number of publications, and the published false positive rate. The results supported the view that a more stringent P value threshold can serve to reduce the rate of published false positive results. Researchers still engaged in P-hacking with the new threshold, but the effort they expended increased substantially and their overall productivity was reduced, resulting in a decline in the published false positive rate. Compared to other proposed interventions to improve the academic publishing system, changing the P value threshold has the advantage of being relatively easy to implement and could be monitored and enforced with minimal effort by journal editors and peer reviewers.

A practical guide to EEG hyperscanning in joint action research: from motivation to implementation

Developments in cognitive neuroscience have led to the emergence of hyperscanning, the simultaneous measurement of brain activity from multiple people. Hyperscanning is useful for investigating social cognition, including joint action, because of its ability to capture neural processes that occur within and between people as they coordinate actions toward a shared goal. Here, we provide a practical guide for researchers considering using hyperscanning to study joint action and seeking to avoid frequently raised concerns from hyperscanning skeptics. We focus specifically on Electroencephalography (EEG) hyperscanning, which is widely available and optimally suited for capturing fine-grained temporal dynamics of action coordination. Our guidelines cover questions that are likely to arise when planning a hyperscanning project, ranging from whether hyperscanning is appropriate for answering one's research questions to considerations for study design, dependent variable selection, data analysis and visualization. By following clear guidelines that facilitate careful consideration of the theoretical implications of research design choices and other methodological decisions, joint action researchers can mitigate interpretability issues and maximize the benefits of hyperscanning paradigms.

First Comes Substance Use, Then Comes Social Media Posts? Examining the Temporal Ordering and Relative Strength of Relations Across Alcohol, Tobacco and Marijuana Use and Posting Behavior

Social media posts represent a major route by which youth share their substance use cognitions and experiences with others. Extant research has primarily examined relations between alcohol-related posts and posters' own alcohol use, yet little is known about the role of social media in the use of less socially accepted substances, namely tobacco and marijuana. Our study represents the first to examine the relative strength of this relation across alcohol, tobacco, and marijuana. The current research used a one-month time lag to tease apart the temporal ordering of substance-use-posting and participants' own substance use. A sample of 282 15-20-year-olds (Mage = 18.4, SD = 1.3, 52.9% female) in the United States completed two self-report surveys, one month apart. Results of a cross-lagged panel model revealed significant effects of alcohol and marijuana consumption on subsequent alcohol- and marijuana-related posting, respectively (i.e., selection effects). However, reverse relations (i.e., self-effects) were not significant. Further, we found no differences in the strength of selection effects across substances, suggesting they are similar for both more (alcohol) and less (marijuana and tobacco) socially acceptable substances. Results point to the importance of using young people's social media posts as a way to help identify individuals at risk for heightened substance use and social media as a mechanism for targeted prevention programming.

Are lexical representations graded or discrete?

Most research on mental lexical representations (lemmas) assumes they are discrete and correspond in number to a word's number of distinct meanings. Thus, homophones (bat), whose meanings are unrelated, have separate lemmas for each meaning (one for baseball bat, another for flying bat), whereas polysemes (paper), whose senses are related, have shared lemmas (the same lemma for printer paper and term paper). However, most aspects of cognition are thought to be graded, not discrete; could lemmas be graded too? We conducted a preregistered picture-word interference study with pictures of words whose meanings ranged from unrelated (homophones) to very related (regular polysemes). Whereas semantic competitors to picture names slow picture naming, semantic competitors to non-depicted meanings of homophones facilitate naming, suggesting distinct lemmas for homophones' meanings. We predicted that competitors to non-depicted senses of polysemes would slow naming, as polysemes' depicted and non-depicted senses presumably share a lemma. Crucially, we aimed to examine the transition from facilitation to inhibition: two groupings (where competitors to non-depicted senses led to facilitation for words with two lemmas but inhibition for words with one lemma) would imply that lemmas are indeed discrete. But a transition that varies continuously by sense relatedness would imply that lemmas are graded. Unexpectedly, competitors to non-depicted senses of both homophones and polysemes facilitated naming. Although these results do not indicate whether lemmas are graded or discrete, they do inform a long-standing question on the nature of polysemes, supporting a multiple-lemma (vs. core-lemma) account.

Athletic Injury Research: Frameworks, Models and the Need for Causal Knowledge

Within applied sports science and medicine research, many challenges hinder the establishment and detailed understanding of athletic injury causality as well as the development and implementation of appropriate athletic injury prevention strategies. Applied research efforts are faced with a lack of variable control, while the capacity to compensate for this lack of control through the application of randomised controlled trials is often confronted by a number of obstacles relating to ethical or practical constraints. Such difficulties have led to a large reliance upon observational research to guide applied practice in this area. However, the reliance upon observational research, in conjunction with the general absence of supporting causal inference tools and structures, has hindered both the acquisition of causal knowledge in relation to athletic injury and the development of appropriate injury prevention strategies. Indeed, much of athletic injury research functions on a (causal) model-blind observational approach primarily driven by the existence and availability of various technologies and data, with little regard for how these technologies and their associated metrics can conceptually relate to athletic injury causality and mechanisms. In this article, a potential solution to these issues is proposed and a new model for investigating athletic injury aetiology and mechanisms, and for developing and evaluating injury prevention strategies, is presented. This solution is centred on the construction and utilisation of various causal diagrams, such as frameworks, models and causal directed acyclic graphs (DAGs), to help guide athletic injury research and prevention efforts. This approach will alleviate many of the challenges facing athletic injury research by facilitating the investigation of specific causal links, mechanisms and assumptions with appropriate scientific methods, aiding the translation of lab-based research into the applied sporting world, and guiding causal inferences from applied research efforts by establishing appropriate supporting causal structures. Further, this approach will also help guide the development and adoption of both relevant metrics (and technologies) and injury prevention strategies, as well as encourage the construction of appropriate theoretical and conceptual foundations prior to the commencement of applied injury research studies. This will help minimise the risk of resource wastage, data fishing, p-hacking and hypothesising after the results are known (HARK-ing) in athletic injury research.

The choice-wide behavioral association study: data-driven identification of interpretable behavioral components

Behavior contains rich structure across many timescales, but there is a dearth of methods to identify relevant components, especially over the longer periods required for learning and decision-making. Inspired by the goals and techniques of genome-wide association studies, we present a data-driven method-the choice-wide behavioral association study: CBAS-that systematically identifies such behavioral features. CBAS uses a powerful, resampling-based, method of multiple comparisons correction to identify sequences of actions or choices that either differ significantly between groups or significantly correlate with a covariate of interest. We apply CBAS to different tasks and species (flies, rats, and humans) and find, in all instances, that it provides interpretable information about each behavioral task.

Notes from the Underground: Seeking the top personality correlates of self-referencing

OBJECTIVE

Self-focused language use has been frequently assumed to reflect narcissism; however, research indicates that the association between first-person singular pronouns (i.e., "I-talk") and grandiose narcissism is negligible.

METHOD

To extend this literature, we progressively identify vulnerable narcissism and rumination as positive correlates of I-talk in five studies (valid Ns = 211, 475, 1253, 289, 1113).

RESULTS

The first study revealed positive correlates of I-talk suggestive of vulnerable narcissism. The second study showed more directly that vulnerable narcissism was a positive correlate but that this association was attributable to shared variance with neuroticism. The third study, a preregistered effort, replicated and extended the results of the second study. The fourth and fifth studies focused on rumination in a preregistered manner.

CONCLUSIONS

All the studies point to a clear distinction: While grandiose narcissism is negligibly related to I-talk, vulnerable narcissism is positively related to I-talk; moreover, rumination is a robust predictor of I-talk. A research synthesis revealed the following constructs significantly capture I-talk: depression (r = 0.10), neuroticism (r = 0.15), rumination (r = 0.14), and vulnerable narcissism (r = 0.12). The association between I-talk and neuroticism was partially mediated by rumination, providing a testable candidate mechanism for neuroticism interventions.

No support for a causal role of primary motor cortex in construing meaning from language: An rTMS study

Embodied cognition theories predict a functional involvement of sensorimotor processes in language understanding. In a preregistered experiment, we tested this idea by investigating whether interfering with primary motor cortex (M1) activation can change how people construe meaning from action language. Participants were presented with sentences describing actions (e.g., "turning off the light") and asked to choose between two interpretations of their meaning, one more concrete (e.g., "flipping a switch") and another more abstract (e.g., "going to sleep"). Prior to this task, participants' M1 was disrupted using repetitive transcranial magnetic stimulation (rTMS). The results yielded strong evidence against the idea that M1-rTMS affects meaning construction (BF01 > 30). Additional analyses and control experiments suggest that the absence of effect cannot be accounted for by failure to inhibit M1, lack of construct validity of the task, or lack of power to detect a small effect. In sum, these results do not support a causal role for primary motor cortex in building meaning from action language.

Value of preclinical systematic reviews and meta-analyses in pediatric research

Similar to systematic reviews (SRs) in clinical fields, preclinical SRs address a specific research area, furnishing information on current knowledge, possible gaps, and potential methodological flaws of study design, conduct, and report. One of the main goals of preclinical SRs is to identify aspiring treatment strategies and evaluate if currently available data is solid enough to translate to clinical trials or highlight the gaps, thus justifying the need for new studies. It is imperative to rigorously follow the methodological standards that are widely available. These include registration of the protocol and adherence to guidelines for assessing the risk of bias, study quality, and certainty of evidence. A special consideration should be made for pediatric SRs, clinical and preclinical, due to the unique characteristics of this age group. These include rationale for intervention and comparison of primary and secondary outcomes. Outcomes measured should acknowledge age-related physiological changes and maturational processes of different organ systems. It is crucial to choose the age of the animals appropriately and its possible correspondence for specific pediatric age groups. The findings of well-conducted SRs of preclinical studies have the potential to provide a reliable evidence synthesis to guide the design of future preclinical and clinical studies. IMPACT: This narrative review highlights the importance of rigorous design, conduct and reporting of preclinical primary studies and systematic reviews. A special consideration should be made for pediatric systematic reviews of preclinical studies, due to the unique characteristics of this age group.

The effect of measurement error on the positive predictive value of PSW methods for SLD identification: How buffer zones dispel the illusion of inaccuracy

Patterns of Strengths and Weaknesses (PSW) methods are widely used for identifying specific learning disabilities (SLDs). Several researchers, however, have reported that the diagnostic accuracy of PSW methods is unacceptably low when strict thresholds were used to identify students with SLDs. We believe these findings give a misleading impression of the magnitude of the diagnostic errors that are likely to arise in PSW assessments. In a simulation study of 10 million cases using a simplified PSW method for demonstration, most of what have been called diagnostic errors were cases in which observed scores and true scores fell on opposite sides of a strict threshold but were still within a buffer zone the size of a typical measurement error. Because small score differences do not result in meaningfully different case conceptualizations, the use of buffer zones reveals that previous estimates of the diagnostic accuracy of PSW methods are misleadingly low. We also demonstrate that diagnostic decisions become increasingly reliable when observed scores are comfortably distant from diagnostic thresholds. For practitioners, we present a flowchart and practical guidelines to improve the accuracy and stability of SLD identification decisions.

Registered reports and replications: An ongoing Journal of School Psychology initiative

Recent psychological research suggests that many published studies cannot be replicated (e.g., Open Science Collaboration, 2015). The inability to replicate results suggests that there are influences and biases in the publication process that encourage publication of unusual-rather than representative-results, and that also discourage independent replication of published studies. A brief discussion of the ways in which publication bias and professional incentives may distort the research literature in school psychology is contrasted against the importance of replications and preregistration of research (i.e., registered reports) as self-correcting mechanisms for research in school psychology. The limitations of current practices, coupled with the importance of registered reports and replications as self-correcting mechanisms, provide the context for this ongoing initiative in the Journal of School Psychology. Processes for manuscript submission, review, and publication are presented to encourage researchers to preregister studies and submit replications for publication.

The majority of observational studies in leading peer-reviewed medicine journals are not registered and do not have a publicly accessible protocol: a scoping review

OBJECTIVES

Observational studies are not subject to the same requirements as randomized controlled trials, such as registration or publishing a protocol. The aim of this scoping review was to estimate the registration rate of observational studies in leading peer-reviewed medicine journals and to evaluate whether protocols were available in the public domain.

STUDY DESIGN AND SETTING

In March 2023, we searched OVID Medline for observational studies published in 2022 in the top five general medicine journals according to impact factor (The Lancet, The British Medical Journal (BMJ), The Journal of the American Medical Association, The New England Journal of Medicine, and Annals of Internal Medicine). We defined an observational study as a cohort study, a case-control study, a cross-sectional study, or a case series. Information on i) the proportion of observational studies that have been registered and ii) the proportion of observational studies that have a protocol available in the public domain was extracted from a random sample of studies.

RESULTS

Our search identified 699 studies; 290 studies were selected as full text, and a random sample of 200 studies was included. For half of the studies, the first author worked at a US institution. Most studies were cohort studies (n = 126, 63.0%) and used administrative healthcare records, electronic healthcare records, and registries. Of the 200 observational studies, 20 (10.0%) were registered. Among those, 14 were prospectively registered. Twenty-four studies (12.0%) had a protocol available in the public domain. Studies that were registered or had a protocol, were more frequently published in the BMJ (n = 12/28, 42.9%), had a first author working in the UK (n = 10/28, 35.7%) and used electronic health care records (n = 13/28, 46.4%) compared to studies with no registration and no protocol.

CONCLUSION

The rate of prospectively registered observational studies is worryingly low. Prospective registration of observational studies should be encouraged and standardized to ensure transparency in clinical research and reduce research waste.

A registered report survey of open research practices in psychology departments in the UK and Ireland

Open research practices seek to enhance the transparency and reproducibility of research. While there is evidence of increased uptake in these practices, such as study preregistration and open data, facilitated by new infrastructure and policies, little research has assessed general uptake of such practices across psychology university researchers. The current study estimates psychologists' level of engagement in open research practices across universities in the United Kingdom and Ireland, while also assessing possible explanatory factors that may impact their engagement. Data were collected from 602 psychology researchers in the United Kingdom and Ireland on the extent to which they have implemented various practices (e.g., use of preprints, preregistration, open data, open materials). Here we present the summarized descriptive results, as well as considering differences between various categories of researcher (e.g., career stage, subdiscipline, methodology), and examining the relationship between researcher's practices and their self-reported capability, opportunity, and motivation (COM-B) to engage in open research practices. Results show that while there is considerable variability in engagement of open research practices, differences across career stage and subdiscipline of psychology are small by comparison. We observed consistent differences according to respondent's research methodology and based on the presence of institutional support for open research. COM-B dimensions were collectively significant predictors of engagement in open research, with automatic motivation emerging as a consistently strong predictor. We discuss these findings, outline some of the challenges experienced in this study, and offer suggestions and recommendations for future research. Estimating the prevalence of responsible research practices is important to assess sustained behaviour change in research reform, tailor educational training initiatives, and to understand potential factors that might impact engagement.

Sex contextualism in laboratory research: Enhancing rigor and precision in the study of sex-related variables

Understanding sex-related variation in health and illness requires rigorous and precise approaches to revealing underlying mechanisms. A first step is to recognize that sex is not in and of itself a causal mechanism; rather, it is a classification system comprising a set of categories, usually assigned according to a range of varying traits. Moving beyond sex as a system of classification to working with concrete and measurable sex-related variables is necessary for precision. Whether and how these sex-related variables matter-and what patterns of difference they contribute to-will vary in context-specific ways. Second, when researchers incorporate these sex-related variables into research designs, rigorous analytical methods are needed to allow strongly supported conclusions. Third, the interpretation and reporting of sex-related variation require care to ensure that basic and preclinical research advance health equity for all.

Skewness and Staging: Does the Floor Effect Induce Bias in Multilevel AR(1) Models?

Multilevel autoregressive models are popular choices for the analysis of intensive longitudinal data in psychology. Empirical studies have found a positive correlation between autoregressive parameters of affective time series and the between-person measures of psychopathology, a phenomenon known as the staging effect. However, it has been argued that such findings may represent a statistical artifact: Although common models assume normal error distributions, empirical data (for instance, measurements of negative affect among healthy individuals) often exhibit the floor effect, that is response distributions with high skewness, low mean, and low variability. In this paper, we investigated whether-and to what extent-the floor effect leads to erroneous conclusions by means of a simulation study. We describe three dynamic models which have meaningful substantive interpretations and can produce floor-effect data. We simulate multilevel data from these models, varying skewness independent of individuals' autoregressive parameters, while also varying the number of time points and cases. Analyzing these data with the standard multilevel AR(1) model we found that positive bias only occurs when modeling with random residual variance, whereas modeling with fixed residual variance leads to negative bias. We discuss the implications of our study for data collection and modeling choices.

Raising awareness of uncertain choices in empirical data analysis: A teaching concept toward replicable research practices

Throughout their education and when reading the scientific literature, students may get the impression that there is a unique and correct analysis strategy for every data analysis task and that this analysis strategy will always yield a significant and noteworthy result. This expectation conflicts with a growing realization that there is a multiplicity of possible analysis strategies in empirical research, which will lead to overoptimism and nonreplicable research findings if it is combined with result-dependent selective reporting. Here, we argue that students are often ill-equipped for real-world data analysis tasks and unprepared for the dangers of selectively reporting the most promising results. We present a seminar course intended for advanced undergraduates and beginning graduate students of data analysis fields such as statistics, data science, or bioinformatics that aims to increase the awareness of uncertain choices in the analysis of empirical data and present ways to deal with these choices through theoretical modules and practical hands-on sessions.

Beyond single paradigms, pipelines, and outcomes: Embracing multiverse analyses in psychophysiology

Psychophysiological research is an inherently complex undertaking due to the nature of the data, and its analysis is characterized by many decision points that shape the final dataset and a study's findings. These decisions create a "multiverse" of possible outcomes, and each decision from study conceptualization to statistical analysis can lead to different results and interpretations. This review describes the concept of multiverse analyses, a methodological approach designed to understand the impact of different decisions on the robustness of a study's findings and interpretation. The emphasis is on transparently showcasing different reasonable approaches for constructing a final dataset and on highlighting the influence of various decision points, from experimental design to data processing and outcome selection. For example, the choice of an experimental task can significantly impact event-related brain potential (ERP) scores or skin conductance responses (SCRs), and different tasks might elicit unique variances in each measure. This review underscores the importance of transparently embracing the flexibility inherent in psychophysiological research and the potential consequences of not understanding the fragility or robustness of experimental findings. By navigating the intricate terrain of the psychophysiological multiverse, this review serves as an introduction, helping researchers to make informed decisions, improve the collective understanding of psychophysiological findings, and push the boundaries of the field.

Publication bias in psychology: A closer look at the correlation between sample size and effect size

Previously observed negative correlations between sample size and effect size (n-ES correlation) in psychological research have been interpreted as evidence for publication bias and related undesirable biases. Here, we present two studies aimed at better understanding to what extent negative n-ES correlations reflect such biases or might be explained by unproblematic adjustments of sample size to expected effect sizes. In Study 1, we analysed n-ES correlations in 150 meta-analyses from cognitive, organizational, and social psychology and in 57 multiple replications, which are free from relevant biases. In Study 2, we used a random sample of 160 psychology papers to compare the n-ES correlation for effects that are central to these papers and effects selected at random from these papers. n-ES correlations proved inconspicuous in meta-analyses. In line with previous research, they do not suggest that publication bias and related biases have a strong impact on meta-analyses in psychology. A much higher n-ES correlation emerged for publications' focal effects. To what extent this should be attributed to publication bias and related biases remains unclear.

Evidence-b(i)ased practice: Selective and inadequate reporting in early childhood autism intervention research

LAY ABSTRACT

When researchers fail to report their findings or only report some of their findings, it can make it difficult for clinicians to provide effective intervention recommendations. However, no one has examined whether this is a problem in studies of early childhood autism interventions. We studied how researchers that study early childhood autism interventions report their findings. We found that most researchers did not register their studies when they were supposed to (before the start of the study), and that many researchers did not provide all of the needed information in the registration. We also found that researchers frequently did not publish their findings when their studies were complete. When we looked at published reports, we found that many of the studies did not report enough information, and that many studies were reported differently from their registrations, suggesting that researchers were selectively reporting positive outcomes and ignoring or misrepresenting less positive outcomes. Because we found so much evidence that researchers are failing to report their findings quickly and correctly, we suggested some practical changes to make it better.

Detecting the corruption of online questionnaires by artificial intelligence

Online questionnaires that use crowdsourcing platforms to recruit participants have become commonplace, due to their ease of use and low costs. Artificial intelligence (AI)-based large language models (LLMs) have made it easy for bad actors to automatically fill in online forms, including generating meaningful text for open-ended tasks. These technological advances threaten the data quality for studies that use online questionnaires. This study tested whether text generated by an AI for the purpose of an online study can be detected by both humans and automatic AI detection systems. While humans were able to correctly identify the authorship of such text above chance level (76% accuracy), their performance was still below what would be required to ensure satisfactory data quality. Researchers currently have to rely on a lack of interest among bad actors to successfully use open-ended responses as a useful tool for ensuring data quality. Automatic AI detection systems are currently completely unusable. If AI submissions of responses become too prevalent, then the costs associated with detecting fraudulent submissions will outweigh the benefits of online questionnaires. Individual attention checks will no longer be a sufficient tool to ensure good data quality. This problem can only be systematically addressed by crowdsourcing platforms. They cannot rely on automatic AI detection systems and it is unclear how they can ensure data quality for their paying clients.

Does darkness increase the risk of certain types of crime? A registered report protocol

Evidence about the relationship between lighting and crime is mixed. Although a review of evidence found that improved road / street lighting was associated with reductions in crime, these reductions occurred in daylight as well as after dark, suggesting any effect was not due only to changes in visual conditions. One limitation of previous studies is that crime data are reported in aggregate and thus previous analyses were required to make simplifications concerning types of crimes or locations. We will overcome that by working with a UK police force to access records of individual crimes. We will use these data to determine whether the risk of crime at a specific time of day is greater after dark than during daylight. If no difference is found, this would suggest improvements to visual conditions after dark through lighting would have no effect. If however the risk of crime occurring after dark was greater than during daylight, quantifying this effect would provide a measure to assess the potential effectiveness of lighting in reducing crime risk after dark. We will use a case and control approach to analyse ten years of crime data. We will compare counts of crimes in 'case' hours, that are in daylight and darkness at different times of the year, and 'control' hours, that are in daylight throughout the year. From these counts we will calculate odds ratios as a measure of the effect of darkness on risk of crime, using these to answer three questions: 1) Is the risk of overall crime occurring greater after dark than during daylight? 2) Does the risk of crime occurring after dark vary depending on the category of crime? 3) Does the risk of crime occurring after dark vary depending on the geographical area?

Publication Bias and Selective Outcome Reporting in Randomized Controlled Trials Related to Rehabilitation: A Literature Review

OBJECTIVE

To investigate the rate of registered protocols published as research papers as a measure of publication bias, and the concordance rates of the primary outcomes between research protocol and published papers as a measure of selective outcome reporting bias in randomized controlled trials (RCTs) related to rehabilitation.

DATA SOURCES

Protocols related to RCTs were extracted from electronic databases, the University Hospital Medical Information Network (UMIN), International Standard Research Clinical Trial Number (ISRCTN), ClinicalTrials.gov, and MEDLINE. Published papers were retrieved from MEDLINE.

STUDY SELECTION

The inclusion criteria were as follows: (1) initial registration (UMIN, ISRCTN, ClinicalTrials.gov) within the designated period; (2) published as a paper from a research protocol in MEDLINE (PubMed); and (3) written in English or Japanese. The search period was from January 1, 2013, to December 31, 2020.

DATA EXTRACTION

The outcome of this study was set as the rate of published papers that were consistent with the extracted research protocol and the concordance rate between the primary outcomes in published papers and in protocols. The concordance rate of the primary outcomes was evaluated by checking whether the description in the research protocol matched that in the paper's abstract and main text.

DATA SYNTHESIS

Out of the 5597 research protocols registered, only 727 were published (13.0%). The concordance rates of the primary outcomes were 48.7% and 72.6% in the abstract and main text, respectively.

CONCLUSIONS

This study revealed major discrepancies between the number of research protocols and published papers, and difference of description regarding the primary outcomes in published papers which were already defined in the research protocols.

The Gene: An appraisal

The gene can be described as the foundational concept of modern biology. As such, it has spilled over into daily discourse, yet it is acknowledged among biologists to be ill-defined. Here, following a short history of the gene, I analyse critically its role in inheritance, evolution, development, and morphogenesis. Wilhelm Johannsen's genotype-conception, formulated in 1910, has been adopted as the foundation stone of genetics, giving the gene a higher degree of prominence than is justified by the evidence. An analysis of the results of the Long-Term Evolution Experiment (LTEE) with E. coli bacteria, grown over 60,000 generations, does not support spontaneous gene mutation as the source of variance for natural selection. From this it follows that the gene is not Mendel's unit of inheritance: that must be Johannsen's transmission-conception at the gamete phenotype level, a form of inheritance that Johannsen did not consider. Alternatively, I contend that biology viewed on the bases of thermodynamics, complex system dynamics, and self-organisation, provides a new framework for the foundations of biology. In this framework, the gene plays a passive role as a vital information store: it is the phenotype that plays the active role in inheritance, evolution, development, and morphogenesis.

Sex-dependent effects of Canagliflozin on kidney protection in mice with combined hypertension-type 1 diabetes

Canagliflozin (CANA) is a sodium-glucose cotransporter 2 (SGLT2) inhibitor with blood glucose lowering effects. CANA also promotes kidney protection in patients with cardiovascular diseases and type 2 diabetes (T2D), as well as in normoglycemic patients with hypertension or heart failure. Clinical studies, although conduct in both sexes, do not report sex-dependent differences in T2DM treated with CANA. However, the impact of CANA in type 1 diabetes, as well in sex-dependent outcomes in such cohort needs further understanding. To analyze the effects of CANA in mice with combined hypertension and type 1 diabetes, diabetes was induced by STZ injection (5 days, 50mg/kg/day) in both male and female 8 weeks old genetic hypertensive mice (Lin), whereas the control (Lin) received 0.1M sodium citrate injections. 8 weeks after STZ. Mice were fed either regular or CANA-infused diet for 4 weeks. 8 weeks after STZ, hyperglycemia was present in both male and female mice. CANA reversed BG increase mice fed regular diet. Male LinSTZ mice had elevated water intake, urine output, urinary albumin to creatinine ratio (ACR), kidney lesion score, and creatinine clearance compared to the Lin control group. Kidney injury was improved in male LinSTZ + CANA group in male mice. Water intake and urine output were not statistically significantly different in female LinSTZ compared to female LinSTZ+ CANA. Moreover, CANA did not improve kidney injury in female mice, showing no effect in creatinine clearance, lesion score and fibrosis when compared to LinSTZ fed regular diet. Here we show that Canagliflozin might exert different kidney protection effects in male compared to female mice with hypertension and type 1 diabetes. Sex-dimorphisms were previously found in the pathophysiology of diabetes induced by STZ. Therefore, we highlight the importance of in-depth investigation on sex-dependent effects of CANA, taking in consideration the unique characteristics of disease progression for each sex.

Predictive coding and stochastic resonance as fundamental principles of auditory phantom perception

Mechanistic insight is achieved only when experiments are employed to test formal or computational models. Furthermore, in analogy to lesion studies, phantom perception may serve as a vehicle to understand the fundamental processing principles underlying healthy auditory perception. With a special focus on tinnitus-as the prime example of auditory phantom perception-we review recent work at the intersection of artificial intelligence, psychology and neuroscience. In particular, we discuss why everyone with tinnitus suffers from (at least hidden) hearing loss, but not everyone with hearing loss suffers from tinnitus. We argue that intrinsic neural noise is generated and amplified along the auditory pathway as a compensatory mechanism to restore normal hearing based on adaptive stochastic resonance. The neural noise increase can then be misinterpreted as auditory input and perceived as tinnitus. This mechanism can be formalized in the Bayesian brain framework, where the percept (posterior) assimilates a prior prediction (brain's expectations) and likelihood (bottom-up neural signal). A higher mean and lower variance (i.e. enhanced precision) of the likelihood shifts the posterior, evincing a misinterpretation of sensory evidence, which may be further confounded by plastic changes in the brain that underwrite prior predictions. Hence, two fundamental processing principles provide the most explanatory power for the emergence of auditory phantom perceptions: predictive coding as a top-down and adaptive stochastic resonance as a complementary bottom-up mechanism. We conclude that both principles also play a crucial role in healthy auditory perception. Finally, in the context of neuroscience-inspired artificial intelligence, both processing principles may serve to improve contemporary machine learning techniques.

Poor statistical reporting, inadequate data presentation and spin persist despite Journal awareness and updated Information for Authors

Sound reporting of research results is fundamental to good science. Unfortunately, poor reporting is common and does not improve with editorial educational strategies. We investigated whether publicly highlighting poor reporting at a journal can lead to improved reporting practices. We also investigated whether reporting practices that are required or strongly encouraged in journal Information for Authors are enforced by journal editors and staff. A 2016 audit highlighted poor reporting practices in the Journal of Neurophysiology. In August 2016 and 2018, the American Physiological Society updated the Information for Authors, which included the introduction of several required or strongly encouraged reporting practices. We audited Journal of Neurophysiology papers published in 2019 and 2020 (downloaded through the library of the University of New South Wales) on reporting items selected from the 2016 audit, the newly introduced reporting practices, and items from previous audits. Summary statistics (means, counts) were used to summarize audit results. In total, 580 papers were audited. Compared to results from the 2016 audit, several reporting practices remained unchanged or worsened. For example, 60% of papers erroneously reported standard errors of the mean, 23% of papers included undefined measures of variability, 40% of papers failed to define a statistical threshold for their tests, and when present, 64% of papers with p-values between 0.05 and 0.1 misinterpreted them as statistical trends. As for the newly introduced reporting practices, required practices were consistently adhered to by 34 to 37% of papers, while strongly encouraged practices were consistently adhered to by 9 to 26% of papers. Adherence to the other audited reporting practices was comparable to our previous audits. Publicly highlighting poor reporting practices did little to improve research reporting. Similarly, requiring or strongly encouraging reporting practices was only partly effective. Although the present audit focused on a single journal, this is likely not an isolated case. Stronger, more strategic measures are required to improve poor research reporting.

Preregistration in practice: A comparison of preregistered and non-preregistered studies in psychology

Preregistration has gained traction as one of the most promising solutions to improve the replicability of scientific effects. In this project, we compared 193 psychology studies that earned a Preregistration Challenge prize or preregistration badge to 193 related studies that were not preregistered. In contrast to our theoretical expectations and prior research, we did not find that preregistered studies had a lower proportion of positive results (Hypothesis 1), smaller effect sizes (Hypothesis 2), or fewer statistical errors (Hypothesis 3) than non-preregistered studies. Supporting our Hypotheses 4 and 5, we found that preregistered studies more often contained power analyses and typically had larger sample sizes than non-preregistered studies. Finally, concerns about the publishability and impact of preregistered studies seem unwarranted, as preregistered studies did not take longer to publish and scored better on several impact measures. Overall, our data indicate that preregistration has beneficial effects in the realm of statistical power and impact, but we did not find robust evidence that preregistration prevents p-hacking and HARKing (Hypothesizing After the Results are Known).

Best practices to promote rigor and reproducibility in the era of sex-inclusive research

To enhance inclusivity and rigor, many funding agencies and journals now mandate the inclusion of females as well as males in biomedical studies. These mandates have enhanced generalizability and created unprecedented opportunities to discover sex differences. However, education in sound methods to consider sex as a subgroup category has lagged behind, resulting in a problematic literature in which study designs, analyses, and interpretations of results are often flawed. Here, we outline best practices for complying with sex-inclusive mandates, both for studies in which sex differences are a primary focus and for those in which they are not. Our recommendations are organized within the "4 Cs of Studying Sex to Strengthen Science: Consideration, Collection, Characterization and Communication," a framework developed by the Office of Research on Women's Health at the National Institutes of Health in the United States. Following these guidelines should help researchers include females and males in their studies while at the same time upholding high standards of rigor.

Do sleep disruptions promote social fragmentation?

Sleep changes predate shifts in mood/affect, thought processing, mental and physical health, civic engagement, and contextual circumstances, among other things. Theory predicts that these changes may lead to shifts in political and social beliefs. Do sleep disruptions shape how individuals see the world, the people around them, and themselves in relation to others? In this article, we use daily survey data from the 77 waves (N ≈ 460,000) of the University of California, Los Angeles's 2019-2021 Nationscape Survey-a nationally representative political survey-to examine the effect of an exogenous short-term sleep disruption on measures of political views, polarization, and discriminatory beliefs. Using this data set, we leverage the modest sleep disruption that occurs at the start (and end) of Daylight Saving Time (DST) and employ a regression discontinuity in time design around the precise DST cutoff (which we supplement with event study models). Despite strong theoretical expectations and correlational connection between measures of sleep and many outcomes related to social fragmentation, we find that the DST change has little to no causal effect on citizens' levels of polarization or their discriminatory attitudes. These effects are precise enough to rule out small effects, robust to a host of specification checks, and consistent across potential subgroups of interest. Our work adds to a small but growing body of research on the social and political effects of sleep disruptions.

Acceptance and Commitment Therapy Processes and Mediation: Challenges and How to Address Them

Acceptance and commitment therapy (ACT) emphasizes a focus on theory-driven processes and mediating variables, a laudable approach. The implementation of this approach would be advanced by addressing five challenges, including (a) distinguishing ACT processes in measurement contexts, (b) developing and rigorously validating measures of ACT processes, (c) the wide use of psychometrically weaker ACT process measures and the more limited use of stronger measures in earlier work, (d) the inconsistency of past evidence that ACT processes are sensitive or specific to ACT or mediate ACT outcomes specifically, and (e) improving statistical power and transparency. Drawing on the existing literature, we characterize and provide evidence for each of these challenges. We then offer detailed recommendations for how to address each challenge in ongoing and future work. Given ACT's core focus on theorized processes, improving the measurement and evaluation of these processes would significantly advance the field's understanding of ACT.

Null regions: a unified conceptual framework for statistical inference

Ruling out the possibility that there is absolutely no effect or association between variables may be a good first step, but it is rarely the ultimate goal of science. Yet that is the only inference provided by traditional null hypothesis significance testing (NHST), which has been a mainstay of many scientific fields. Reliance on NHST also makes it difficult to define what it means to replicate a finding, and leads to an uncomfortable quandary in which increasing precision in data reduces researchers' ability to perform theory falsification. To solve these problems, in recent years several alternatives to traditional NHST have been proposed. However, each new test is described using its own terminology and practiced in different fields. We describe a simple, unified framework for conceptualizing all these tests so that it is not necessary to learn them separately. Moreover, the framework allows researchers to conduct any of these tests by asking just one question: is the confidence interval entirely outside the null region(s)? This framework may also help researchers choose the test(s) that best answers their research question when simply ruling out 'no effect at all' is not enough.

The positive-negative-competence (PNC) model of psychological responses to representations of robots

Robots are becoming an increasingly prominent part of society. Despite their growing importance, there exists no overarching model that synthesizes people's psychological reactions to robots and identifies what factors shape them. To address this, we created a taxonomy of affective, cognitive and behavioural processes in response to a comprehensive stimulus sample depicting robots from 28 domains of human activity (for example, education, hospitality and industry) and examined its individual difference predictors. Across seven studies that tested 9,274 UK and US participants recruited via online panels, we used a data-driven approach combining qualitative and quantitative techniques to develop the positive-negative-competence model, which categorizes all psychological processes in response to the stimulus sample into three dimensions: positive, negative and competence-related. We also established the main individual difference predictors of these dimensions and examined the mechanisms for each predictor. Overall, this research provides an in-depth understanding of psychological functioning regarding representations of robots.

No evidence for contextual cueing beyond explicit recognition

Many studies claim that visual regularities can be learned unconsciously and without explicit awareness. For example in the contextual cueing paradigm, studies often make claims using a standard reasoning based on two results: (1) a reliable response time (RT) difference between repeated vs. new stimulus displays and (2) a close-to-chance sensitivity when participants are asked to explicitly recognize repeated stimulus displays. From this pattern of results, studies routinely conclude that the sensitivity of RT responses is higher than that of explicit responses-an empirical situation we call Indirect Task Advantage (ITA). Many studies further infer from an ITA that RT effects were driven by a form of recognition that exceeds explicit memory: implicit recognition. However, this reasoning is flawed because the sensitivity underlying RT effects is never computed. To properly establish a difference, a sensitivity comparison is required. We apply this sensitivity comparison in a reanalysis of 20 contextual cueing studies showing that not a single study provides consistent evidence for ITAs. Responding to recent correlation-based arguments, we also demonstrate the absence of evidence for ITAs at the level of individual participants. This lack of ITAs has serious consequences for the field: If RT effects can be fully explained by weak but above-chance explicit recognition sensitivity, what is the empirical content of the label "implicit"? Thus, theoretical discussions in this paradigm-and likely in other paradigms using this standard reasoning-require serious reassessment because the current data from contextual cueing studies is insufficient to consider recognition as implicit.

manymome: An R package for computing the indirect effects, conditional effects, and conditional indirect effects, standardized or unstandardized, and their bootstrap confidence intervals, in many (though not all) models

Mediation, moderation, and moderated mediation are common in behavioral research models. Several tools are available for estimating indirect effects, conditional effects, and conditional indirect effects and forming their confidence intervals. However, there are no simple-to-use tools that can appropriately form the bootstrapping confidence interval for standardized conditional indirect effects. Moreover, some tools are restricted to a limited type of models. We developed an R package, manymome, which can be used to estimate and form confidence intervals for indirect effects, conditional effects, and conditional indirect effects, standardized or not, using a two-step approach: model parameters are estimated either by structural equation modeling using lavaan or by a set of linear regression models using lm, and then the coefficients are used to compute the requested effects and form confidence intervals. It can be used when there are missing data if the model is fitted by structural equation modeling. There are only a few limitations on some aspects of a model, and no inherent limitations on the number of predictors, the number of independent variables, or the number of moderators and mediators. The goal is to have a tool that allows researchers to focus on model fitting first and worry about estimating the effects later. The use of the model is illustrated using a few numerical examples, and the limitations of the package are discussed.

Guidance on protocol development for EFSA generic scientific assessments

EFSA Strategy 2027 outlines the need for fit-for-purpose protocols for EFSA generic scientific assessments to aid in delivering trustworthy scientific advice. This EFSA Scientific Committee guidance document helps address this need by providing a harmonised and flexible framework for developing protocols for EFSA generic assessments. The guidance replaces the 'Draft framework for protocol development for EFSA's scientific assessments' published in 2020. The two main steps in protocol development are described. The first is problem formulation, which illustrates the objectives of the assessment. Here a new approach to translating the mandated Terms of Reference into scientifically answerable assessment questions and sub-questions is proposed: the 'APRIO' paradigm (Agent, Pathway, Receptor, Intervention and Output). Owing to its cross-cutting nature, this paradigm is considered adaptable and broadly applicable within and across the various EFSA domains and, if applied using the definitions given in this guidance, is expected to help harmonise the problem formulation process and outputs and foster consistency in protocol development. APRIO may also overcome the difficulty of implementing some existing frameworks across the multiple EFSA disciplines, e.g. the PICO/PECO approach (Population, Intervention/Exposure, Comparator, Outcome). Therefore, although not mandatory, APRIO is recommended. The second step in protocol development is the specification of the evidence needs and the methods that will be applied for answering the assessment questions and sub-questions, including uncertainty analysis. Five possible approaches to answering individual (sub-)questions are outlined: using evidence from scientific literature and study reports; using data from databases other than bibliographic; using expert judgement informally collected or elicited via semi-formal or formal expert knowledge elicitation processes; using mathematical/statistical models; and - not covered in this guidance - generating empirical evidence ex novo. The guidance is complemented by a standalone 'template' for EFSA protocols that guides the users step by step through the process of planning an EFSA scientific assessment.

Open Science Standards at Journals that Inform Evidence-Based Policy

Evidence-based policy uses intervention research to inform consequential decisions about resource allocation. Research findings are often published in peer-reviewed journals. Because detrimental research practices associated with closed science are common, journal articles report more false-positives and exaggerated effect sizes than would be desirable. Journal implementation of standards that promote open science-such as the transparency and openness promotion (TOP) guidelines-could reduce detrimental research practices and improve the trustworthiness of research evidence on intervention effectiveness. We evaluated TOP implementation at 339 peer-reviewed journals that have been used to identify evidence-based interventions for policymaking and programmatic decisions. Each of ten open science standards in TOP was not implemented in most journals' policies (instructions to authors), procedures (manuscript submission systems), or practices (published articles). Journals implementing at least one standard typically encouraged, but did not require, an open science practice. We discuss why and how journals could improve implementation of open science standards to safeguard evidence-based policy.

Presentation and publication skills: Publication governance and pitfalls to avoid

There are several pitfalls in the publication process that researchers can fall victim to, and these can occur knowingly or unknowingly. Although some of these errors may have occurred in good faith, disregard of publication governance is a dangerous practice and could bring authors and their co-authors into disrepute. We highlight some of these potential pitfalls, acquaint the reader with some rules that need to be adhered to in research and publishing, and help the reader learn how to avoid tripping-up on the road to publication.

The trustworthiness of the cumulative knowledge in industrial/organizational psychology: The current state of affairs and a path forward

The goal of industrial/organizational (IO) psychology, is to build and organize trustworthy knowledge about people-related phenomena in the workplace. Unfortunately, as with other scientific disciplines, our discipline may be experiencing a "crisis of confidence" stemming from the lack of reproducibility and replicability of many of our field's research findings, which would suggest that much of our research may be untrustworthy. If a scientific discipline's research is deemed untrustworthy, it can have dire consequences, including the withdraw of funding for future research. In this focal article, we review the current state of reproducibility and replicability in IO psychology and related fields. As part of this review, we discuss factors that make it less likely that research findings will be trustworthy, including the prevalence of scientific misconduct, questionable research practices (QRPs), and errors. We then identify some root causes of these issues and provide several potential remedies. In particular, we highlight the need for improved research methods and statistics training as well as a re-alignment of the incentive structure in academia. To accomplish this, we advocate for changes in the reward structure, improvements to the peer review process, and the implementation of open science practices. Overall, addressing the current "crisis of confidence" in IO psychology requires individual researchers, academic institutions, and publishers to embrace system-wide change.

Registered Reports in Child Development: Introduction to the Special Section

Registered Reports (RRs) are an emerging format for publishing empirical journal articles in which the decision to publish an article is based on sound conceptualization, methods, and planned analyses rather than the specific nature of the results. This article introduces the Special Section on Registered Reports in Child Development by describing what RRs are and why they are necessary, outlining the thought process that guided the Special Section, describing key thematic insights across the eight articles included in the collection, and providing recommendations for developmental researchers interested in publishing via the RR format. This article also serves as a formal announcement that RRs will be a standard publishing option at Child Development, effective immediately.

On the scope of scientific hypotheses

Hypotheses are frequently the starting point when undertaking the empirical portion of the scientific process. They state something that the scientific process will attempt to evaluate, corroborate, verify or falsify. Their purpose is to guide the types of data we collect, analyses we conduct, and inferences we would like to make. Over the last decade, metascience has advocated for hypotheses being in preregistrations or registered reports, but how to formulate these hypotheses has received less attention. Here, we argue that hypotheses can vary in specificity along at least three independent dimensions: the relationship, the variables, and the pipeline. Together, these dimensions form the scope of the hypothesis. We demonstrate how narrowing the scope of a hypothesis in any of these three ways reduces the hypothesis space and that this reduction is a type of novelty. Finally, we discuss how this formulation of hypotheses can guide researchers to formulate the appropriate scope for their hypotheses and should aim for neither too broad nor too narrow a scope. This framework can guide hypothesis-makers when formulating their hypotheses by helping clarify what is being tested, chaining results to previous known findings, and demarcating what is explicitly tested in the hypothesis.

Reproducibility in Neuroimaging Analysis: Challenges and Solutions

Recent years have marked a renaissance in efforts to increase research reproducibility in psychology, neuroscience, and related fields. Reproducibility is the cornerstone of a solid foundation of fundamental research-one that will support new theories built on valid findings and technological innovation that works. The increased focus on reproducibility has made the barriers to it increasingly apparent, along with the development of new tools and practices to overcome these barriers. Here, we review challenges, solutions, and emerging best practices with a particular emphasis on neuroimaging studies. We distinguish 3 main types of reproducibility, discussing each in turn. Analytical reproducibility is the ability to reproduce findings using the same data and methods. Replicability is the ability to find an effect in new datasets, using the same or similar methods. Finally, robustness to analytical variability refers to the ability to identify a finding consistently across variation in methods. The incorporation of these tools and practices will result in more reproducible, replicable, and robust psychological and brain research and a stronger scientific foundation across fields of inquiry.

Preregistration: Practical Considerations for Speech, Language, and Hearing Research

PURPOSE

In the last decade, psychology and other sciences have implemented numerous reforms to improve the robustness of our research, many of which are based on increasing transparency throughout the research process. Among these reforms is the practice of preregistration, in which researchers create a time-stamped and uneditable document before data collection that describes the methods of the study, how the data will be analyzed, the sample size, and many other decisions. The current article highlights the benefits of preregistration with a focus on the specific issues that speech, language, and hearing researchers are likely to encounter, and additionally provides a tutorial for writing preregistrations.

CONCLUSIONS

Although rates of preregistration have increased dramatically in recent years, the practice is still relatively uncommon in research on speech, language, and hearing. Low rates of adoption may be driven by a lack of understanding of the benefits of preregistration (either generally or for our discipline in particular) or uncertainty about how to proceed if it becomes necessary to deviate from the preregistered plan. Alternatively, researchers may see the benefits of preregistration but not know where to start, and gathering this information from a wide variety of sources is arduous and time consuming. This tutorial addresses each of these potential roadblocks to preregistration and equips readers with tools to facilitate writing preregistrations for research on speech, language, and hearing.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.21644843.