Gender-equal funding rates conceal unequal evaluations

Negotiating science funding: The interplay of merit, bias, and administrative discretion in grant allocation in Kazakhstan

This paper analyzes 4,488 applications from a grant funding competition held in 2017 in Kazakhstan. The competition had a two-stage design: first, anonymous subject matter experts evaluated the applications' scientific potential; then, open panels of local science managers made the final decisions. We analyze a range of bibliometric, institutional, and demographic variables associated with the applications and show that review scores account for only a small variation in success rates. The most important factor is the organizational closeness to decision-making. Gender also plays a role: we find that, net of academic merit, men and women investigators receive similar review scores, yet the panelists grant awards to men more often than to women. We further demonstrate that the gender gap emerges due to decisions made in a specific domain-Natural Resource Management.

Assessing the potential of a Bayesian ranking as an alternative to consensus meetings for decision making in research funding: A case study of Marie Skłodowska-Curie actions

Funding agencies rely on panel or consensus meetings to summarise individual evaluations of grant proposals into a final ranking. However, previous research has shown inconsistency in decisions and inefficiency of consensus meetings. Using data from the Marie Skłodowska-Curie Actions, we aimed at investigating the differences between an algorithmic approach to summarise the information from grant proposal individual evaluations to decisions after consensus meetings, and we present an exploratory comparative analysis. The algorithmic approach employed was a Bayesian hierarchical model resulting in a Bayesian ranking of the proposals using the individual evaluation reports cast prior to the consensus meeting. Parameters from the Bayesian hierarchical model and the subsequent ranking were compared to the scores, ranking and decisions established in the consensus meeting reports. The results from the evaluation of 1,006 proposals submitted to three panels (Life Science, Mathematics, Social Sciences and Humanities) in two call years (2015 and 2019) were investigated in detail. Overall, we found large discrepancies between the consensus reports and the scores a Bayesian hierarchical model would have predicted. The discrepancies were less pronounced when the scores were aggregated into funding rankings or decisions. The best agreement between the final funding ranking can be observed in the case of funding schemes with very low success rates. While we set out to understand if algorithmic approaches, with the aim of summarising individual evaluation scores, could replace consensus meetings, we concluded that currently individual scores assigned prior to the consensus meetings are not useful to predict the final funding outcomes of the proposals. Following our results, we would suggest to use individual evaluations for a triage and subsequently not discuss the weakest proposals in panel or consensus meetings. This would allow a more nuanced evaluation of a smaller set of proposals and help minimise the uncertainty and biases when allocating funding.

Gender differences in Dutch research funding over time: A statistical investigation of the innovation scheme 2012-2021

BACKGROUND

In 2015, the Dutch research council, NWO, took measures to combat gender bias disadvantaging female applicants in a popular three-tiered funding scheme called the Talent Programme. The innovation scheme consists of three grants for different career stages, called Veni, Vidi and Vici.

OBJECTIVES

This paper studies the question whether or not NWO has been successful in removing gender differences in their funding procedure.

METHODS

Using all available data from 2012 onwards of grant applications in the Talent Programme (16,249 applications of which 2,449 received funding), we study whether these measures had an effect using binomial generalized linear models.

RESULTS

We find strong statistical evidence of a shift in gender effects in favour of female applicants in the first tier, the Veni (p < .001). Significant gender differences are not found in the two other tiers, the Vidi and Vici schemes.

CONCLUSIONS

In recent years, female applicants are more likely to be awarded with a Veni grant than male applicants and this gender gap has increased over time. This suggests that gender differences still exist in the assessment of Talent Programme submissions, albeit in a different direction than a decade ago.

Gender inequality in cum laude distinctions for PhD students

Resource allocation in academia is highly skewed, and peer evaluation is the main method used to distribute scarce resources. A large literature documents gender inequality in evaluation, and the explanation for this inequality is homophily: male evaluators give more favorable ratings to male candidates. We investigate this by focusing on cum laude distinctions for PhD students in the Netherlands, a distinction that is only awarded to 5 percent of all dissertations and has as its sole goal to distinguish the top from the rest. Using data from over 5000 PhD recipients of a large Dutch university for the period 2011-2021, we find that female PhD students were almost two times less likely to get a cum laude distinction than their male counterparts, even when they had the same doctoral advisor. This gender gap is largest when dissertations are evaluated by all-male committees and decreases as evaluation committees include more female members.

Systemic racial disparities in funding rates at the National Science Foundation

Concerns about systemic racism at academic and research institutions have increased over the past decade. Here, we investigate data from the National Science Foundation (NSF), a major funder of research in the United States, and find evidence for pervasive racial disparities. In particular, white principal investigators (PIs) are consistently funded at higher rates than most non-white PIs. Funding rates for white PIs have also been increasing relative to annual overall rates with time. Moreover, disparities occur across all disciplinary directorates within the NSF and are greater for research proposals. The distributions of average external review scores also exhibit systematic offsets based on PI race. Similar patterns have been described in other research funding bodies, suggesting that racial disparities are widespread. The prevalence and persistence of these racial disparities in funding have cascading impacts that perpetuate a cumulative advantage to white PIs across all of science, technology, engineering, and mathematics.

Is novel research worth doing? Evidence from peer review at 49 journals

There are long-standing concerns that peer review, which is foundational to scientific institutions like journals and funding agencies, favors conservative ideas over novel ones. We investigate the association between novelty and the acceptance of manuscripts submitted to a large sample of scientific journals. The data cover 20,538 manuscripts submitted between 2013 and 2018 to the journals Cell and Cell Reports and 6,785 manuscripts submitted in 2018 to 47 journals published by the Institute of Physics Publishing. Following previous work that found that a balance of novel and conventional ideas predicts citation impact, we measure the novelty and conventionality of manuscripts by the atypicality of combinations of journals in their reference lists, taking the 90th percentile most atypical combination as "novelty" and the 50th percentile as "conventionality." We find that higher novelty is consistently associated with higher acceptance; submissions in the top novelty quintile are 6.5 percentage points more likely than bottom quintile ones to get accepted. Higher conventionality is also associated with acceptance (+16.3% top-bottom quintile difference). Disagreement among peer reviewers was not systematically related to submission novelty or conventionality, and editors select strongly for novelty even conditional on reviewers' recommendations (+7.0% top-bottom quintile difference). Manuscripts exhibiting higher novelty were more highly cited. Overall, the findings suggest that journal peer review favors novel research that is well situated in the existing literature, incentivizing exploration in science and challenging the view that peer review is inherently antinovelty.