Refereeing mortus est, vivat refereeing

An Examination of Sources of Peer-Review Bias

Peer-review ratings of 1,983 posters submitted for three annual conferences of a professional society were examined for evidence of bias. Hypotheses derived from the literature on the better-than-average effect were tested by analyzing 7,383 sets of ratings. Reviewers who authored posters gave lower average ratings than reviewers who did not author posters. Posters having authorship that included at least one reviewer received higher ratings than those having only nonreviewing authors. Reviewers' experience and professional role were also explored as biasing factors. The ratings were converted into z scores, and differences in reliability and acceptance decisions were examined. Implications for current peer-review practices are discussed.