Do Peer Reviewers Prefer Significant Results?

I’ve lengthy been creating about difficulties in how science is communicated and released. 1 of the most perfectly-regarded issues in this context is publication bias — the tendency for outcomes that verify a speculation to get released more effortlessly than individuals that will not.

Publication bias has many contributing things, but the peer evaluate approach is widely noticed as a essential driver. Peer reviewers, it is widely considered, are likely to search more favorably on ‘positive’ (i.e. statistically important) outcomes.

But is the reviewer desire for constructive outcomes genuinely legitimate? A a short while ago released examine suggests that the effect does exist, but it’s not a substantial effect.

Scientists Malte Elson, Markus Huff and Sonja Utz carried out a intelligent experiment to determine the effects of statistical significance on peer evaluate evaluations. The authors have been the organizers of a 2015 convention, to which researchers submitted abstracts that have been matter to peer evaluate.

The keynote speaker at this convention, by the way, was none other than “Neuroskeptic (a pseudonymous science blogger)”.

Elson et al. designed a dummy summary, and had the convention peer reviewers evaluate this synthetic ‘submission’ along with the actual kinds. Just about every reviewer was randomly assigned to acquire a model of the summary with both a important final result, or a non-important final result the facts of the fictional examine have been otherwise similar. The final sample dimension was n=127 reviewers.

The authors do go over the ethics of this somewhat unconventional experiment!

It turned out that the statistically important model of the summary was offered a better ‘overall recommendation’ rating than the non-important one particular. The variance was around one particular point on a scale out of ten, was statistically important despite the fact that marginally (p=.039).

The authors conclude that:

We noticed some evidence for a smaller bias in favor of important outcomes. At the very least for this specific convention, nevertheless, it is not likely that the effect was substantial sufficient to notably impact acceptance fees.

The experiment also tested no matter whether reviewers had a desire for primary reports vs. replication reports (so there have been four versions of the dummy summary in overall.) This revealed no variance.

Effects of significant vs. nonsignificant results

So this examine suggests that reviewers, at the very least at this convention, do in fact prefer constructive outcomes. But as the authors acknowledge, it’s tough to know no matter whether this would generalize to other contexts.

For case in point, the abstracts that have been reviewed for this convention have been limited to just 300 phrases. In other contexts, notably journal posting assessments, reviewers are delivered with much more information to foundation an view on. With just 300 phrases to go by, reviewers in this examine might have paid out notice to the outcomes just since there was not significantly else to choose on.

On the other hand, the authors take note that the participants in the 2015 convention might have been unusually mindful of the problem of publication bias, and thus more probable to give null outcomes a reasonable listening to.

For the context of this examine, it is applicable to take note that the division (and its management at the time) can be characterized as relatively progressive with regard to open up-science beliefs and practices

This is absolutely legitimate right after all, they invited me, an anonymous male with a web site, to speak to them, just on the power of my writings about open up science.

There have only been a handful of earlier reports making use of similar styles to probe peer evaluate biases, and they typically uncovered larger effects. 1 1982 paper uncovered a substantial bias in favor of important outcomes at a psychology journal, as did a 2010 examine at a clinical journal.

The authors conclude that their dummy submission system could be useful in the examine of peer evaluate:

We hope that this examine encourages psychologists, as individuals and on institutional degrees (associations, journals, conferences), to carry out experimental investigate on peer evaluate, and that the preregistered discipline experiment we have reported may well serve as a blueprint of the form of investigate we argue is required to cumulatively establish a demanding understanding foundation on the peer-evaluate approach.