Reinventing Peer Review

The Authorea Team

Peer review is arguably necessary for effective communication amongst researchers.  Authors, editors, and the public rely on peer review to ensure a first measure of trust in scientific communication.  While peer review is considered to be integral in scholarly communication by most, its shortcomings are becoming evident. Former editor of JAMA and NEJM Drummond Rennie once said, "if peer review was a drug it would never be allowed onto the market." Is this true? Does peer review, as it is done today, cause more harm than good?

Is Peer Review Broken?

"Peer review is easily abused, and there are many examples of authors reviewing their own papers, stealing papers and ideas under the cloak of anonymity, deliberately rubbishing competitors’ work, and taking a long time to review competitors’ studies." - Richard Smith, former editor of British Medical Journal.

Why is the Peer Review so Slow?

The time it takes from submission to publication is on average two years. Many authors attribute the lengthy reviewing times to editor and reviewer fatigue. Decision whether the paper will be published or not can be easily delayed by reviewer unavailability. Some fields have a very small pool of experts. Even though journals try to keep a group of loyal reviewers, multidisciplinary publishers have to often reach outside of their list. Multiple rounds of reviews, outsourcing manuscript management or reviewers intentionally delaying the publication for personal reasons are cited as other possible explanations for the long reviewing process (Nguyen 2015). In the age of data-rich papers reviewers are requesting more data, revisions and new experiments than they used to which may cause further delays.
Does it take too long to publish research? Source: Nature.com

Because of the publish or perish system, extreme delays can negatively impact early career researchers. Publishing delays affect grant applications and can hinder subsequent work. In the race to publish, authors have to consider the cost of lost opportunity when other group is able to publish research faster. More importantly, delay in the dissemination of new knowledge comes as a cost to the scientific community and general public (Nguyen 2015).
A 2008 Research Information Network report estimated that the unpaid costs of peer review amount to £1.9 billion per year (Golden 2012).

Peer Review Bullying

"We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong." - Richard Horton, editor of the Lancet.
Peer review can make or break academic careers. It's even more precarious when its objectivity is threatened by cognitive and social bias linked to gender, affiliation, ethnicity, and affiliation. Suzanne C. Iacono from National Science Foundation (NSF) reveals that only 18% of African-American researchers submitting grant proposals to NFS are successful. National Institutes of Health (NIH) sees a similar pattern - researchers of African descent “receive awards at 55% to 60% the rate of white applicants”. In general women and African-Americans tend to be judged harsher than their white male peers (Kaatz 2014). 
Some types of cognitive bias and potential relevance to scientific peer review (Kaatz 2014).       
Studies have found a strong evidence of bias towards women in the process of awarding grants (Wenner 1997). Research shows that pervasive cultural traits depicting women as more community-oriented and men as more goal-oriented can lead to implicit assumption that women are less competent in typically male-dominated areas (Biernat 2012). While direct gender bias is slowly disappearing, implicit gender bias can influence even well-intentioned reviewers. Study by Lloyd shows that papers authored by women have a much higher acceptance rate when reviewed by female reviewers (62% vs 21%) (Lloyd 1990).
PloS One reviewer suggests that adding “one or two male biologists” as co-authors would improve the analysis.
To prevent this conscious or subconscious stereotyping many journals now offer double and triple-blind reviews as well as open review. Study following introduction of double-blind review by Behavioral Ecology journal showed an increase of papers with first female authors compared with other similar journals (BUDDEN 2008). Many scientists recommend journals to diversify panel boards and structure external processes to help minimize this bias.
Work by Peters and Ceci study showed that research from a well-known institution was perceived as more valuable than the one associated with an unknown institution (Peters 1982 & Ceci 2014) Even though the study was published over 30 years ago, researchers continue to uncover similar problems.  
Screenshot of sample posts with #sixwordpeerreview hashtag        

Bias Against Negative Results

Editorial peer review is strongly skewed towards positive results. While this is hardly surprising, failure to publish negative outcomes biases the information base of medicine. It means that thousands of failed experiments, null or negative results of clinical trials never see the light of day. Dr. Natalie Matosin argues that “Science is, by its nature, a collaborative discipline, and one of the principal reasons why we should report negative results is so our colleagues do not waste their time and resources repeating our findings” (Matosin 2014).
Emergence of journals dedicated to negative results such as Elsevier’s New Negatives in Plant Science is the first step to tackle the publishing bias. Word Heath Organization (WHO) is supporting the movement by encouraging researchers to publish all previously unreported results including negative findings.

Is Double Blind Review the Way to Go?

Reputation, gender or perceived power of the author may positively affect the acceptance of the paper. Double or triple blind-reviews were imposed to reduce this bias. However a high degree of specialisation in modern science does very little to preserve anonymity. Pre-publications, citations of author's previous work, and tight professional circles allow reviewers to easily guess author's identity. In one of the studies 46% of reviewers were able to correctly identify masked authors (Fisher 1994). Although effects of this guessing game have not been extensively studied, it can contribute to broad ambivalence towards double-blind peer review in science. 
Concerns are that opting for double-blind review might be seen as hiding a disadvantaged position. A survey conducted by Nature Publishing Group revealed that only 25% of submitting authors have opted for anonymity. Even though both reviewers and authors are blinded, the editors are not. In journals like Nature, editors conduct first screening of the papers and can reject them prior to peer review. Anecdotal evidence suggests that editors might feel inclined to recommend authors they met in the past, are familiar with, or who come from a well known institution.
Peerage of Science is among journals that experimented with triple-blind peer review. Unfortunately it proved to be "too big obstacle for most editors". In a press statement issued earlier this year Peerage of Science's founder, Janne-Tuomas Seppänen explained that "the leap of faith to trust fully anonymous peer reviewers was too much for most. The fact is, while we know that our reviewers are qualified, the editors did not."

Recognition for Review

It is rare for reviewers to get public recognition for their efforts. Nor are they mentioned when the paper is published. This time consuming and often labor intensive work is done out of duty to accelerate science. Level of anonymity varies, but in general the identity of the reviewer is never disclosed.
Platforms like Reviewer Recognition Platform and Publons are slowly changing this trend. Researches can track their review performance and receive various perks based on the number of completed reviews. Perks include badges, certificates of recognition, and discounts. More and more journals publish reviewers names as part of annual list. Reviewers can also claim Continuing Medical Education (CME) and Continuing Professional Development (CPD) credits.
How should the reviewers be recognised and rewarded? What incentives should be put in place to encourage them to spare time in a continuous race to publish? When does the system becomes too incentivised?

Collaborative and Open Peer Review

Science is continuously evolving - following new feedback papers should be transparently updated and republished. In the open peer review both authors and reviewers are named, and comments on the manuscript are publicly available. As former British Medical Journal editor Richard Smith points out, the research should be published online to let “the world…decide what’s important and what isn’t”.
Several journals including Nature and OBM experimented with open peer review. ACP Editor Ulrich Pöschl lists a number of positive impacts of OPR:
  1. Quick knowledge dissemination and uncensored scientific communication
  2. A way to officially document controversy and discussions surrounding the manuscripts
  3. A way to reduce plagiarism
  4. A higher chance to discover manuscript flaws
  5. Final work is based on joint work of authors, reviewers, and the interested public 

The Authorea Way

We find it hard to believe that in the digital age some papers might take a year or more to get out of the lab and into the world. Authorea is actively looking for ways to make it more efficient and transparent. Using our platform researchers can write preprints as peer review, annotate documents openly, and directly add editors to Authorea articles. Git-powered document versioning allows for more dynamic communication and gives a full view of article history and individual contributions.
It's happening already: a paper written on Authorea was submitted to a major journal where it is linked back to the Authorea article allowing readers to leave feedback.

Peer Review Week

Help us reinvent peer review. Get involved in the Peer Review Week and review our Open Peer Review article. How would you fix Peer Review? Is it fixable? What measures would you put in place? Is OPR the way to go? Let us know what you think: comment, edit, discuss, and contest.

Finished article will be published on Authorea with full recognition given to its reviewers.

References

  1. bibliography entry failed to display
  2. Mary Golden, David M. Schultz. Quantifying the Volunteer Effort of Scientific Peer Reviewing. Bull. Amer. Meteor. Soc. 93, 337–345 American Meteorological Society, 2012. Link

  3. Anna Kaatz, Belinda Gutierrez, Molly Carnes. Threats to objectivity in peer review: the case of gender. Trends in Pharmacological Sciences 35, 371–373 Elsevier BV, 2014. Link

  4. Christine Wennerås, Agnes Wold. Nepotism and sexism in peer-review. Nature 387, 341–343 Nature Publishing Group, 1997. Link

  5. Monica Biernat. Stereotypes and Shifting Standards. 1–59 In Advances in Experimental Social Psychology. Elsevier BV, 2012. Link

  6. Margaret E. Lloyd. Gender factors in reviewer recommendations for manuscript publication. J Appl Behav Anal 23, 539–543 Society for the Experimental Analysis of Behavior, 1990. Link

  7. A BUDDEN, T TREGENZA, L AARSSEN, J KORICHEVA, R LEIMU, C LORTIE. Double-blind review favours increased representation of female authors. Trends in Ecology & Evolution 23, 4–6 Elsevier BV, 2008. Link

  8. Douglas P. Peters, Stephen J. Ceci. Peer-review research: Objections and obligations. Behavioral and Brain Sciences 5, 246 Cambridge University Press (CUP), 1982. Link

  9. Stephen Ceci, Douglas Peters. The Peters & Ceci Study of Journal Publications. The Winnower The Winnower LLC, 2014. Link

  10. N. Matosin, E. Frank, M. Engel, J. S. Lum, K. A. Newell. Negativity towards negative results: a discussion of the disconnect between scientific worth and scientific culture. Disease Models & Mechanisms 7, 171–173 The Company of Biologists, 2014. Link

  11. bibliography entry failed to display

[Someone else is editing this]

You are editing this file