nature re-jiggers peer review 06.21.2006, 8:21 AM
posted by ben vershbow
Nature, one of the most esteemed arbiters of scientific research, has initiated a major experiment that could, if successful, fundamentally alter the way it handles peer review, and, in the long run, redefine what it means to be a scholarly journal. From the editors:
...like any process, peer review requires occasional scrutiny and assessement. Has the Internet bought new opportunities for journals to manage peer review more imaginatively or by different means? Are there any systematic flaws in the process? Should the process be transparent or confidential? Is the journal even necessary, or could scientists manage the peer review process themselves?
Nature's peer review process has been maintained, unchanged, for decades. We, the editors, believe that the process functions well, by and large. But, in the spirit of being open to considering alternative approaches, we are taking two initiatives: a web debate and a trial of a particular type of open peer review.
The trial will not displace Nature's traditional confidential peer review process, but will complement it. From 5 June 2006, authors may opt to have their submitted manuscripts posted publicly for comment.
In a way, Nature's peer review trial is nothing new. Since the early days of the Internet, the scientific community has been finding ways to share research outside of the official publishing channels -- the World Wide Web was created at a particle physics lab in Switzerland for the purpose of facilitating exchange among scientists. Of more direct concern to journal editors are initiatives like PLoS (Public Library of Science), a nonprofit, open-access publishing network founded expressly to undercut the hegemony of subscription-only journals in the medical sciences. More relevant to the issue of peer review is a project like arXiv.org, a "preprint" server hosted at Cornell, where for a decade scientists have circulated working papers in physics, mathematics, computer science and quantitative biology. Increasingly, scientists are posting to arXiv before submitting to journals, either to get some feedback, or, out of a competitive impulse, to quickly attach their names to a hot idea while waiting for the much slower and non-transparent review process at the journals to unfold. Even journalists covering the sciences are turning more and more to these preprint sites to scoop the latest breakthroughs.
Nature has taken the arXiv model and situated it within a more traditional editorial structure. Abstracts of papers submitted into Nature's open peer review are immediately posted in a blog, from which anyone can download a full copy. Comments may then be submitted by any scientist in a relevant field, provided that they submit their name and an institutional email address. Once approved by the editors, comments are posted on the site, with RSS feeds available for individual comment streams. This all takes place alongside Nature's established peer review process, which, when completed for a particular paper, will mean a freeze on that paper's comments in the open review. At the end of the three-month trial, Nature will evaluate the public comments and publish its conclusions about the experiment.
A watershed moment in the evolution of academic publishing or simply a token gesture in the face of unstoppable change? We'll have to wait and see. Obviously, Nature's editors have read the writing on the wall: grasped that the locus of scientific discourse is shifting from the pages of journals to a broader online conversation. In attempting this experiment, Nature is saying that it would like to host that conversation, and at the same time suggesting that there's still a crucial role to be played by the editor, even if that role increasingly (as we've found with GAM3R 7H30RY) is that of moderator. The experiment's success will ultimately hinge on how much the scientific community buys into this kind of moderated semi-openness, and on how much control Nature is really willing to cede to the community. As of this writing, there are only a few comments on the open papers.
Accompanying the peer review trial, Nature is hosting a "web debate" (actually, more of an essay series) that brings together prominent scientists and editors to publicly examine the various dimensions of peer review: what works, what doesn't, and what might be changed to better harness new communication technologies. It's sort of a peer review of peer review. Hopefully this will occasion some serious discussion, not just in the sciences, but across academia, of how the peer review process might be re-thought in the context of networks to better serve scholars and the public.
(This is particularly exciting news for the Institute, since we are currently working to effect similar change in the humanities. We'll talk more about that soon.)
Jesse Wilbur on June 21, 2006 9:30 AM:
Really the question is, "what took them so long?" As journals have increasingly moved over to the web to present their materials (behind subscription blocks or not) it seems that web-based tools (forums, comments, chat) would inevitably transform print based isolation into a more collaborative effort. It's great to see such a well known journal spearheading this effort - let's hope some others catch on.
Sally on June 21, 2006 11:22 AM:
The NINES project will also explore open source peer review and comprehensive metadata specifically for 19th century scholarship: www.nines.org (unforunately, it's down this week)
Miguel Guhlin on June 21, 2006 2:44 PM:
Howdy! We've been having this conversation over here:
Thank you for adding this component to it!
Around the Corner - MGuhlin.net/blog
bowerbird on June 21, 2006 4:36 PM:
pre-publication peer-review was a big necessity
back when "publication" meant "print", because
print is an expensive and thus scarce commodity.
if you can only print up _some_ of the stuff,
it makes perfect sense to print up "the best".
but with our new unlimited cyber-shelf-space,
"publication" is no longer scarce or expensive.
so we can let _everything_ see the light of day.
which is good, because sometimes we didn't know
(at the time) which stuff really was "the best"...
of course, since the goal of science is "truth"
-- ok, let that homily pass without challenge --
we still have an aching need for _evaluation_...
but that evaluation can occur _after_ publication.
post-publication peer-review is some hot shit...
peer review? yes!
can happen more quickly
when new models don't have
to fight for the right
to simply exist...
Jeffrey Ellis, MD on August 23, 2006 11:45 AM:
The development of effective & sustainable models of scholarly communication that provide barrier-free access to quality information is a challenge for us all.
Post publication peer review, and post publication dialogue greatly enhances the depth of understanding one can obtain from published literature.
http://www.JournalReview.org is one such project that allows physicians and scientists to enter into a discussion with colleague's world wide about any literature published via PubMed.
Participation is certain to enrich the communities understanding of the scope of published works.