Scientific Communication

Peer Review in Scientific Publishing

When you go to see your doctor with a health complaint, he or she might make a preliminary diagnosis and then send you to a specialist to get a more detailed exam. For example, if you have foot pain, you might be sent to a podiatrist. A rash might be indicative of a skin disorder and merit examination by a dermatologist, or it may be due to some type of allergy, thus calling for an allergist's opinion. Patients complaining of chest pains might be sent to multiple specialists, including a cardiologist, pulmonologist, gastroenterologist, or others depending on the nature of their pain. General practitioners know that there are far too many different and highly specific medical afflictions for one person to understand them all in detail, so they rely on a team of specialists to keep their patients healthy. Similarly, scientists rely on a team of other scientists, each with his or her own specialty, to review and comment on research before it is published, funded by grant agencies, presented at scientific meetings, or otherwise disseminated to the scientific community.

Scientific journal editors (see our Understanding Scientific Journals and Articles module), funding program directors, and chairs of scientific meetings receive many highly diverse and detailed research submissions from scientists. For example, in just one issue of the journal Science (Figure 1), research articles can be found on subjects as diverse as the genetic factors that influence body size in dogs, the influence of changing climate on the fruiting patterns of fungi, the geomorphology of ice deposits on Mars, and new technology to generate small electrical currents using ultrasonic waves (Science, 2007). It would be impossible for one person to possess the background and experience necessary to review and evaluate these submissions. As a result, journal editors rely on the process of peer review, in which they ask other scientists who are specialists in those fields and are therefore likely to be familiar with methods presented in those papers to review and comment on the research.

Peer reviewers provide comments regarding the validity of the methods used, the reasonableness of the data analysis techniques, the logic of the interpretations drawn by the authors, and the quality of the writing and graphics. Peer reviewers commonly make recommendations regarding whether they feel a manuscript should be published or a research presentation should be made at a scientific meeting, or, in the case of a grant opportunity, whether a study should be funded. Peer review is also used by academic and other institutions to help provide feedback when a scientist is seeking to earn tenure or promotion to a higher rank.

History and development of peer review

The origin of peer review is closely tied to the development of scientific institutions and scientific journals. When the Royal Society of London (see our Scientific Institutions and Societies module) began publishing its journal Philosophical Transactions in 1665 (Figure 2), the review process was solely the responsibility of the editor (at the time, Henry Oldenburg). The editor sometimes enlisted other scientists to provide opinions as necessary, but it was at his sole discretion whether works were reviewed by others or not. In 1752, the Society itself took over the editorial responsibilities for the journal and instituted a review policy wherein each manuscript was sent to a small group of experts in the field before being published (Spier, 2002). This practice was the beginning of the peer review process as we know it today.

The process spread sporadically through the sciences in the nineteenth century, but became a standard for scientific review in the mid-twentieth century. Some have linked the spread of peer review in the twentieth century to the popularization of Xerox® photocopying machines, which facilitated the replication and distribution of manuscripts to reviewers (Spier, 2002). Whatever the motivation, both Science and the Journal of the American Medical Association implemented consistent peer review practices in the 1940s, and other scientific journals followed suit.

Today, more than 5,000 different scientific journals exist, and each of them receives many manuscripts from scientists who feel their work should be published. The journal Science alone receives as many as 200 manuscripts per week. To evaluate the appropriateness of each manuscript for publication, journal editors identify two or three peer reviewers to read and comment on each submitted manuscript.

Peer reviewers are knowledgeable scientists who are not directly involved with the research being evaluated but who are familiar with the field of study or the research methods used. The reviewers may not know the scientists who wrote the paper or grant proposal, they may be colleagues, or they may even be scientific competitors with the authors. So, in an attempt to remove bias from the review process, most manuscripts are independently considered by multiple reviewers, and it is left to the journal editor to make the final decision regarding publication based on the reviewers' comments.

Reviewers consider the relevance of the work to the journal or funding program being targeted, the validity of the methods and interpretations, the originality of the finding, and the clarity of the writing. Reviewers then provide feedback, generally in written form, on the work they have read. And these comments are commonly sent back to the authors so that the authors can use the reviewers' comments to refine and improve the text of their manuscript or their research. Reviewers can often choose whether they wish to remain anonymous to the authors or not. Journal editors and grant program directors rely on the reviewers' feedback to guide their decisions – they may choose to accept a work as is, they may ask the author(s) for revisions, or they may reject the work based on the peer review comments.

Peer review in practice

On January 22, 2007, Conrad Mauclair and colleagues submitted a manuscript entitled "Quantifying the effect of humic matter on the suppression of mercury emissions from artificial soil surfaces" for consideration to the journal Applied Geochemistry. The manuscript was sent out by the editor to two peer reviewers, who were given one to two months to complete the review. The reviewers sent comments to the editor, and after considering the reviewers' comments, the editor chose to accept the manuscript with revisions, and responded as such to the authors approximately five months after their initial submission. Excerpts from the letter the editor wrote to the authors detailing this decision are printed below.

The reviewers' comments were attached to the editor's letter with the names of the reviewers removed. In the case of journal manuscripts accepted with revision, the authors have the opportunity to read and respond to the reviewers' comments and make changes to the article in question. In the case of grant submissions, scientists read the reviewers comments and use these to strengthen their submission the next time they apply for a grant. For the article by Mauclair and colleagues, the reviewers had a number of recommendations for improving the article, as the excerpts provided below detail.

Once comments are received regarding a manuscript, it is up to the authors to address those comments, or in cases where they disagree with a reviewer's comment, provide an explanation as to why they have not addressed the comment. In the above case, the author's addressed the majority of the reviewers' comments and sent a letter back to the editor on June 10, 2007, detailing the changes made to the article and discussing why some changes were not made:

The comments from Reviewer 2 highlight the fact that peer review helps the scientific publishing system to assure that manuscripts meet certain minimal standards. Reviewer 2 commented on the originality of the submission, the perceived importance of the work in the field of science, the manner of presentation of the writing in the text, the quality of the figures and tables and data analysis in general, whether he/she found any serious flaws in the work, and specifically the appropriateness of the manuscript title and abstract since these are the parts of the paper that will be cataloged by literature databases (see our Utilizing the Scientific Literature module) and thus widely read.

In addition, reviewers may recommend that authors clarify the text or add certain references that they had not previously considered; they might suggest changes because they feel that the authors' interpretations are not supported by their data; they may recommend additional research to clarify questionable points in the study; or they may recommend that a manuscript be rejected completely because of questions about the research methods, data collection, or interpretation.

Similarly, grant proposal reviewers may make specific recommendations for improving a study and recommend that the authors resubmit their proposal in another grant cycle after it has been improved. Grant reviewers might also recommend that more background research be conducted before the authors submit their proposal again, that another scientist with a different expertise be included on the research team, or that the scope of the research be broadened (or narrowed).

Mauclair and colleagues made the majority of the changes requested by the reviewers to their manuscript. They revised the title, added additional explanation to the text, and even conducted additional experiments to satisfy a question raised by Reviewer 1. In the reviews, Reviewer 2 calculated a mass balance of mercury in the experiments and concluded that the samples lost more mercury than was added to the system; the editor then followed this comment with a suggestion as to why this might have occurred and suggested reporting the data in a different manner. The authors address this comment by correcting an erroneous assumption of the reviewer, and explain why they have chosen not to report the data in a different manner as suggested by the editor. Thus, while they did not follow the suggestion made in the review, they provide a detailed explanation as to why.

Once a revised manuscript is resubmitted, the editor reads the authors' response, and if satisfied that the authors adequately addressed all of the issues raised, moves the manuscript forward in the journal's publishing cycle. In some cases, where major revisions are required, the editor may redistribute the manuscript to the peer reviewers a second time before accepting it as complete. The Mauclair manuscript was accepted for publication in August 2007, was first published on the journal's website in January 2008, and was finally published in the March 2008 issue of the printed version of the journal (Mauclair et al., 2008).

Implications of peer review

It is worth noting the lengthy time frame involved in publishing scientific articles. From the initial submission to the final printing, the described article took 14 months, which does not even include the time spent doing the initial work that led to the publication. The sluggishness of the peer review process is often criticized, but it reflects the understanding that published work enters the scientific literature permanently, as work that can be built upon by other scientists, and thus should be carefully considered.

Additionally, journals and funding agencies vary in their selectivity and research focus. Consequently, scientists choose where to submit their manuscript based on the perceived impact of the research, the likelihood of acceptance, and the size of the audience they wish to reach. In turn, reviewers consider the appropriateness of the research to the journal's audience. For example, while the journal Applied Geochemistry focuses on research articles that discuss chemical transformations and process that take place in the environment, the journal Cell publishes articles focused on biological process related to cell function. The article by Mauclair and colleagues was published in Applied Geochemistry; however, it would likely have been rejected by Cell.

As part of the scientific process, reviewers are expected to keep the information in a manuscript confidential until it is published, but it is rare that the work comes as a complete surprise to the entire scientific community. This is because peer review is integrated into almost every step of science, including requests for public funding for research. Funding decisions are made by a committee of peer review scientists who debate each proposal's likelihood of success, the validity of its approach, and the importance of the question being asked. The research methods and ideas published by Mauclair and colleagues in 2008 were reviewed as part of a grant submitted to the Research Foundation of the City University of New York, which was funded in 2005 (Carpi & Frei, 2005). Once funded, the research begins, and preliminary data may be presented at scientific meetings. This allows the findings to be described and debated with colleagues prior to publication. Data that were eventually used in the manuscript were presented to the scientific community in August 2006 at a large international conference on the pollutant mercury (Mauclair et al., 2006), and thus the final publication was anticipated by some scientists in the field.

Peer-reviewed publications and funded research proposals carry significance for the individual scientist beyond simply doing science. In many cases, hiring, promotion, and award decisions are made on the basis of the number and quality of peer-reviewed publications or grants authored by an individual. Scientists also benefit in many ways from serving as peer reviewers – being asked to review a manuscript or proposal is an acknowledgement of one's expertise in an area. All scientists both receive reviews from their peers and review the work of others, and this process comes with a cost. A recent report by the Research Information Network estimated the cost of volunteered time provided by scientists for peer review at $3.7 billion (Research Information Network, 2008). So why do scientists volunteer so much time to this process? Because it is one of the obligations of their profession and one factor that helps build the community of science (see our Scientists and the Scientific Community module).

One of several mechanisms of validating science

Peer review is just one of several mechanisms embedded within the process of science that help validate the work of scientists. While it helps to validate journal and grant submissions, it is not a fool-proof filter that assures quality in scientific publishing, especially when the authors of a study are engaged in fraud or deception (see our Scientific Ethics module). For example, between 2000 and 2003, Jan Hendrick Schon and colleagues published over 25 papers on superconductivity, all of which passed through the peer review process. After several of the papers were published, Professor Lydia Sohn and Professor Paul McEuen noticed that different experiments carried out under very different conditions and published in different papers displayed the same background error (Figure 4). When confronted with the problem, Schon first claimed that a graph had been mistakenly reproduced in several papers. Shortly thereafter, Bell Labs, the research institute where Schon worked (see our Scientific Institutions and Societies module), conducted an investigation; they found numerous instances of misconduct and fraud and consequently fired Schon. At least sixteen of Schon's papers have since been declared to be false, and the journal Science has withdrawn eight of his papers (Bao et al., 2002).

While cases of scientific misconduct can be embarrassing because of the publicity they receive, they highlight the self-correcting nature of science. A key aspect of science is that research results must be reproducible and well-documented. Instances of scientific misconduct that have gotten through the peer review system are often quickly exposed when other scientists scrutinize the data and attempt to reproduce the results. To keep this system working, scientific articles include detailed descriptions of research protocols that enable others to reproduce experiments, and they include tabular or graphical presentations of data so that they can be scrutinized by the community at large (see our Understanding Scientific Journals and Articles module). One of the first pieces of evidence that raised suspicion over Schon's work was the fact that other scientists had trouble reproducing his experiments with similar results. As occurred with most of Schon's publications, scientific research articles can be retracted if they are found to be in error (whether or not that error is a result of misconduct), thus removing them from the literature of science.

The truth is that retractions are rare, and retractions due to scientific misconduct are even rarer. In an analysis of the scientific literature cataloged in the MEDLINE database maintained by the National Library of Medicine, Sara Nath and colleagues identified 395 articles that were retracted in the two decades from 1982 to 2002 out of over 8.5 million citations listed in the MEDLINE database for that same period (Nath et al., 2006). Of those articles retracted, only 27% were found to be cases of scientific misconduct, 62% were identified as unintentional errors, and 11% of retractions could not be categorized. Nath and colleagues also found that an additional 2,772 errata were published during this same period, which are simple corrections of small mistakes in published manuscripts.

Consequences of peer review

One of the consequences of the peer review system is that it can influence the dissemination and progress of scientific research. It is the peer reviewers that make recommendations as to what research is published in what journals. And it is the peer reviewers that influence the types of research studies that receive funding. This is generally a positive effect as it opens the process to the scientific community at large. But bias among reviewers can negatively impact this process. For example, some researchers have suggested that peer reviewers can be biased in favor of research that reports positive effects (i.e. that drug x has an effect) over research reports that report a no effect (i.e. drug y has no significant effect) (Callaham et al., 1998). Thus, published studies showing positive effects far outnumber ones showing none. Another complication that affects the peer review process is that in a closed review system, where reviewers are kept anonymous from the authors, it is possible for reviewers to pass unnecessarily harsh judgment – or unworthy praise – on a manuscript or application for funding, simply because they have personal differences or friendships with authors.

The widespread use of electronic publishing has prompted a recent re-evaluation of the peer review process as a whole. While scientists still largely agree on the value of peer review, they are sometimes discouraged by the length of time involved from submission of a manuscript, through review, revision, and resubmission, which may take a year or more – as in the case of the article described here. As a result, some authors have suggested that the time-consuming and closed pre-publication peer review process be abandoned entirely in favor of open access, online publishing that allows for constant reviews and updates. A number of scientific publishing media are adapting to the changing nature of publishing. For example, the Public Library of Science (PLoS) project publishes a number of journals in the PLoS family that make their formal peer reviews available to the public and then they further provide a mechanism for additional public comment and review of published articles on their website. Even the journal Nature, which has been published for over 130 years, has recently experimented with open access peer review for submitted manuscripts.

Publications without peer review

There are publications in the sciences that are not peer-reviewed. For example, many journals, including Science and Nature, publish news and commentary sections in which they provide weekly updates on major scientific events or issues. A number of journals also publish "letters." Letters to journals include commentary on previously printed articles, but they may also report new, preliminary and intriguing scientific results that have not yet been tested and replicated enough to pass full peer review.

Scientists also write material specifically for non-peer reviewed publication. For example, the evolutionary biologist Steven J. Gould was a prodigious writer and became well known for his books and magazine articles on topics ranging from evolution to baseball. The theoretical physicist Stephen Hawking is well known for his books aimed at explaining cosmology, which include such popular titles like A Brief History of Time and The Universe in a Nutshell. And the astronomer Carl Sagan not only wrote numerous popular magazine articles, but also authored the best-selling book Contact, which was turned into a blockbuster film. The articles and books that scientists write for sources other than peer-reviewed journals have an important, but very different, purpose than the peer-reviewed literature. These pieces are often directed at explaining science in more common language to non-scientists and thus serve a crucial role in describing the impact of science to the general public. As such, they are generally, but not always, based on the peer-reviewed literature that forms the basis of our scientific knowledge.

Summary

Peer review is an important part of the process of science. This module describes the history of peer review and shows how the review process helps validate the work of scientists and ensure that the quality standards are met. The process is illustrated by actual correspondence among authors, reviewing scientists, and the editor of a scientific journal.

Key Concepts

• Scientific manuscripts and funding proposals are reviewed by several peer scientists who are familiar with the field of research and who make recommendations on whether or not the work should be published and/or funded.
• Peer review works on many levels and is a fundamental component of the process of science.
• After publication, scientific papers and other forms of research dissemination are further scrutinized by the scientific community when scientists read or try to reproduce the research.
• Scientists conduct peer review as part of their responsibility to the scientific community, and are themselves evaluated by the peer review process.

QUIZ

1. The process of peer review verifies whether or not research described in a manuscript is correct.

(A)__true

(B)__false

2. Peer reviewers are

(A)__senior scientists.

(B)__colleagues.

(C)__junior scientists.

(D)__All of the choices are correct.

3. Which of the following is not a function of peer review?

(A)__Peer reviewers comment on the originality of the research.

(B)__Peer reviewers comment on the impact, or importance, of the research.

(C)__Peer reviewers make recommendations to improve tables and figures.

(D)__Peer reviewers comment on the popularity of the scientists conducting a study.

4. When responding to peer review comments, the authors of a manuscript must make all of the changes the reviewers recommend.

(A)__true

(B)__false

5. In addition to peer review, what else is scientific research scrutinized?

(A)__By a panel of government experts.

(B)__By a team of University reviewers

(C)__By the entire scientific community after publication.

(D)__By the American Association for the Advancement of Science.