Over at Unqualified Offerings, “Thoreau” offers some musings about peer review. I saw this and said, “Aha! The perfect chance to dust off an old post, and free up some time…” Sadly, I already recycled the post in question, so I feel obliged to be less lazy and contribute some new content.
I generally agree with most of what he says, but I would raise one quibble about his list of criteria:
What scientists are looking for when we evaluate a paper is whether the paper clearly addresses 3 points:
1) What is the question or issue being studied in this work?
2) What are the methods being used, and are they described in a sufficiently detailed manner so that somebody else can replicate the work? (Remember that replication is the real gold standard of scientific knowledge. Until we have independent replication of a result, it’s suspect. Hell, even after independent replication we’re still skeptical.)
3) Does the data presented support the conclusions that the author is drawing?
I think this leaves out one important question:
1.5) Is the result interesting?
“Interesting” is a loaded word, and it covers a lot of ground. It includes both the normal sense of “is anybody likely to care about this result?” but there’s also an important element of “is this a new result?”
This is probably a more significant criterion in my mind than for some other people, as most of my refereeing has been for Physical Review journals, which are in the upper echelon of physics journals. Physical Review Letters in particular asks that papers be important to the field and of general interest to physicists. It’s not uncommon to have a referee say that a paper is perfectly good, but not interesting enough for PRL. (In my opinion, it should happen more often than it evidently does, based on the large amount of highly specialized and uninteresting crap that’s in PRL, but that’s a different rant…)
This is an important point for any scientific paper, though– the work presented needs to be new (you’re not going to get a research publication out of proving that objects near the Earth’s surface fall with a constant acceleration due to gravity), and it needs to produce a result that’s interesting in some way (you’re not going to get a paper out of “If you put an atom in this type of potential field, absolutely nothing will happen to it”). The threshold for “interesting result” isn’t all that high– there’s some pretty “meh” stuff published every week– but null results are pretty much out.
Other than that, I agree with “Thoreau.” His conclusion is also worth quoting, and might generate some interest around here, though I can’t spend much time on it:
The simple fact of peer review is that it’s actually quite a modest hurdle. All you have to do is find one editor and a couple of reviewers who find the work plausible and well-executed. Once you’ve been on both sides of the process, you realize that it’s just a preliminary quality check, a first pass before it’s put out there for a wider audience. Some laymen seem to attribute too much significance to it, and other laymen seem to recoil against that misperception by concluding that peer review is too weak of a system. The truth is that it’s not supposed to be a stringent filter. It’s just supposed to be a first pass.