Peer Review Does Not Define Science

31 comments

1. I completely disagree with you on this Prof. Orzel! Peer revue in the widest sense has always been the basis of what is accepted within science as science. A consensus amongst your peers is that which establishes or ruins your reputation as a scientist. This has always been so and will always remain so. It is not a perfect system and there are many examples of the scientific community ignoring or even rejecting theories that later became standard mainstream science. I am not a big fan of Thomas Kuhn, and largely reject his whole concept of scientific revolutions and paradigms, but one thing that he did get very right is the concept of the scientific community and its role as final arbiter as to what is and what is not science.

   Put very simply something is science because the majority of scientist say it is science.

2. 100% yes.

   This is key to science education, too: building an understanding of what it means to ‘do science’. Walk through a science fair and you’ll see hits and misses on this concept. For every “What food is best for plants?”, you’ll see another “All about surface tension.*” The latter explores a science concept, while the former is really doing science. And I’d argue that’s much more important.

   *OK, that was my 6th grade science fair project. I should have known better. (But maybe that’s the point!)

3. Peer review is a relatively new thing – Nature mag started implementing it only in the late 1960s, the reason being to clear up the clutter of piles and piles of unpublished papers stacked up ceiling-high in the offices (editors were overwhelmed, so they engaged outside scientists to help them out with the job). The idea of peer review as “quality control” came later.

4. Sorry Thony C, but I completely agree with Prof. Orzel. Aristotle was never published in peer reviewed journals, but he was still a scientist. An NYT article with published tests and data would be science, because the results could be duplicated. Pseudoscience falls down at this hurdle, not because it isn’t peer reviewed, but because it never could be due to a lack of transparency.

   Saying “something is science because the majority of scientist say it is science” sounds very much like a caste of priests dictating from on high. It does not sound like science.

5. This peer review holiness, is odd. Correctness is what we want. I don’t want to know whether someone checked a paper, I want to know if the result is accurate. Peer review is a GOOD THING but it is only a thing that helps us have better confidence about a paper in a journal. If tomorrow the UN banned all peer reviewed journals in the world would science collapse? No it would just find another way to get confidence in given results. Today with the assault of the reality challenged wackos in areas like vaccines, climate, evolution, and so on peer review is is often used to at least keep areas for real discourse. That, to me, is its real purpose.

   Absolutely the NYT should be doing science. They should be testing things, not just asking around for a review. They should also be doing part of the review themselves. When a study comes out or someone makes a claim, where are the reporters trying to tell how much credence we should give. What are the limits, how was this done, is it internally consistent? Does it contradict already known facts. This is also part of reporting. The mass media has abandoned this idea if it ever had it.

6. While I agree that it is important for the media to “do” science accurately, I can’t agree that they MUST collect and present original data. (Working in publishing has given me some insight into budgetary constraints, and I can guarantee that the NYT is not going to outfit all of its lifestyle writers with geiger counters and science labs so they can collect data to confirm their stories.) What’s more, I think in many cases it’s counterproductive for them to do that. What’s more important (in my mind, at least) is for the media to present science ACCURATELY. If someone claims that your countertop is going to give you radiation poisoning, the job of a good reporter is not necessarily to get a geiger counter and wander through Home Depot. However, the job of the good reporter IS (and always has been, and always should be) to find FACTS to back up his or her position.

   This is the problem with all of the pseudoscience articles out there (e.g., so-called “investigative reports” on the nightly news that interview someone who fainted after getting a vaccine and now claims to be an expert on the “dangers” of vaccination). The reporters do not check their information. They don’t bother to consult with anyone who actually does know the facts.

   I don’t think it’s reasonable (and I don’t think it’s what’s being suggested here) to expect every reporter to conduct experiments to collect data for an article. But I do think it’s reasonable to expect said reporter to at least talk to someone who HAS actually collected the data, and to give that person’s insight some weight in the story.

   It’s just as much our responsibility as scientists to provide accurate and understandable information to the media as it is the media’s responsibility to seek out said accurate information.

7. Add me to the “disagree” group.

   The NYT article wasn’t science. It was also very bad journalism. Asking someone who’s been trained in, say, journalism to learn enough radiation physics to experiments on granite countertops is way beyond possibility.

   The article just repeated ludicrous assertions by others. Repeating the nonsense that other people say is a problem with *journalism*. Ask the writers and editors to do good journalism. Leave the science to the scientists.

8. Coturnix: Peer review is a relatively new thing – Nature mag started implementing it only in the late 1960s, the reason being to clear up the clutter of piles and piles of unpublished papers stacked up ceiling-high in the offices (editors were overwhelmed, so they engaged outside scientists to help them out with the job). The idea of peer review as “quality control” came later.

   As much as I’m against the fetishization of “peer review,” I also don’t want to sell it short. It really is essential for modern academic and institutional science– there’s simply no way for any one editor to have broad enough expertise to be able to identify interesting and correct results in even a field-specific journal, let alone Nature. They need to be able to farm things out to people who are experts.

   At the same time, though, the peer evaluation that really matters is not manuscript review, but replication of results. The real standard for scientific facts isn’t whether a referee thought the paper was worth publishing, but whether other people get the same result when they do the same experiment.

   Jacob, responding to Thony C.: Saying “something is science because the majority of scientist say it is science” sounds very much like a caste of priests dictating from on high. It does not sound like science.

   That’s exactly my point, said better than I said it. Thank you.

   Mr. Troll, making the same point as Dr. Kate: The NYT article wasn’t science. It was also very bad journalism. Asking someone who’s been trained in, say, journalism to learn enough radiation physics to experiments on granite countertops is way beyond possibility.

   They don’t need to be experts in the field– if nothing else, they could always ask somebody to do the tests for them. But it shouldn’t be acceptable for them to just say “Well, here are a bunch of stories other people told us about their countertops.” At a minimum, the question deserves to have somebody do what the Health Physics Society did, and go look at a bunch of granite samples.

   “Do granite countertops emit dangerous radiation?” is a question of fact, and it ought to get a scientific answer.

9. I agree with that, Chad. I am on your side in this discussion 100%. I just wanted to point out that formalized peer-review is relatively new development in science publishing.

10. I disagree, mostly. I agree that peer review itself isn’t the core of science, but I also think that the scientific establishment is such an important part that it is best to include it in the definition.

    If a loner genius makes a great discovery, but refuses to publish, that doesn’t yet count as science. For it to be science, people have to know about it, and it needs to be confirmed. Until then, the discovery is useless, and indistinguishable from crackpottery.

    Oh sure, we have science education, science popularization, and any pre-teen can do science for their school’s science fair. But this is only science in the abstract sense, in that they’re following the scientific method. This is the same sense in which we might call political science a science. After all, political scientists also come up with ideas and try to test them. Add to that philosophy, economics, and more.

    It is also the same sense in which we would say that the resurrection of Jesus is a scientific claim (yes, I went there!). It’s an idea, and we could theoretically test it.

11. I agree that peer review itself isn’t the core of science, but I also think that the scientific establishment is such an important part that it is best to include it in the definition.

    Isn’t the key point that peer review is an important part of being a professional scientist? Not everything that is science is professional, and so you don’t need to have something peer reviewed for it to be meaningful science.

    Sure, for work to enter the canon, publication in a peer reviewed journal is the professional gold standard. But it’s not the only standard by which a result can be judged to be scientifically useful or not. As you say, the key thing is that “people have to know about it, and it needs to be confirmed.”

    Mythbusters is the canonical counter-example to “peer review == science” we keep coming back to, and it does exactly what you say: It tests an idea, and people know about it. As long as the process includes this element of testing, the result is science, even though it might not be going-into-my-tenure-package level science.

12. I agree with Chad and Scott H. The key thing is to formulate the question, answer it, and publish the result in enough detail that somebody else who wanted to replicate the result could do so.

    For example, most professional scientists (including myself) occasionally attend conferences where we present our latest work and find out about the latest work of our friends and colleagues. These conferences are an important part of how science is done, at least in my field, but much of what is presented at these conferences has not (yet) been peer reviewed.

    Peer review does not imply that a paper is correct. It merely implies that experts in the field (between one and four, depending on journal and circumstances) have looked at the manuscript and believe the results to be (1) significant enough to warrant publication and (2) supported by the underlying data. The system is still vulnerable to honest mistakes and reviewer laziness (more than once I have read an article and reacted along the lines of “how did this get past the referees?”) as well as fraud.

    One major weakness I see in the peer review system is that there is a perverse incentive at work. Because most journals use anonymous refereeing (some journals in my field allow referees to identify themselves after the fact, but during the review process they are only known to the authors as Reviewer #N), there is no reward for doing a good job of it–in fact, the best reviewers are typically rewarded with more review work, and the time you spend refereeing a paper is time you aren’t spending doing something else (your own research, teaching, spending time with the family, etc.). I know some scientists who are already feeling the strain, and I expect it to get worse as budgets tighten.

13. the fact that einstein’s articles were not peer reviewed suffices to close the case

    http://usefulchem.blogspot.com/2008/07/einstein-rejected-peer-review.html

14. Peer review may work in physics but in evolutionary biology it is a ‘paradigm enforcer’ and an obstacle to real theoretical progress

    http://darwiniana.com/2008/08/12/peer-review-and-evolution/

15. If you cannot defend yourself before a knowledgeable peer then your work is too weak to enter the world. Peer review tosses the worst of the bad ideas, tosses the best of the good ideas, and lavishes praise upon mediocre ideas. Aim for the 75th %-tile or present an experimental observation that persists no matter who doesn’t like it.

    Anybody who criticizes peer review has volunteered to install an improvement. Submit your suggestion to peer review.

16. Modern science needs peer review like modern society needs laws. It doesn’t make science perfect, but it keeps standards high enough that by and large, the work of others can be trusted and cited confidently. Without peer review in this day and age, there would be no initial check in place – made by people with suitable expertise – against flaws in reasoning, poorly worded arguments, mistakes in methodology, failure to cite past literature, and fraud. Broadly speaking, every paper I’ve submitted – and every one I’ve reviewed – is improved by the crucible of peer review. Last but not least, without peer review the pseudoscientific hokum of Intelligent Design and Creation Science might actually be treated, at least initially, as valid science.

    Put another way, peer review is the core of communication of science.

17. I agree, but I suspect not many other pro-science people would, though.

    Reliance on peer-review is simply a heuristic, much like Occam’s Razor is simply a heuristic. Peer-review does not guarantee correct results. Lack of peer-review does not guarantee erroneous results.

    Others might say you need to use a different heuristic, such as journal impact factor.

    I think those heuristics are fine if you only want to superficially evaluate the state of a scientific debate. If you’re really interested in a subject, however, I’d suggest a more accurate method is to try to understand the points and counter-points of the debate, evaluate the logic of the arguments, methodology, look at data, etc. And yes, I think it’s generally possible for lay people to this to some extent.

    Dismissing a claim on the basis that it is not peer-reviewed might be appreciated by peer-reviewed scientists, but I don’t think many other people buy it as a proper way to address a point. That’s why I don’t often use that as a way to quickly deal with claims I don’t agree with.

18. Tēnā koutou katoa!

    Publishers of so-called scientific journals are much the same as any other publisher in many respects. They have their reputations to look after. Those reputations are earned in the same way as any publisher earns a reputation.

    If a so-called scientific journal has earned a good reputation, then so-called scientists will want to have their papers published by that journal and it will flourish. This is a peer review of sorts, but it is one that is bound by financial success of the publisher decided by popularity.

    The acid test here is asking who contributes to the success of the so-called scientific journals. The answer to that is implicit, for the non-scientists are not likely to want to buy them however wonderfully ‘scientific’ their articles may deem to be. But the fact that The NYT is successful does not make it a journal, let alone a scientific journal. To come to that conclusion would be most unscientific.

    Further to this discussion, original results/data/observations, call them what you will, are not at all essential for investigations to be scientific. Why should they be original? Authentic? Yes. Reproducible? probably, but not necessarily so. Original? No. Original research can be done on data that’s been tried and tested for centuries. As far as that’s concerned the more tried and tested the better. If it takes a century for enough reproducible evidence to be accumulated so be it.

    The scientific method does not define who has to make the observations or even how or when these are to be made necessarily. Neither does it hinge on the observations being reproducible. The July 1994 collisions of the so-called Shoemaker-Levy comet with planet Jupiter cannot be reproduced. That does not mean that the observations made of that event were not scientific. But these observations served to prove that a scientific theory was correct.

    Ka kite
    from Middle-earth

19. Aristotle was never published in peer reviewed journals, but he was still a scientist.

    Ignoring the fact that to call anybody before 1834 a scientist is an anachronism, Aristotle’s scientific work survived and is still readable today because his contemporary supporters and critics, other scientists, recognised his work as being, in their sense, science. Had his work been rejected by the same people as worthless or non scientific it would not have survived. This is the process of peer review (in the widest sense) by the scientific community.

    Saying “something is science because the majority of scientist say it is science” sounds very much like a caste of priests dictating from on high. It does not sound like science.

    Many of Kuhn critics said very much the same thing but it doesn’t stop his claim being fundamentally right. Science is in the last instance controlled by the “caste of scientist”, a fact that unfortunately, falsely interpreted, was a contribution to raise of post-modernism and its claims of the relativity of scientific truth.

    You would almost certainly, correctly, claim that the criteria on which new science is judged are in some sense objective however, who determines which criteria are to be used in that judgement and in what sense are they objective. The answer is quite simple scientist! Who determines, who has the right to determine such criteria? Again the answer is scientists? All evaluation within science of scientific content is carried out by scientists and it is also scientist who determine who has a right to call himself a scientist. A historian would have a hard job correcting the finals exam papers of a student of nuclear physics!

    The scientific community is a closed shop that exercises the right to determine membership of the community and also to determine the content of that which we call science.

20. The questioning of the value of peer review is often perceived as a call to abolish it. Few people are really saying that.

    In my view the main problem is that PR is being used as a surrogate for providing sufficient experimental details to reproduce or debug an experiment. Aside from a glaring lack of self-consistency, reviewers generally start with the assumption that the summarized experimental information provided by authors is correct. They have to because they don’t have access to the lab notebooks and primary evidence.

    This is not a technical problem – we just currently have a scientific publication system that does not require full disclosure.

    Even if reviewers had access to all the raw data they cannot possibly evaluate all of it. But researchers attempting to reproduce a specific experiment do often need it and an open review of that specific detail at that time would make sense.

    With the advent of Web2.0 we have the ability for the first time to easily implement a system supporting the dissemination and evaluation of useful but “unpublishable” information stored in failed experiments. But this will require a paradigm shift of how people think about science.

21. I finally went and looked at the Health Physics Society’s information sheet. I guess it can’t be said to have been through a peer review process, perhaps the HPS press office OK’d it. If a journal were to ask me to referee this, I wouldn’t let it go as it is. The original “experiment” was an unnamed Radon contractor measuring radiation levels in a house using a geiger counter in what HPS says is a non-approved way.

    HPS goes to a granite supply depot, measuring radiation levels next to numerous samples of granite, finding that radiation levels next to these samples was barely elevated. Then comes the fun computation. They compute the level of Radon gas emanation that would cause the radiation level they observed, making a factor of two adjustment along the way for the fact that the granite is not infinitely thick, then, even more fun, they calculate what the radiation level would be on various assumptions about airflow, … to find that the experiment carried out by the contractor could not have given the result it did.

    Meanwhile, the NYT says that “with increasing regularity in recent months, the Environmental Protection Agency has been receiving calls from radon inspectors as well as from concerned homeowners about granite countertops with radiation measurements several times above background levels. “We’ve been hearing from people all over the country concerned about high readings,” said Lou Witt, a program analyst with the agency’s Indoor Environments Division.” In other words, Joe Radon detector’s experiment has been reproduced all over the country, but it looks as if HPS has not attempted to reproduce the experiment themselves at all. We can argue about the medical significance of the raised Radon levels in kitchens because of granite countertops — the NYT article does — but I’m not much impressed by HPS’s experimental and theoretical methodology.

    This is not to say that HPS’s calculations are without any merit, just that they ought to make an attempt to reproduce the original experiment and to reconcile their calculations with them.

22. I’m with Markk on this one. Peer review is a Good Thing but it does not define science. Rather it is one of the current methods we use to distingiush what we want to publish as science.

    Things can be scientific, and they can be good science, without going through the peer review system. However having a peer review system there helps to ensure that rubbish does not get through.

    As far as the media goes, their job is too report things, surely. Journalists shoudl not be ‘doing science’ in the same way they should not be ‘doing police work’ or ‘doing medicine’. Their job is too report on what is happening, and on the work done by other people.

    This is not, by the way, exclusionary. Any more than saying that TV companies carrying out random surgery is exclusionary. If you have the materials, methods and funding available to do science, go for it. If not, don’t. Or rather, don’t pretend it’s science.

23. aaand, I’ve just noticed I made about twenty spelling errors. Which probably makes my point of view invalid *sigh*.

24. Peer review does not imply that a paper is correct. It merely implies that experts in the field (between one and four, depending on journal and circumstances) have looked at the manuscript and believe the results to be (1) significant enough to warrant publication and (2) supported by the underlying data. The system is still vulnerable to honest mistakes and reviewer laziness (more than once I have read an article and reacted along the lines of “how did this get past the referees?”) as well as fraud.

    There is another point here as well … whether or not you want to call it a “mistake” but it may very well be that the data was interpreted incorrectly. Not all of these instances are a result of investigator laziness, or mistakes … sometimes the field needs to advance a bit further, and given those new advances, the data previously collected will be seen in a new light and support the new conclusions.

25. I think it’s important to differentiate between “journal-publishing-peer-review” (i.e how peer review has been defined in the past couple of decades) and the more general idea of peer review. To say that peer-review is not the essence of science, but that the core is “a matter of attitude and procedure” creates confusion because the attitude and procedure includes exposing one’s results to other scientists for validation and confirmation (i.e. “peer review”). This has been the case from Aristotle’s days onwards, while journal-publishing-peer-review is a 20th century invention.

    I agree with Chad’s later comment when he says “the peer evaluation that really matters is not manuscript review, but replication of results”. But replication of results is peer review, in its most fundamental form. Had the NY Times conducted an experiment, described their experiment in detail, and released their data, they would have had a “peer-reviewable” article, although not a journal-publishing-peer-reviewed article.

26. Chad (@ #8): I think we can agree on what you said in reply to me. The issue is getting the facts right, and the point is: there are facts.

    Actually, the NYT article did mention some facts, but these were buried in a much longer presentation of quotes by people who are worried about the radiation. This is how journalism gets done these days (cynic alert): “I’m worried about radiation!” “I think vaccines harmed my child!” The journalists factually present the statements of the parties in question, yet reporting other peoples opinions really doesn’t matter — let’s get the facts, and let’s present the facts.

27. Thony C –

    Science beginning in 1834? 1543 maybe. 1620 sure. But 1834?

28. Science beginning in 1834? 1543 maybe. 1620 sure. But 1834?

    Where and when science begins depends on how you define science I, however, said nothing about the beginnings of science.

    I said, [i]gnoring the fact that to call anybody before 1834 a scientist is an anachronism. The word scientist was coined by William Whewell in 1834! (Wikipadea says 1833!)

    P.S. Camilla you make my point so much better than I do.

29. My view has always been that peer-review is basically a reasonable but flawed quality control filter, and that successfully passing through that filter just grants a finding/argument entry to the highest level of the debate. It does not guarantee that finding/argument is accurate or correct, let alone the final word on the topic.

    But that is a very useful function.

30. “The essence of science is looking at the world and saying “Huh. I wonder why that happens?” And then taking a systematic approach to figuring it out.”

    I would have preferred to read “critical” instead of “systematic”. IMHO, critical thinking is what peer review is about. It is also NOT a majority call or some sort of attempt to guess about the validity of a result in absence of all the data. Even if all the data is available, the ideas, logic, interpretations might be flawed or already known. To detect this, authors are obviously not well placed-due to their direct involvement in the work. This is the main reason why external reviewer are consulted.

31. I think those heuristics are fine if you only want to superficially evaluate the state of a scientific debate.

Comments are closed.