Tuesday, August 01, 2006

Science and Nature

When I was in grad school, we used to have a saying:
Just 'cause a paper's in Science [Nature] doesn't automatically mean it's wrong
In all seriousness, while those two journals have published a lot of groundbreaking work, a disproportionately large fraction of what gets published there turns out to be crap. The reason is that these two journals evaluate papers differently than other journals. Science and Nature put a high premium on newsworthiness, while most other journals do not. As a result, people trying to get their papers into Science or Nature often "sex up" the paper's conclusions a little further than perhaps the data warrants. And the most speculative papers are often the most newsworthy --- but also the most often wrong. A final problem is that the strict length limit means that many details of the analysis must be left out (this is less true now that details can be put into electronic supplements). This dearth of details hinders the identification of problems by peer reviewers. Thus, I would tend to agree with the views of the Capitol Hill staffer described in this Prometheus post:
An interesting detail was that one staffer spoke about a "discounting" of scientific results conditional upon in which journal the result was published. Science is associated with a low level of credibility whereas GRL is considered a generally "good" source.

12 comments:

Hans von Storch said...

In my written statement of my testimony to the US house Rep.'s Committee for Energy and Commerce (vulgo Barton Comm.) I had this paragraph on this issue.

"Another relevant aspect is the functioning of the two prestigious journals “Science” and “Nature”. The journals enjoy high esteem within and outside of the scientific community as having the highest scientific standards, which is not always the case. The contents of Nature and Science also receive exceptional attention in the media world-wide. However, different from “normal” scientific journals, the editorial decision to accept a scientist’s contribution to Science or Nature is also based on the newsworthiness of the research contribution. The presented results must not only be valid and innovative but must also be of interest for a wider community of readers. Such a criterion is reasonable from a economic point-of-view, but it clearly introduces a filter in what is reaching the public is not solely based on the scientific merit of research. Research results with stronger media appeal fare better in this competition of scientific findings; results biased towards higher sensitivity to human interference are more interesting to a broad audience than findings that report low sensitivities. In addition, there may also be a bias towards certain authors, who are well known, because they enjoy public visibility, or command appealing writing skills, “sell” well. Sometimes such contributions are invited."

The full statement is available on my web page or from prometheus.

TCO said...

This was my independent observation from grad school work as well. The best place to publish is full papers in specialty journals. This is why I advised Steve that his Urals misdating paper did not belong in Nature (or Science). It was insufficiently newsworthy, too much hinged on a matter of detail, and as it in the end conveys a methodology caution, it needed to go to people in the field more then people in general. There are a host of dendro journals, climate journals, etc. He could have put it in a fine home.

Anonymous said...

John Fleck says -

I think the news media piece of the Science/Nature problem is huge. We don't read GRL, so we end up biasing our coverage toward the more speculative Nature/Science-type work without recognizing and sharing with readers the speculative nature of what we're covering. I took a crack at this some time ago when the Bryden paper came out:

http://www.inkstain.net/fleck/?p=1239

Andrew Dessler said...

John-

Another factor is that publishing in Science and Nature is seen as prestigious among scientists. Your colleagues are impressed and perhaps a little jealous if you can get your paper into one of those journals. I can't think of a rational reason why that's so, but it certainly is. I think this reinforces the view of journalists that anything published in those journals must be special.

Regards.

Daniel Collins said...

Don't forget the cover letter. If you can't sell your article, no matter how good the research is, or how newsworthy in its own right, a poor cover letter sinks the ship. Bad research has passed this first round, and good research failed. Maybe Nature's current reviewing experiment will improve these things.

Andrew Dessler said...

Daniel-

What's "Nature's current reviewing experiment"?? I have not heard anything about this.

Regards

EliRabett said...

I am shocked, shocked that there are thing in Science and Nature that are sometimes proven wrong.....

Well guys, first you start with the FACT that Science and Nature are about an order of magnitude ahead of everything else in terms of impact (see ISI impact factors).

Second, they publish things that ARE important and new, which is where you are going to find both the most important things and those which are most subject to error.

Third, having appeared in S&N more people are going to follow along the same track and try and repeat and extend the work. The fact is that most of what is published in other journals is just ignored. This alone guarantees a higher correction ratio

Andrew Dessler said...

Eli-

I agree with your points --- those journals do publish some groundbreaking work. However, none of your points really contradict mine, that people "sex up" results to get their paper into S&N, and that too little detail is given in some of the papers.

On that last point, one thing that particularly irritates me is the following scenario: 1) someone publishes a paper in S or N and says that, because of length requirements, they cannot give full details, 2) then they publish a longer paper in a specialty journal and refer to the S or N paper as "showing that our method is fully validated and accepted." This kind of scientific boot-strapping really burns me up.

Regards.

Steve Bloom said...

Also, isn't GRL considered a little cutting edge compared to many journals? If so, that would make the staffer's comparison a little odd.

Steve Bloom said...

Andrew, see James Annan's blog (the next most recent post as of now, I think) regarding the Nature reviewing experiment.

Daniel Collins said...

Other than what's on Nature's website (pretty easy to find), I think it's insightful to remember that it was Nature that compared Wikipedia with Britannica.

Andrew Dessler said...

Steve Bloom-

GRL is (in theory) for rapid communications of broad interest to the community. That's why the papers are mandated to be short. In other words, GRL is supposed to compete directly w/ Science and Nature. So you're right, it is supposed to be cutting edge. In reality, I don't think it's really lived up to its potential for a number of reasons I won't go into here.

As far as Nature's reviewing experiment, they've done a TERRIBLE job publicizing it. I had not heard anything about it. After reading about it, I do think it's interesting, and I hope that something useful results. For those interested, a similar system is successfully working at Atmos. Chem. Phys. (atmos-chem-phys.org).

Regards.