The following conversation has been condensed and lightly edited for clarity
Q: What gives Science its reputation as a reliable source of accurate information? Why does it have such a high regard among scientists?
A: There are a few factors. Science is a really selective journal. We publish research in all different fields, and our goal within these fields is to publish the best, most exciting, most broadly accessible research out there ― the big steps forward in a field. Science, of course, is built on lots of little steps towards gaining knowledge, and refining methods and different things, but we’re sort of looking for these big steps forward. So the papers that we eventually publish are often those sorts of papers.
By virtue of being an exclusive journal, I think that creates a bit of incentive for people to want to publish in Science and submit their best work to us. So it kind of feeds itself. By the attention and the press and the association with being published in Science, people then submit their best work to us.
We operate a little bit differently than most other journals. We have a staff of full time editors, such as myself, whose job it is to evaluate the potential impact of these papers within their field, within all fields of science, and really try to get the groundbreaking stuff that I was talking about. A lot of other journals that are more specialty journals within a discipline, they have mostly professors or other academics who are acting as the editors, and they’re doing it in their spare time or maybe a day a week or something like that. We are working on that sort of thing full time, and we look at the papers from very different perspectives, with the goal in mind of finding those big advances.
Q: What is your methodology as an editor? Once you receive a research paper, how do you go about evaluating it and making sure that it’s good enough to be published in Science?
A: We’re looking for the general appeal of a paper. If it’s going to be impactful, maybe not just within say environmental science, but also public health or something like that. Then, of course, we’re also interested in the technical aspects of the paper. Is it good science? Did they do the right experiment? Are they over-interpreting their results? These sorts of things.
So the first thing we do when we get a paper is the editors look at it, and we have a staff of 20 or so editors who handle research papers. Because we handle all of the different fields within science, we have to handle multiple fields. Each editor handles a few related areas to one another. We’re all trained scientists, so we’re equipped to kind of gauge, at a first glance anyway, the suitability for the paper in Science. Is it going to be one of these big discovery papers?
After that we send it out to experts in the field, and we have an advisory board that we consult with during this process. But also it’s going through the standard peer review process that all of the journals go through as well. We have a very high bar for getting a paper into Science, not just in terms of its broad appeal but also for its technical achievements, so we ask the experts all of the questions about statistics and reproducibility and the quality of their data, these sorts of things. Then we go back and forth with the authors after that, and maybe the reviewers again, to try to really make the paper as good as possible, or assess whether the paper is going to be good enough to get published.
Q: You mentioned looking for the most exciting, groundbreaking studies. Does publication bias come into play at all? Is there a greater risk of studies being overturned when you’re trying to publish studies that are more in the “groundbreaking” realm of science?
A: Yeah, I think that’s true. It’s a lot of trying to figure out if the risk is going to be worth it. Ultimately we’re trying to push the field forward. There are papers we publish that we don’t know at the time if they’re going to be right or not. We just don’t have the ability to do the experiments yet or we don’t have the data yet or something like that. But the result is compelling enough and based on what we know at the time, it’s as right as we think it can be, we’ll publish it if we think it’s going to be a big step forward. It’s for the scientific community then to go through and test it even more. Eventually a paper may be proven wrong, but I think we’re okay with that if we did all we were supposed to do at the time.
It’s a bit different than if people are trying to manipulate their data to get a paper into science, to make it more exciting or something like that. That’s a bit trickier to spot, and also to deal with. Those often lead to retraction or corrections or things like that.
Another part of that is our papers are also highly visible and they often get more attention so people are more likely to scrutinize them and draw attention to problems that come up in the paper as well.
Q: Do these risks ever come back to bite you and the reputation of the journal? If so, how do you deal with that?
A: Anytime we publish a paper that, say, immediately there starts to be questions about how did Science miss this or how did the reviewers they chose miss the obvious glaring error and what were they thinking, those are the sorts of errors we would want to avoid. And anytime we do publish a paper like that our reputation takes a hit. And that happens. It happens, I don’t know how regularly it happens, but it happens a couple times a year where there’s a paper that we publish that draws enough concern from the onset that we probably should have been able to catch it earlier. And the only thing we can do it at that point is learn from our mistakes and, depending on what the errors are, we can try to move forward to issue a correction or even retract the paper if the results don’t hold up anymore or something like that. But ultimately yeah, we’re trying to avoid things like that.
Q: As a scientist, how do you determine if a journal in your field is a reputable journal? Do you look at impact factor?
A: I would say that the [scientific] community uses impact factor a lot. I think it does have some value in terms of looking at a journal specifically, but not any particular paper within the journal. Even if a paper is published in Science, you know you can look up Science‘s impact factor and say wow this must be a really good paper, but it doesn’t mean anything about a paper. And if you look at the citations of papers that were published in Science they vary widely by field, by time published, I mean there are so many different things that control how many times a paper has been cited, why a paper is being cited, that I don’t think it’s a very good measure for an individual paper.
For a journal it sort of gives you a rough idea of how much attention papers in that journal receive from the [scientific] community. That’s vaguely about all you can do. Of course the biomedical field overall is very large, so they have higher impact factor journals than, say, environmental science does.
But for me personally, I look at who publishes it, what company publishes it or organization or whatever. Then, if I really have never heard of it, I’ll look through their editorial board, look through and see which papers they’re publishing, and if I recognize people who are publishing in the journal. It doesn’t happen that often to people who are in the discipline, that they hear of journals that aren’t reputable. There are a lot of journals out there, and there are more and more, if people haven’t heard of them by the time that they’re experts and working scientists in the field, there’s a good chance that it’s less reputable. You get so familiar with the literature and what people publish, where they publish it, that if you start to see all these new journals pop up and you’re getting asked to write a review for some really obscure journal, warning bells are going off. But I wouldn’t say there’s any real strong metric to use.
Q: Are there usually just a few specialized journals for each field and then the bigger, more general ones?
A: Yeah, probably. I think a lot of fields have a handful or journals that they’re more comfortable with. And then of course I think a lot of people are familiar with Science and Nature and PNAS [Proceedings of the National Academy of Sciences] or whatever. Some people work on very different topics and so they may have to know more journals.
Now too, with everything online, and so many search engines, your research is going to be discovered. Even if somebody does publish in a very obscure, low impact journal, people can find it if it’s related to what they’re working on, and they’ll cite it if it’s good work and if it’s important to cite. I think where you publish is becoming less important now than it ever has been.
Q: Do you think on more controversial topics (genetically modified foods, global warming, vaccination, things like that) statements put out by the AAAS [American Association for the Advancement of Science] Board of Directors and other science institutions, national academies, are a good place to go to for information on those topics?
A: In terms of just a general comment, yeah absolutely. I think they represent the thinking of large organizations of scientists and I think we need to trust the judgment and competency of these organizations to help us make decisions about these controversial topics.