Making Peer Review Pay

Aaron Swartz was a genius and a hero. The prosecution that presumably contributed to his death was an appalling act of overkill, particularly in contrast to the failure to prosecute financial sector crooks. But that doesn’t mean he was always right.

Scientific Peer Review is one of the most important pillars on which our society is based. It’s the difference between going to the doctor and getting medicine that will probably make you better and receiving very expensive lolly water. It’s the difference between having the forewarning that could save us from the fate of many civilisations that messed too much with their local ecosystems and sticking your finger in the air and saying “climate change is bullshit because it’s cold today”.

And peer review doesn’t come cheap. The current mechanism we utilise for operating peer review is not a good one. It involves pretty substantial profits for publishing companies, works good scientists to the bone, and still publishes plenty of rubbish. Still I’m not convinced it is the worst possible scenario, or that breaking it would ensure something better rises in its place.

The idea of open access publications is certainly appealing. As a science journalist I usually can’t access the papers I’m writing about unless the authors email me a copy. That’s not always a problem. Often the papers are so full of jargon I don’t learn much when I do read them, and its almost always the case that interviewing the scientist is more productive. For other researchers in the field however, the situation would be different. With the vast majority of this research being publicly funded there is clearly something wrong with the idea that the public that paid for it can’t access it without paying extraordinary costs to Elsevier and their smaller competitors.

As I understand it Swartz was driven by the quest to take this information out from behind the paywalls and release it to the general public. He may not have expected the ferocity of the legal attack he received as a result, but he knew there were likely to be consequences and did it anyway for what he perceived as the public good. So definitely a hero. I’m in no position to judge whether his actions were even illegal, but at worst they were minor crimes that did not deserve anything like what was thrown at him.

However, it is worth considering the consequences of him succeeding in his larger goals. Publishing no longer requires expensive printing presses, but if there is to be quality control it still requires editors, who need to be paid if they are to devote serious time to it. If not from those who want to access the material, where is this money to come from?

Ideally I’d like to see it come from the government. Compared to the cost of actually doing the research, the cost of publishing is pretty small, so this is certainly affordable. But that does not mean it would be afforded. Go tell the US Republicans, Tony Abbott, or even Wayne Swan that they should suddenly stump up millions (or in the US case tens of millions) of new money to take over the world’s most credible journals, or found new ones. Moreover, I think quite a few people might have concerns about putting government funded bodies so directly in charge of what gets published.

Philanthropy is another possibility, but so far the funding stream for open access journals is not enough to publish more than a tiny fraction of the research conducted each year.

Some open access journals work by having the author pay. There are circumstances where this succeeds. David Lindenmayer thought his paper on landscape traps (discussed here) was so important to public discussion he paid for it to be open access out of his own pocket. In general however, this strikes me as an even worse approach than having the reader pay. Only rich scientists get to publish? The richer you are the more prestigious journal you can publish in? Not good.

Moreover, I think the need for funding is going to grow. I wrote a feature last year for Australasian Science on bad papers getting through peer review. No one seems to have found a way to measure it, but it seems that reviewers are increasingly overworked and this may be letting more seriously wrong papers get published because the reviewer just doesn’t have the time to notice the issues. There is a fair chance that soon we are going to have to start paying reviewers as well as editors if we want to maintain standards. If not through readers, where will this money come from?

The journals are aware of these issues. Some have experimented with an open review process, where draft versions are placed online for people to critique. Some scientists were reluctant to take part for fear of their work being stolen, but the biggest problem was that, while lots of people thought it was a good idea, few were willing to spend the time going through draft papers carefully to find errors. Hardly surprising – they have their own work to do.

So let me be clear, I’m very aware of the current problems with peer review, of which the cost of access is one. But I’m also pretty wary of upending the single most important barrier between us and a new dark age (a very hot dark age) without being sure we not only know how we want to replace it, but that such a replacement will occur.

Advertisements

About Stephen Luntz

I am a science journalist, specialising in Australian and New Zealand research across all fields of science. My book, Forensics, Fossils and Fruitbats: A Field Guide to Australian Scientists is out now through CSIRO Publishing. I am also a professional returning officer for non-government organisations. I'm very politically active, but generally try to restrict this blog to scientific matters.
This entry was posted in Australasian Science, Science policy. Bookmark the permalink.

5 Responses to Making Peer Review Pay

  1. Hmm. I’m not entirely convinced. Yes, every industry based on information has a problem about how to fund itself, but I think academic publishing has it less than most. Editing a learned journal isn’t really that hard: it’s not like a newspaper, you don’t have to employ reporters. People mostly send you stuff; you just need enough competence in the field to filter it a bit before sending to to referees. If universities would just kick in a bit of money – just a fraction of what their libraries currently have to spend on journal subscriptions – academics would be able to do most of it in their spare time. You’d probably have less risk of dodgy journals proliferating than you do under the present system, when publishers have an interest in continually putting out more of the things because they’ve got a captive market.

  2. It’s probably true that academic publishing has less of a problem than some other industries, but its also more important than a lot of other areas. If illegal downloads cause the collapse of TV drama that’s sad but not civilisation threatening. Not so with peer review.

    You’re probably talking to the wrong person when you say editing a learned journal isn’t that hard. I’ve watched my father edit the Torts Law Journal for years (admittedly not peer reviewed) and the workload is phenomenal, even for something that comes out only three times a year.

    When I wrote the article on peer review editors talked about how hard it was to find appropriate people to review articles. One of the big dangers is in sending a paper to someone whose field is not that closely related and consequently is not really in a position to detect flaws in the work. Once you’ve found the right person you then have to lean on them to take it on – everyone agreed good reviewers were getting increasingly overworked and therefore more resistant to accepting papers to review. Then, if the reviewer agrees the work is suitable for publication but suggests substantial changes there can be a level of negotiation with the researchers if they disagree. This week I interviewed a scientist who had had something recommended for deletion by one of the peer reviewers. She thought it was important enough that, after losing the debate with the editor she’d rewritten the paper to get it published and was now trying to publish the contentious bit in a separate paper somewhere else.

    All that said, the much more important question is whether your suggestion of having universities (and presumably other research institutions) pay people to edit journals would work. In theory it would, and it would definitely be a step up on the current situation (although I’m not sure how you conclude it would cost a fraction of what they currently pay – unless that fraction is 9/10. The question is would they? It seems to me a classic free ride problem. Universities pay for journal subscriptions because they have no choice. Ask them to voluntarily pay to have their academics take time off their own work to edit and I’m not sure they’ll come to the party. I was told recently about a project that is seeking funding from universities at a level that would be far less than 1% of what this would cost. Most of the universities in the country have signed on, but one openly told the organisers “we get the benefit whether we pay or not” so they’ve decided to coast on the contributions of the others. Understandably this scared the hell out of the organisers, because they are not sure how long any of the others will pay under the circumstances. (I’d love to name and shame the uni in question, but pretty sure I don’t have permission).

    Ideally government funding to universities should include taking responsibility for academic journals as part of their duties – either a requirement or something that contributes to the amount of funding they receive. But I’m not sure that will be an easy shift to achieve.

  3. Why do I think it would only cost universities a fraction of what they currently pay to run the journals themselves? Well, I think the vast bulk of the subscription cost of journals today is being paid by universities (mostly by their libraries, some by departments). That’s covering 4 things:
    (a) the actual cost of editing
    (b) cost of printing & distribution
    (c) monopoly rents to the publishers
    (d) repeat of (a)-(c) for a bunch of dud journals that nobody needs.
    With online publishing, only (a) is really necessary, and I’m confident it’s a lot less than 9/10 of the total.

  4. You could certainly slash the costs of hard copy journals by going online only. However, in most cases I think that is effectively what has happened. Most of the journals you can access through a university these days can only be accessed online. Maybe they are still printing copies, although probably many aren’t, but the vast bulk of their subscribers are just getting electronic access. So the savings there would be small.

    Cutting out the monopoly rents to the publishers would be nice, but I’d be surprised if in most cases they amount to a very large portion of the cost. Nature and Science and some of the medical journals have a monopoly status based on the credibility they have built up, But they are swamped by thousands of smaller journals that simply can’t make large excess profits or they’d be undercut by new competitors.

    In regard to d) I don’t know how many dud journals the universities subscribe to, but I suspect its actually not many. There are thousands of journals out there that are only relevant to small numbers of researchers, but they are very, very important to those scientists. Look at it this way: I believe the number of working research scientists in the world today is about 3 million. On average they publish about one paper a year, with an average of 4-5 authors, so that is close to a million papers a year. Sure some of this is rubbish, but if you assume most of these people are doing worthwhile work, then its reasonable to think they can produce something that adds to the sum of human knowledge once a year, even if only in a small way.

    It takes a lot of journals to publish close to a million papers a year. Dud journals exist, either designed to make money or to push ideological barrows, but I’m not sure how many universities subscribe to them, and even if they do, it’s still a small proportion of what is produced. For every Energy and Environment there are dozens of Journal of Invertebrate Cytotherapies and Physics of the Dark Universe (to pick the two that came up first when I went to the Elsevier website).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s