Adversarial phenomena in the publishing system
Posted by: djb
Date: 30 May 2013 08:02
I think that the real problems with our current publishing system come from cases where different parties have conflicting incentives. Some of the comments I heard at the IACR membership meeting yesterday seemed to start from the notion that these parties won't make any attempt to act rationally in their own self-interest; this notion strikes me as naive.
Financial example of conflicting incentives: Publishers make money, primarily by acquiring and exploiting copyright monopolies. Universities would rather spend the same money on students, faculty, etc.
Non-financial example: Authors submit inaccurate papers; the incentive here is the _time_ needed to carefully write and check proofs, software, etc. Readers expect papers to be accurate; same incentive. We blame the author for inaccuracies---the reader is the innocent user; the author is the attacker---because it's obviously much more efficient for the author to do the requisite checking than for the readers to do it.
Another example: Authors submit papers with significant portions that aren't actually new. Here there's (1) a time incentive against checking the literature properly, combined with (2) an incentive to take as much credit as possible. Readers expect papers to be new, except for clearly labeled review to set context; having the same ideas published again and again and again is a huge waste of time. Again it's clear that the reader is the innocent user and the author is the attacker.
Another example, with a new attacker: Reviewers send in inaccurate reviews; again the main incentive is the time required for accuracy. Readers expect that the review process will be accurate, so that acceptance actually conveys some meaningful information (although there are obviously limits to how much information an ``accept'' or ``reject'' can convey to the public). Here reviewers are the attackers, and readers (plus, to some extent, authors) are the innocent users.
Another example, involving collusion between parties: A narrow topic X is published more frequently than justified by its importance to the community as a whole, because reviewers working on X are inflating grades for papers on X. (This isn't necessarily intentional---the incentives are so clear that one would _expect_ this behavior to appear as a consequence of randomness combined with evolutionary pressure. Eliminating, or compensating for, this type of randomness takes real community effort.) Readers expect that the distribution of acceptances across topics reflects the importance of those topics to the community.
Nigel, in explaining two days ago the IACR Board's strawman proposal for a Proceedings of the IACR, emphasized the "high review load" created by papers that are submitted again and again and again "often unmodified". The conflict of incentives is again quite clear. What's completely unclear is how a Proceedings of the IACR is supposed to reduce the review load.
I'm not disputing the ability of a centralized mechanism to rule out the extreme case that Nigel points to, namely resubmission of a paper that's "unmodified". But this trivially recognized case is already quite rare. As a reviewer I often see authors submitting a paper repeatedly without making _the critical changes that I keep asking for_, but this judgment requires expert review. It's essentially never true that the paper is "unmodified"; adding some superficial rule against resubmission of "unmodified" papers will affect only a few cases, and even in those cases the authors _will_ adapt to the rule in the obvious way, so there won't actually be any reduction in the review load.
Of course, there are also many cases where papers _do_ make the critical changes, and cases where reviewers are simply wrong. (Last year I made a serious mistake in a review---I could blame the paper for misleading me, but I could have and should have caught the mistake myself. Fortunately the mistake became clear to me during the committee discussion.) By pointing to "unmodified" papers Nigel seems to be trying to suggest that resubmissions are never worthwhile; this exaggeration seems central to the current design of the Proceedings of the IACR.
I'm not saying that it's impossible to stop misbehavior by authors and by reviewers. A new system can create new incentives that override the existing incentives---consider, for example, the incentives that would be created by exposing submissions and reviews to the public. What I'm saying is that it's important to think through how authors and reviewers will actually behave.