Get Rid of Peer Review and Academic Journals

I've come to the conclusion that we should get rid of peer review and academic journals in their current form entirely.

Peer review as a "quality check" does not work, yet leads people (myself included!) to take unwarranted shortcuts.

Double-Dipping on Reputation

Currently, researchers often take two mental shortcuts when evaluating a study (again, myself included!). Is the study published in a reputable journal? If, so, the result is probably "right" and can be cited. Is the study from a famous author and in a reputable journal? Now I'm even more certain the result is believable, even if I haven't gone through the methods section myself and looked closely at their data. Effectively I'm weighting reputation twice when looking at a study.

Again, we know that this process is suboptimal. Peer review does not hold up to its promise.

An Incomplete Case for the Current System of Peer Review

Naturally, there are arguments for the way the system currently works. Even if peer review is sub-optimal, surely things would be worse if it wasn't there at all! When everyone "just publishes", how do we know what studies we can cite? We can't read every study in detail ourselves, there's not time for that! Also people will take studies out of context and people will just believe wrong things!

Well, realistically that's the case already, isn't it? But taking the argument serious, consider this:

Every time you read "a study has shown", it's exactly because a reporter relied on reputation. "A study has shown" is a reputational shortcut where the journalist relies on the status of the journal and the fact that the study is "peer reviewed" to confer legitimacy on the result. Your run-of-the-mill journalist definitely hasn't checked the methods section of that paper in detail.

If you take away this reputational shortcut, you take away the only indicator that allows shallow journalists to push out these misleading articles.

Necessary Shortcuts

At the same time, it is true that we (as researchers or the public) need some kind of shortcut, at least when doing high-level surveys of a field. If you want to keep up with a field, it's impossible to read every paper in excruciating detail and run the replication code (if that even exists). You'd spend all your time reading and none of your time doing research – and that is with editors and reviewers doing gatekeeping right now.

If we get rid of peer review, we also need to get rid of academic journals as they exist today.

Instead of the academic journals as we know them today, what we need instead is paid Substacks that wade through the torrent of published studies, review them, and discuss what they find. This is what academic journals should be.

Apart from providing a shortcut when considering a paper, the Substack model of academic journals has some other, additional benefits.

First, it incentivizes actual review in that a paid "newsletter" is only going to be successful if it does a good job at reviewing. And since it's an industry publication where actual professionals are the consumers, the dangers of "click-bait reviewers" seems quite low.

Second, through that financial incentive this model also solves one of the biggest injustices the current academic publishing model has: the unseen and unpaid labor of reviewers. Currently the system operates purely on a reciprocal honor model, where acceptance of your paper means you're (more or less) "on the hook" to review other papers – but since acceptances < submissions, you end up reviewing more papers than you publish. Yet reviewers see nothing of the profits generated through their work – not even in the form of reputation, like high citation counts contribute to tenure packages.

Once reviewing is a paid activity, you could live off doing reviews alone and possibly even fund your own research doing so, reading literature you would need to read anyways.

Here Comes the Blockchain

Okay, let's get rid of peer review and journals in their current form, let everyone just publish and start reviewing newsletters to fund their research. But can't we do that today already?

You can use pre-print servers like arXiv, bioRxiv, or SocArXiv, and no one is stopping you from starting a paid substack that reviews the literature! How does that connect to blockchains?

That's true, you can do all of the above right now already. What I think is important to consider though is in what context this would actually work. And, in my mind, blockchains do offer compelling features that make them worth considering – even though there are a bunch of things I haven't worked out yet myself.

Publishing scientific papers on a blockchain has a bunch of benefits that even a diverse ecosystem of journals does not (and often can not) have.

First, censorship resistance. If it's on-chain it's on-chain, and there's no single server, editor or moderator that can be made to remove (or block access to) a study. But what about retractions! Yup, that's a problem that needs to be solved off-chain, but not a critical flaw.

Second, immutable pre-registration. By encrypting the pre-registration and publishing decryption keys with the full study, everyone can check one against the other without the author needing to fear being front-run. If every encrypted pre-registration also includes un-encrypted author IDs, the public ledger of pre-registrations disincentivizes pre-registering a hundred variations of the same study and then publishing the one that's closest. But because there's no gatekeeping on publishing anymore, this tactic is less likely to be used anyways, since it's only really useful in the context of persistent publication bias in the current system.

Third, public and immutable replication material. Any study should publish as much replication data as is possible – if I'd have my way with a complete replication tool-chain like the targets R-package that lets you document every processing step from the raw data to the finished analysis. Sure, there are cases where that's ethically impossible – you can't publish full interview transcripts in qualitative research for example, or geo-location information on survey respondents. The point is that replication data should be included and again impossible to censor or mess with.

These three reasons alone would make me very much in favor of blockchain-based academic publishing (sidenote: I'd call this chain the Maester's Chain), but the tweet that pushed me to write this up has some great others.

What I haven't figured out yet is the incentive structure of such a blockchain. What's the reward for providing the necessary computation and storage? I'm very wary of incentivizing "reputation" of the submitted works in any way – the blockchain itself should not do any kind of "verification" of the results. So if you have any ideas, examples, or links to good writing that covers possible incentive designs, please reach out!

Related Topics