Peer review should be more like Hypothes.is than Hypothes.is should be like peer review!

I didn’t even have to watch the whole video about Hypothes.is to know I want to give them some money on Kickstarter. It hits all the right buttons: distributed, open, platform independent, …

But I’m not sure about the direction of its foundational metaphor: peer review for the internet. It suggests that the future reputation and fact checking system of the internet should be more like academic peer review. But that’s the wrong way to look at it. Academic peer review should be more like Hypothes.is.

The avowed purpose of academic peer review is to assure quality. It is supposed to take away some of the difficulties individual readers have in making judgments about the trustworthiness of the presented information. Despite many high profile problems (discussed here and here), it does an ok job at that, as long as one understands its limitations. I am always a little uncomfortable when I see peer reviewedness of material being used to clobber a position, even one I think is off the wall (such as intelligent design). Saying about something: “it has not appeared in a peer reviewed journal” says nothing about its quality. Nor does the opposite. Peer review safeguards the average, run of the mill, plodding scholarship. It does not safeguard against outright fraud or pervasive prejudice. Peer review in the 1800s would have sanctioned anything published about ether – a non-existent substance. Nor does peer review guarantee that great scientific breakthroughs get through. Kurt Godel first presented his theorems to a roomful of polite disinterest. So peer review is relatively bad at the very best and the very worst and mediocre to OK at the middle. Given the millions of units of peer reviewed output a year, this is not a bad thing. But it does not exactly encourage innovation and while it weeds out the worst, it also limits access to a lot of great stuff. Some of its features like anonymity are likewise problematic. As has been discussed in many places (e.g. here), there are good reasons for it being anonymous, but this also leads to a lack of accountability. And it need not be that way.

One aspect of peer review that has been recently recognized is that of labor and control (e.g. here). Organizing peer review is a relatively laborious process. So it helps if there’s an infrastructure in place – such as that provided by an academic journal. The academics mostly provide their labor for free (although some journals still pay – I was recently offered the equivalent of $35 by a Czech journal for a peer review – so I imagine this varies by country and subject area). But whether they pay or not, it is the journals who keep control of the labor and its output. Which leads to “peer reviewed” knowledge from being easily accessible to the public. If an academic had to pay the full price for every paper she is citing in an average work (about 50), she would have to pay about $1,500 just to write the thing. That makes it very difficult for scholars from currency exchange disadvantaged countries to engage with this sort of knowledge. Private individuals are pretty much locked out.

Another disadvantage of peer review locking scholarly output in journals is the perpetuation of artificial knowledge chunking. I still remember my epiphany when I heard the philosopher of education Richard Smith say “Isn’t it curious how academic knowledge seems to fit into articles of about 5,000 words or books of about 50-70,000 words.” [I’m paraphrasing from memory] Of course, it isn’t curious! It’s stupid. These lengths are given by the logistical constraints of publishing as well as the constraints of peer review (Note: Books typically only have editorial review). It makes absolutely no sense in the digital age to perpetuate the 20th century device of the academic paper. The only thing that’s holding it up are the vested interests of the publishing industry and partially the constraints of formal, anonymous peer review. To be fair, many academics would probably argue that 5-7,000 is the natural length of an article – out of sheer academic conservatism. To be even fairer, some journals make space for shorter work and some are even dedicated to it. I love the BBS format of thesis followed by discussion (called Open Peer Commentary).

So why am I excited about Hypothes.is? Definitely not because I think it will make it easier to know whether what I’m reading online is “true”.

Skeptical software tools suggests that Hypothes.is should be “some sort of tool you could point at any piece of information and have it tell you whether it was true or false”

But this is precisely what neither traditional peer review nor Hypothes.is can or should be. Truth and expertise are relative things, and they’re not what peer review is for. Sure some basic fact checking and review of methodology is great but it’s very easy to build towers of nonsense from stones of truth. Peer review is about basic trust. It says what you’re about to read is by someone who’s looked at the literature, formulated a real problem and applied the right methods. It says: “the author is one of us”. It does not say: “what the author is saying is true”. In other words, it is about trust, not truth. But because it is also opaque, we never know if we can trust the reviewers. Wouldn’t it be nice if we could query the peer reviewers of a debunked paper and look at other papers they let through?

Having been at all three points of the academic peer review triangle – commissioned peer review, peer reviewed and been peer reviewed – I don’t have a lot of confidence in it. The anonymity is supposed to be good for the low-ranked, new scholar. And it probably is. (Although what are the chances editors won’t publish a star author no matter what the reviewers say about the paper?) But the quality of the reviewers fluctuates wildly. Many have their own agendas, complain about not being cited, disagree with conclusions instead of premises, don’t really know enough about the methodology, or simply misunderstand what the paper is about. When I’ve been asked to peer review something, I’ve always tried to avoid commenting on the conclusions. Just the premises. But having seen others’ reviews, I know that this is not always the case.

Opening the peer review to a wider population won’t remove the need to make judgments about veracity, it will simply add new ones. Now, we won’t have to just question the work itself but also its reviews and reviewers. That’s not a bad thing. Reviewing the reviewers is important. Which is why I like the “pseudonymous” rather than “anonymous” aspect of Hypothes.is. It is important that we can see which other works a reviewer has commented on and how, we can better judge her assessment of the work we’re looking at. We don’t need to know who she is, just what her track record is.

But I think the pseudonymity is only there for those who think they need it. One only needs to look at book reviews or other polemics in journals to see that many academics have no qualms about savaging their peers’ work in public. I certainly don’t, but I know many who do. It would be nice if there was a way to combine pseudonymity with declared authorship for the purposes of online reputation but it’s not essential. As long as there is a cryptographic signature that can be tracked, this should be enough.

So if Hypothes.is won’t help with the truth all that much, why am I that excited about it? And excited to the tune of $25? It is because I hope it will help free scholars and their output from the confines of journal and book publishing. It is easy to forget that the whole idea of the web was designed for academics to more easily share their ideas. Initially, it proved to be a bit too difficult for the majority but with the advent of blogging software, the sharing of information became within reach of even the most technophobic of scholars. But one thing was still missing. A way of getting credit for online academic labor. I’ve suggested this as the main obstacle to really freeing the academic world from the shackles of traditional modes of doing business.

I’m hopeful (with a fair amount of caution) that Hypothes.is could go even beyond that. There is really no reason why academics (scholars being institutionally supported for knowledge work) should be the only ones who can make reasonable contributions to knowledge. Or at least why they should do it in the traditional ways. Language Log is an example of a blog that has done serious academic work and there’s no reason why Mark Liberman’s Breakfast Experiment (TM) should not be getting the same recognition a formally published paper does. Research Blogging also collates lots of extremely important online work.

The key here is to not just democratize content sharing (the web has done that already) but to democratize and diversify reputation and trustworthiness. And to disrupt the monopoly academia has had on knowledge. The last of the principles of Hypothes.is is “Work with the best” which complements its sixth principle “Merit based”. But what Hypothes.is needs to do is find a way of bootstrapping merit and “being best”. In a sense, it needs to seed the reputation wheel with some of the traditional sources of reputation but with the aim of jetisonning these, as soon as possible (kind of like BitCoin). It also needs to recognize the ease with which reputation systems become corrupted or subverted by centralization.

But should it become a successful “layer” on the internet, I can see it being a real force in the reputation game that could challenge some existing players in this area.

I can for instance envision a complete rethinking of academic journals as sort of collectives of peer reviewers and readers (literally webs of trust). Authors would simply publish their work on the web and submit it to a journal collective who would then elect a few members to “review” the work. The collective could then periodically publish bundles of peer reviewed materials (or links to them) into journal-length or book-length or something-else-length collections. But all of this would be done in public, in a federated, non-exclusive way. Like today, journals could serve as curators of work but in a much more flexible and democratic way. There would also be much less harm in “publishing” the same work in more places.

Also, citations could serve as reviews. Finally, we could more easily say whether a number of citations is negative or positive.

There’s also no reason why comments on blogs or Wikipedia edits could not be peer reviewed or serve as peer review at the same time. Finally, academics could get some proper credit for engaging in online endeavors. But the line between academics and non-academics would be blurred at the same time (see a suggestion here or here). Since belonging to an institution could count for less than it does now, many more people (scholarly hobbyists) could contribute more of what has so far been kept as separate “second-class” knowledge. Assuming we can disturb the vortex of conformism that underlies the still waters of what peer review is today.

But we could do even further. Hypothes.is could run alongside a badge-like system for an Uncollege Undegree. Why couldn’t a new scholar request a peer review of her work by a few reputable sources who could then recommend her work to be acceptable as a PhD or some other qualification. The system is not that different now except it happens behind suspiciously closed doors. Institutions could certainly charge for reviews (it does take work) as long as this would be transparent and the reputation of the university would be dependent on the quality of its reviews. And there’s no reason a PhD would have to look like a PhD does now. It could simply be a collection of peer reviewed material (blog posts, tweets, YouTube videos, reports, books, literature surveys, etc.)

A lot of the stuff that’s written and published is too long or just unnecessary. Hypothes.is could make it possible to get credit for short form contributions (like comments on blogs) which would make the world a better place where a bit less could be a lot more.

The “skeptics” cited above say: “There is a huge amount of misinformation out there. People believe in pseudoscience, the paranormal and more. They make bad decisions based on these beliefs that have very bad consequences. The job of scientific skepticism is to point out the errors in the information underlying these belief systems, and help people learn to find their way away from them.” But this is exactly the kind of one-sidedness, I’m hoping Hypothes.is will help to solve. There is no doubt that many current scientific convictions will seem like faith in the paranormal in the future. Blind faith in the theory of evolution has caused as much damage to humans as homeopathy has. Hopefully, Hypothes.is will be a route to making peer review a two-way street. There won’t be one party that corrects and another one that sits up and takes note. The reviewers will have to be just as accountable as the authors. And every person will have to make careful judgments as to what network of trust to accept. If Hypothes.is is successful, it won’t be any easier to make the actual judgments but it will be easier to gather the material to base the decision on. And it will be easier to generate those materials and share in the work of evaluation. No matter who we are and where we’re from.

I am definitely imagining a utopia, here. Almost certainly, Hypothes.is will be very slow to make an impact and it could take decades before a distributed system could take on the mantle of the current reputation monopolies. But that doesn’t mean it couldn’t happen and it’s worth 25 bucks to help make it happen. At least, I’ll get a cool Hypothes.is T-shirt.