💡 Feature Discussion: Reviews

Reviews

We think of new documents like a stem cell. The same piece of content can turn into functionally very different things. The work can turn into a blog post, a research paper, a thesis, a conference proceeding, or a book. The distinctions between these things are the human processes that occur around and to the document. Reviews are a big part of that.

To support the diversity of documents types, we likewise want to support a diversity of review types. We are designing reviews to be a very general item in PubPub. To do so, we identify a few key components:

  1. Review Policy
  2. Reviewers
  3. Review Map

Review Policy

A review policy outlines the plan for a review. It may contain details such as:

  • how many reviewers it will have
  • what constraints will be placed on the review reports (single blind, double blind, etc)
  • what metadata is expected
  • how long it should last
  • etc…

The review policy is something that can made into a template for easy re-use and for simple communication. Often we find review procedures are written in plain language on journal or conference sites. PubPub Review Policies simply provide a structure for codifying that so it can be shared, reused, and programmatically implemented.

Reviewers

Reviewers are the people and machines that provide feedback to a document. In traditional paths, this is a set of 2-5 people with relevant expertise who will provide a written response and perhaps enter some form field information. In addition, we envision machine reviewers that can function as autoformatters, language checkers, spell checkers, grammar checkers. We’re excited about the places for expansion in this path: tools that help you find related but missing citations, tools that encourage best-practices around data-sharing, tools that help push you towards language that is an appropriate level of jargon.

Review Map

A review map is the thing that tells you what actually happened. In an ideal case, it simplify verifies that what was stated in the Review Policy actually happened. In reality though, there are lots of reasons why a Review Policy and a Review Map could differ. The number of reviewers can change, the time taken can change, the expected metadata can change. Review maps are something we hope other publishers will begin to adopt and share, such that a common language of ‘what actually happened’ can be understood.

Our approach

The challenge of building tools to cleanly manage and enable reviews is known to be exceedingly complex and difficult. Many review management tools have existed and grown into tools to be loathed. Many have simply failed because they become too bloated. One difference that we think matters here is a top-down vs bottom-up approach. Most tools that struggle seem to have been building structures that precisely solve a problem for an existing workflow or journal. In trying to then scale and adapt itself into a tool that solves the problem for a wide variety of publishers, it becomes too broad and suffers from conflicting preferences and constraints.

Our approach is to build a set of tools that let the publisher themselves craft their process. We’re not prescriptive in giving different stages names or required follow-on steps. One tradeoff this entails is that the PubPub review system will likely not be setup for your exact workflow out of the box. We’re hoping that in time a number of templates and best practices will emerge to reduce such hurdles - so please share your feedback and ideas. The benefit (we hope) from taking such an approach is that we will be able to scale a review system that remains flexible yet powerful.

We’ll be talking with many of you in the coming months and slowly rolling out Review features. To share a bit of visual thought on the matter, a set of early, unpolished mockups can be seen here.

1 Like

here’s an example of a machine review activity built as a third party extension to HotCRP (an open-source review platform) which attempts to identify undisclosed conflicts of interest by using a domain-specific bibliograpy of computer science authors & publications (DBLP.org)

I see that there is now a review area located at www.journal.url/pub/asdasdfasdf/review/1 if someone has clicked the submit for review button. However, it is not clear to me what I can do with this. I don’t appear to be able to add reviewers, I can only post a comment (not clear who has access to that comment). Thanks.

Hey Devin,

Sorry about the delay getting back to you – we’re still working on implementing the review feature fully. More to come soon!

Gabe
https://discourse.knowledgefutures.org/t/feature-discussion-reviews/166
discourse.knowledgefutures.org
crawl=CC-MAIN-2020-10
20416
application/http; msgtype=response
sha1:QECM73XKI45RRTAJLAO7LQLMLEQQJ6BJ
<urn:uuid:5a94e30d-b369-47fc-931b-2f6dece93ba6>
2020-02-22T06:44:50Z
66.220.12.139
text/html
sha1:GLR6VUCZWTYG3RLCRWEGKSFWULWV2BZG
<urn:uuid:85640538-3651-4d72-8eae-4a7c5e59df2c>
https://discourse.knowledgefutures.org/t/feature-discussion-reviews/166
response
<urn:uuid:568e1875-de1a-4b75-b13f-f88afaae5ac3>
isPartOf: CC-MAIN-2020-10
publisher: Common Crawl
description: Wide crawl of the web for February 2020
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-10.ec2.internal
software: Apache Nutch 1.16 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.1-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/