What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 149 Stack Exchange communities.

I recently received this rather amusing message while reviewing a user's first post:

Congratulations!

This was only a test, designed to make sure you were paying attention. This post has already been removed, but thanks for taking time to leave feedback for the author.

  • What is the purpose of this test?
  • How does it work?
  • What happens if I fail?
  • Do test reviews count toward the reviewer badges?
  • Do robo-reviewers dream of electric sheep?

Return to FAQ index

share|improve this question
10  
Is this being done on all sites, including betas? –  Neil Fein Mar 4 '13 at 4:01
9  
@NeilFein currently they are active on Stack Overflow, Super User, Programmers and Server Fault. –  Rory Mar 29 '13 at 12:34
7  
@TimPost, can you confirm whether this is a network wide now, or limited to certain sites. –  MPD Jun 13 '14 at 19:01

1 Answer 1

up vote 176 down vote accepted

What is the purpose of this test?

The test (known as a review audit) is designed to help new reviewers hone their moderation skills, while nudging more experienced users that don't seem to be paying close attention to what they're reviewing. Some people were in the habit of up-voting everything they reviewed without regard to quality; extremely low quality posts with a high score are very problematic in a system where the best information naturally rises to the top, so "audit" tasks are mixed in to make sure folks are paying attention.

This is currently active only on Stack Overflow, Server Fault, Super User, Programmers, Ask Ubuntu, and Mathematics.

How does the audit work?

  • First Posts, Late Answers and Low Quality - The test presents a reviewer with a post that was previously removed by a moderator and known to be of extremely poor quality (flagged as spam, not an answer or very low quality) or high quality (many up-votes, no down-votes, etc).

  • Close Votes and Reopen Votes - The test presents a reviewer with a question which the system has determined should be open or closed based on various criteria (highly-voted with no close votes, quickly closed with no answers, etc).

  • Suggested Edits - The test presents a reviewer with a randomly generated edit utilizing the Markov chain algorithm to randomly insert various words that may appear to be useful at first glance but actually make no sense and are meant to vandalize the post.

If a user fails a review audit, they are presented with a message that they didn't pass and encouraged to pay more attention in the future. If a user passes the test, nothing happens and they continue reviewing. In some review tasks, they are informed that they passed and thanked for their time and feedback.

The result is then recorded for analysis. Only diamond (♦) moderators or Stack Exchange employees can see the data; it is not accessible publicly even to high reputation users.

What happens if I fail?

For most, nothing. You'll be given some guidance on why the post was inappropriate and what you could have done instead, and then allowed to continue reviewing.

STOP

Repeated failures, especially in short succession can lead to your review privileges being suspended for a week or longer, depending on the frequency of the failures. Stack Exchange is continuing to analyze the data to determine the best action to take in these cases.

Do review audits count toward the Reviewer badge?

Passed audits count as a successful review toward the Reviewer badge. A failed audit does not count toward the badge, but does not decrement your progress; it simply adds none.

Do robo-reviewers dream of electric sheep?

Robo-reviewers don't sleep. They wait.

Important Note:

The information contained in this answer is subject to change as the review system is further refined. Changes that impact the mechanics of the test, or consequences of failing it repeatedly will be reflected in this answer as they're made, but not necessarily immediately after they're made.

share|improve this answer
116  
I'd like a "I still disagree, your test if flawed" button for "failed" tests. –  CodesInChaos Nov 30 '12 at 15:33
12  
@CodesInChaos: keep in mind, these test posts have already been deleted - so if you disagree strongly, you should probably flag them for moderator attention / bring them up on Meta / etc. –  Shog9 Nov 30 '12 at 17:33
26  
I got the "failed" message when closing some questions. It presented a real one and I clicked on the "Close" button just to see how many other people had considered it a "closable" question, but it immediately said I failed the test despite not yet submitting a close vote... I quite often open that close dialog up just to see what other people are thinking (in the event I am overlooking why the question is being voted to be closed). –  dreamlax Apr 24 '13 at 17:32
23  
@dreamlax You should not review based on what others are thinking, because that only amplifies existing reviews. The review systems works because multiple people form an independent opinion. If there is (semi-)consensus that probably means the consensus is true and actions can be based on that consensus. However, if that "consensus" is because everyone agrees with the ones before him only the first independant opinion determines the consensus and thus it becomes worthless. –  dtech Apr 27 '13 at 12:15
17  
@dtech: I see your point and fair enough. For me though, I don't blindly follow what everyone else is thinking, I only check to see if perhaps my mind has had a severe lapse of judgement. –  dreamlax Apr 28 '13 at 3:02
14  
Can users ever graduate out of having to take these tests? I've done hundreds of reviews, and have passed every test thrown at me. How long before the system figures out that I can recognize garbage? Maybe when I earn the Steward badge? –  Barmar Jul 3 '13 at 19:14
14  
I suppose. It just feels a little condescending after a while. –  Barmar Jul 3 '13 at 19:22
26  
I just failed my first audit by clicking "no action needed", on an answer that had already been deleted, which I knew by clicking through to the full question. Because my understanding of reviewing was to recommend future action (or to take action myself), I concluded that the problem had already been resolved, so no further action was needed. Have I misunderstood the purpose of reviewing, such that I shouldn't be looking at the live post to see the current state of affairs in its larger context? Is my misunderstanding of a different flavour? Advice/clarification would be appreciated. –  nmjk Jul 8 '13 at 19:52
8  
Is it theoretically possible for the random algorithm to generate helpful edit suggestions? And, if that happens to me, can that be taken as conclusive proof that this is the matrix, and I am the Chosen One? –  root Aug 15 '13 at 16:47
10  
I was thinking about posting a new question, but hopefully a comment here will suffice. My problem with this is that I sometimes want to view some close reasons, which I can do before closing, except when these tests occur. I wanted to check if something was a duplicate by opening the close dialog, but was then "busted". –  keyser Aug 16 '13 at 14:11
3  
@Keyser Don't worry about occasionally hitting them as long as you're actually paying attention. Nothing bad will happen if you do that once in a while, it's sort of expected. –  Tim Post Aug 16 '13 at 14:27
3  
Pffiew, happy to have read this. (I failed an audit because I was curious how much close votes an obvious good question got.) –  NGLN Nov 22 '13 at 21:50
10  
@dtech Your point, that reviews should not be based on what others think, is correct. However, consider this case: I notice a question has had close votes, but see nothing wrong with it. Before I comment "this post is fine", I'd like to see why it was close-voted in the first place. For instance, it's considered a duplicate, and maybe it is? So I take a look at the linked post to double-check. It is only then that I can decide whether the post truly was a duplicate, or if people may have misjudged the post; maybe they only look similar, but actually ask 2 different things? –  Nolonar Jan 29 '14 at 23:28
13  
I would very much like an "I respectfully disagree" button added to the failure dialog. I suppose it may be a character flaw, but if someone solicits my opinion (as SO does by asking me to review things), and then tells me I'm wrong, I expect a chance to argue my position. It just seems polite. –  MrTheWalrus Apr 3 '14 at 23:25
6  
I don't get these at all; if the point isn't to go along with the herd, then what's the point of the test? If 5 people said a question is poor and should be closed, but I don't think it should be, then there's a disconnect. Either you are encouraging groupthink or you encourage people to make their own decisions, but the fact there's some hidden algorithm that determines if you reviewed "correctly" defeats the purpose of thinking independently. –  Wayne M Jul 18 '14 at 19:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .
http://meta.stackexchange.com/questions/157121/what-are-review-tests-audits-and-how-do-they-work
meta.stackexchange.com
crawl=CC-MAIN-2015-32
91859
application/http; msgtype=response
sha1:46ZVWJHQ2NRJVEBLO52D3KCVTQLUFLQI
<urn:uuid:0e8a2f00-fb86-45e9-8676-96bbd2be22d5>
2015-08-04T09:47:00Z
104.16.12.128
sha1:Y2BUXFHMM4LDJN2DQQQEMWW22KC3XPKC
<urn:uuid:212cd3db-a7ca-4fc4-a30e-1f0b4ae633ce>
http://meta.stackexchange.com/questions/157121/what-are-review-tests-audits-and-how-do-they-work
length
response
<urn:uuid:ce63a1fa-e3e5-413d-8197-9f445d0ed134>
robots: classic
hostname: ip-10-236-191-2.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2015-32
operator: CommonCrawl Admin
description: Wide crawl of the web for August 2015
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf