11

From the timeline of this question, it seems the Reopen review which was initiated after my edit became invalidated.

Any idea how this could happen? Three reviewers have voted to reopen, but the question isn't open yet. No other events in the timeline could have interfered with the review somehow. The expected behaviour is that the review continues, until five people vote to reopen, the question is reopened outside the queue, or three people vote 'Leave Closed'.

timeline of the question reopen review queue

  • 1
    I've been seeing a fair few reviews get invalidated for no reason I can determine. May need to investigate more thoroughly. – Nathan Tuggy Dec 5 '16 at 20:52
10

A tale of complexity and unintended consequences

The review task was added in response to this edit which you made.

Which is a good thing, generally. That edits can result in the review of a previously-closed post.

However...

There's a check, in the logic that determines whether an edit qualifies to do this, which excludes edits by folks who've also flagged the post. And by "flagged" I mean not just Very Low Quality flags and spam flags, but also close and reopen votes (which internally are just types of flags). The only flag that doesn't result in an edit being ignored is "in need of moderator intervention". The idea here is to avoid taking the actions of someone who is, say, editing vile language out of a post and turning it into an implicit request to see more of that sort of thing.

And this is still all well and good, except for the bit where it doesn't distinguish between flags raised before the edit and flags raised after the edit. Or indeed, flags raised after the edit had already put the question into review.

So what happened here is, you edited the question, and then came back a bit later after the question had entered review and voted to reopen it. And... The system said, "Oh no, same person flagged and edited, can't count the edit" - and now there was no reason to have a review.

But wait! Don't reopen votes normally trigger a review? So what if the edit is no longer valid; there's three votes! Weeeelll... Turns out the logic that checks for votes that would trigger a review has a time-limit on it that's meant to prevent a question from entering review for at least 15 minutes after the first vote is cast, to give things time to happen organically and such.

...And all of the votes came in within 15 minutes. Within two minutes, actually. So at the point when the edit was no longer valid, the votes weren't yet valid.

Thus, the review task became invalid. And since all of the votes happened after the question was already in review, they didn't count either - otherwise, there'd be no getting questions out of review if they got even a single vote, no matter how many people said "leave closed".

...And therein lies the bug: a conflict between two criteria for allowing a question into review inadvertently caused the question to drop out.

So what should we do about this?

  • Well, we could just ignore flags that happen after an edit. The edit only counts if it happens after the post is already closed, so it'd be a pretty rare occurrence for (for example), someone to edit a question and then decide it not only needed to be closed but also deleted. One might imagine a scenario involving spam or something I guess.

  • We could also just ignore reopen flags when it comes to invalidating edits. This would allow someone to potentially put a question through review twice if they voted to reopen then, after that failed and their vote aged away, edited. In practice, this would require both patience and precise timing - there's only 5 days after a question's creation where edits work to put a question into review, and aging doesn't start for four days after the last vote... So the question'd have to be closed immediately, then you'd have to cast your reopen vote pretty quickly thereafter, the question would have to go through review promptly, get no more reopen votes for four days, and then you'd have a bit less than a day to make your edit and hope the review went your way... 'cause on day 5, the task would once again become invalid. In spite of this obscure loophole, I kinda prefer this option: it's dead-simple.

  • Or we could drop the 15-minute delay, or make it based on the time the question was closed instead of when an edit or vote was cast. This is the worst option, IMHO: generally-speaking, you only get one shot in review, so you want to give folks time to make all necessary edits or finish up any discussions in comments before you fire that off.

  • 3
    Wow. Many thanks for your lengthy explanation. I had no idea that it would be so complicated and initially blamed it on cosmic rays. When such issues happen at our projects, I usually just tell the customer 'some miraculous coincidence of events happened; we took some measures to prevent it the next time', and that's it. I prefer the second option as well, and hope that it will be implemented soon / in 6-8 weeks. – Glorfindel Dec 6 '16 at 7:16
  • 2
    I like the option of ignoring reopen flags for invalidating edits. I don't see a question getting reviewed twice as a huge problem given the typical size of our review queues and the complexity of making it happen on purpose. – ColleenV Dec 6 '16 at 15:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .
https://ell.meta.stackexchange.com/questions/3348/why-was-this-reopen-review-item-invalidated/3351
ell.meta.stackexchange.com
crawl=CC-MAIN-2019-35
116417
application/http; msgtype=response
sha1:BQ33JMCT6LVPR62FQKV53DQSD5ZOBPCC
<urn:uuid:8a702484-2e0f-46c1-80ab-a01eeb5ad358>
2019-08-17T10:32:09Z
151.101.193.69
text/html
sha1:2EYHOIZE7C2ZCVIJOWJJPKWAXZ5YJZFH
<urn:uuid:2eab2099-407e-4c2f-8300-bdf9024cb88f>
https://ell.meta.stackexchange.com/questions/3348/why-was-this-reopen-review-item-invalidated/3351
response
<urn:uuid:439f701e-4892-4fe1-8ec8-cdaa45f381a1>
isPartOf: CC-MAIN-2019-35
publisher: Common Crawl
description: Wide crawl of the web for August 2019
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-159.ec2.internal
software: Apache Nutch 1.15 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.1-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/