187

Warning: this is long, rambling and extremely boring. I'm writing it because I tend to get a lot of questions regarding the rationale for changes to the moderator tooling on SO, and I'm hoping to have something to point to next time. If you already know all there is to know about this - or just don't care - then go mine some Unicoins instead.

Moderating a site with as many users and as much daily traffic as Stack Overflow is... Challenging. Just ask any of the hardy souls who've volunteered to do it. Conventional wisdom holds that when a community grows beyond a certain size, self-moderation becomes effectively impossible: either the lion's share of moderation must be performed by a group of dedicated moderators, the community itself fragments into smaller, more cohesive groups, or the group decides to close the doors and stop accepting new members entirely outside of some controlled approval process. Clay Shirky wrote about this years ago:

And, finally, you have to find a way to spare the group from scale. Scale alone kills conversations, because conversations require dense two-way conversations. In conversational contexts, Metcalfe's law is a drag. The fact that the amount of two-way connections you have to support goes up with the square of the users means that the density of conversation falls off very fast as the system scales even a little bit. You have to have some way to let users hang onto the less is more pattern, in order to keep associated with one another.

By rights, Stack Overflow should have died already, turned into an irredeemable cesspool by a combination of outsider influx and insider burnout. You can argue (and many do) that we're headed that way - but we've been headed that way since day one. The best we can hope for is a stable orbit, forever falling but never crashing. I believe there are two major reasons why Stack Overflow has managed to scale far beyond the expected limits of a group:

  1. Conversations not required. When a question is asked on a traditional forum, answering it often demands some amount of participation from at least a portion of the community. Details are fleshed out, the problem is clarified, solutions are proposed and debated, others with similar problems chime in with their experiences, tangential points are made, and eventually - anywhere from hours to months later - the conversation dies out. It's a very social, very natural way to interact. And it suffers mightily from the problem that Shirky talked about: all that back-and-forth and associated latency kills any hope of scale. On Stack Overflow, we close or delete questions that can't be answered straight away - it's not very sociable, but it scales wonderfully by effectively enabling a vast, human-powered computational grid.

  2. Tools that allow decoupling moderation from communication without separating moderators and users. While Stack Overflow does have a powerful "moderator class" elected by the community, a fairly large portion of the actual moderation is performed by individual members of the site, those who've participated enough to demonstrate sufficient familiarity with the community. While this has been a fundamental part of the system for a very long time, I didn't fully appreciate how it relates to scale until I started working with very small Stack Exchange sites: the proportional cost of moderation is much higher, even though the total volume of work is lower. Many hands make (relatively) light work... As long as the system puts tools in those hands.

Number 1 has been discussed endlessly here and elsewhere; if you're a Meta regular, you're more than familiar with how it works. Number 2 is still something that seems to prompt a lot of questions and confusion though, so that's what I'm here to talk about today.

By the community, of the community

Stack Overflow - and by extension, Stack Exchange - was founded with a very simple principle:

Stack Overflow is you.

This is the scary part, the great leap of faith that Stack Overflow is predicated on: trusting your fellow programmers. The programmers who choose to participate in Stack Overflow are the “secret sauce” that makes it work. You are the reason I continue to believe in developer community as the greatest source of learning and growth. You are the reason I continue to get so many positive emails and testimonials about Stack Overflow. I can’t take credit for that. But you can.

The essential nature of the software here is that of a force multiplier, enabling the group of skilled individuals to work together to tackle problems larger than they would be able to without it. It strives to afford and encourage uses which align with our goals (creating a knowledge base of questions and answers) while discouraging those that do not (extended discussion, socialization). But ultimately, it is up to those using it - YOU - to decide what it is used for.

This applies to moderation as well to the actual creation of content. Unlike many similar systems, Stack Exchange moderation does not depend entirely or even primarily on a small group of official moderators. Rather, participation within the system itself grants you the ability to moderate it:

We designed the Stack Exchange network engine to be mostly self-regulating, in that we amortize the overall moderation cost of the system across thousands of teeny-tiny slices of effort contributed by regular, everyday users.

That was a brilliant concept, but after a couple years it was clear that the implementation was flawed: debris were accumulating. Not-answers, blatantly off-topic questions, unintelligible posts of every kind... Small bits of decay, insignificant alone but harder and harder to ignore as they piled up. Virtual mounds of rotting, festering excrement, clogging search results and discouraging participants.

Streets of NYC before sanitation reform
(Source: Harper's Weekly via Collector's Weekly)

User-moderators turned supplicants: the creation of a flagging culture

Something interesting happened back at the start of 2011:

Another change we’ve instituted, based on the popular Newgrounds flash game portal, is the concept of “flag reliability”. If a particular user keeps moderator flagging for reasons that we consider invalid, their flag weight decreases. And for those users who continually flag reliably, their flag weight increases.

This change had a dramatic effect on how Stack Overflow was moderated. Prior to this change, flagging was mostly used for very serious situations: spam, blatantly-offensive content, and whatever problems a strongly-motivated individual might find worth typing up a note about (perhaps in verse). After the change, folks would flag for any problem they couldn't easily solve themselves. Motivated by a tantalizingly-visible "flag weight" score, many were even motivated to seek out problems, just so they could flag them. And the amount of work asked of elected moderators skyrocketed:

moderator-handled flags by month
Moderator-handled flags by month

Some of the moderators you elected rose to the challenge, using their experience and quick-thinking to handily evaluate and dispatch hundreds of posts a day, but the strain was immense: 2011 was the first - and to date, only - year in which we've run two moderator elections, purely to handle the increase in flags.

These moderators deserve your utmost respect and gratitude, not just for the work they did but also for the abuse they put up with. For though they acted as proxies for vast numbers of people, they shouldered all the blame for mistakes and oversight and inconsistency and lack of friendly hand-holding. This small band was - and to a large extent still is - tasked with protecting the community against the barbarian hordes, at any cost: rather a few borderline posts be deleted than the site be overwhelmed with garbage. With hundreds of flags every day, hand-holding and friendly close messages ain't gonna cut it; you need a tool that lets you dispatch crap quickly and with a minimal amount of fuss:

the final solution

These flaggers were invaluable in helping to point out problems, but next to worthless when it came to solving them. Indeed, the system rather perversely motivated them not to solve problems they came across: why fix a post or even down-vote it, when the only thing that would boost your Flag Score was to leave it ugly and flag it? Like the paranoid shut-in who calls 9-1-1 for everything from raging house fires to someone dropping a gum wrapper, these zealous souls solved one problem but created another.

It was crucial that we find a way to distribute the load more evenly.

Tiered support

When Sam implemented the first review system, the goal was simple: deputize members of the site and put this posse to work mentoring new users, guiding them in a more positive, constructive direction. Around the same time, we switched up the purpose of the flag queue shown to 10K users: instead of the (relatively-rare, often-misused) spam and offensive flags, it would be fed with the plethora of Not an Answer and Very Low Quality flags, and they would be withheld from the moderator queue for a day to give trusted members of the site a chance to handle them on their own.

These two changes helped... A little:

community-handled flags by month and type
Community-handled flags by month

The number of community-handled flags doubled, putting a nice dent in the moderator workload, which remained relatively stable... Until the next summer. By this time, it was becomming clear that these tools - as nice as they were - weren't actually doing very much to distribute the burden: the number of users willing to step up to be 2nd-level support was simply insufficient. It didn't help that for the vast majority of users on the site the tools were either inaccessible or toothless: only 10K users could say, "this flag is unwarranted" - and even when they did so, the flag still went into the moderator queue.

So we started handing out work to everyone who could do it, and gave them tools that actually did something.

Revamping review

Let's face it: this was a huge gamble. Suddenly, a bunch of folks who had barely ever used the moderation privileges they'd earned were invited - encouraged - to jump in and help out. We did our best to provide guidance, both in the instructions presented to folks opening each queue for the first time, and in the limited selection of tools available for each task... But no one knew for sure what would happen once it went live. When we turned on the first queues, each had a backlog of tens of thousands of tasks - we added a few at a time, expecting it to take months or years to bring them under control... But with one notable exception, it took weeks. Accuracy wasn't bad either. Unfortunately, it quickly got worse... And then it got a lot worse.

We spent the next year working out kinks in this system. We didn't want to make it too easy to delete things, but at the same time throwing out legitimate flags and leaving trash around is just replacing one problem with... Well, the original problem.

Lessons learned:

  • There's a "default" action in every queue that folks tend to gravitate to when in doubt. Make sure it's relatively harmless or requires multiple reviews to take effect.
  • Audits are critical as a teaching tool. Folks need feedback on how they're doing, especially right away.
  • Some folks see this as just another game to "win" and will do so at any cost. Many of these d00ds are also "cheating" elsewhere on the site. The system should be able to tolerate a few of these, but kick them out before they break it for everyone else.

After a number of tweaks to the behavior of the system (and a fair bit of helpful feedback from the moderator team), the false-negative rate for reviews is back down to a reasonable level. Which finally sets the stage for some actual progress...

Where we're going with all this

Last July, we finally rolled Close and Very Low Quality flags into the queues - you can see the effects on both moderator-handled flags and community-handled flags in the charts above. The next step is adding Not an Answer flags, which are pretty fast to process but make up for that in sheer volume. The Low Quality queue has been rigged such that flagged posts always require multiple reviews, and tasks completed without a clear consensus on what should be done are then forwarded to the moderator queue.

This should bring the moderator queue back to mostly exceptional situations, which is where we started: moderators as exception-handlers. Because you didn't elect them to handle the easy problems, right?

Of course, the devil's in the details: a lot of "other" flags aren't all that exceptional - they just represent situations we haven't developed tools to handle yet. The same could be said of probably 90% of comment flags - they're not really a good use of moderators' (or anyone's) time, we just haven't been able to figure out how they should be handled automatically. And for those truly exceptional situations, the UI for flag-handling leaves a lot to be desired - it could use some serious revamping with a "more meat, less busywork" design focus. To say nothing of the many, many pages of outstanding bugs and valuable suggestions for /review that would no doubt improve the ability of the greater community to find and lend a hand in areas that interest them.

Escape velocity, one step at a time...

14
  • 2
    I completely agree with the comment flag statement. Perhaps they should be rolled into the Very Low Quality queue, but with maybe only 3 votes in the queue to come to a resolution? – Blue Ice Apr 1 '14 at 4:58
  • 9
    @Blue, let's say I come across a huge back-and-forth drawn-out discussion in the comments. So I flag each and every comment: "This answer is incorrect because it tries to foo the bar." and then "Fooing the bar is the appropriate action when you're dealing with baz" followed by "baz isn't relevant in this case because of qux" we end up with 3 totally separate reviews in the queue that are impossible to judge without reviewing a bunch of other stuff and actually reading the question. That is the difficulty with comments. They are plentiful, connected, and require grokking of context. – jmac Apr 1 '14 at 5:54
  • 1
    @jmac Okay, sounds good. Thank you! – Blue Ice Apr 1 '14 at 6:09
  • 2
    The difference between Stack Overflow and Newgrounds is that the very nature of Newgrounds content means there's very little that needs to be whistleblown, which makes it inordinately difficult to attain a deity whistle. (Of course, there's still enough idiots posting questionable content that a number of people have managed to statwhore their way to deity whistles anyway, but it isn't nearly as widespread a problem as on SO I imagine.) I even ended up with a garbage whistle for a while, though I've since been returned to having a normal whistle. – BoltClock Apr 1 '14 at 7:55
  • Oh and whistle levels aren't numbers. They are levels. You don't see results until your whistle levels up, which can take a while. That might be related too. – BoltClock Apr 1 '14 at 8:03
  • 1
    You actually expect me to read all of this? Oh my.. – Seth Apr 2 '14 at 2:06
  • 2
    Oh I just read the header... Yeah, I'm lazy ;P – Seth Apr 2 '14 at 2:07
  • 1
    Over the last few weeks I have noticed that I'm burning through my 20 review items of this type for low-quality posts in minutes, are you suggesting a new queue or would comments be added to this queue. Maybe I have miss-understood but we risk making this queue unmanageable no? – Ashley Medway Apr 9 '14 at 7:37
  • 1
    The Low Quality queue is saturated, @Ashley - if we started directing comments into /review, it'd be in a different queue, with a separate cap. Not convinced that's a workable plan though. – Shog9 Apr 9 '14 at 7:59
  • @Shog9 To be honest I like the idea of a separate queue, my only issue with the queues at the moment is I hit the limits to quickly and can't help more. I think for the most part they help :) – Ashley Medway Apr 9 '14 at 8:11
  • 7
    Am I the only one who went through all of that? – Anonymous Pi Apr 22 '14 at 15:34
  • 35
    @AnonymousPi: No. – Jonathan Leffler Apr 24 '14 at 13:54
  • 1
    I especially like the Digression on Genetic Algorithms – Félix Gagnon-Grenier Mar 16 '15 at 4:25
  • 1
    Hey, kewl graphs! Can they be updated for the last year? – bjb568 Nov 16 '15 at 16:53
69

Let me say something first: since I became a moderator, one of the things that's continually amazed me about Stack Exchange is the amount of effort they put into tools that only a few dozen people will ever know about. These tools allow a relatively tiny number of people to handle the issues of sites used by millions. Over the short time I've had access to them, they've evolved rapidly in response to how moderators and the community actually behaved. The developers deserve our thanks for working on things that, if they work right, most of us won't even know were there.

Looking at the big picture, I see elected moderators having a few key tasks that only they can handle. These are:

  • Cleaning up instances of spam and identifying and destroying spam networks
  • Mitigating the disruption caused by trolls
  • Enforcing the honesty of the voting system by handling sock puppets, voting rings, and serial downvoting
  • Educating people exhibiting patterns of bad behavior (rudeness, plagiarism, rage-quitting, abusing the review or edit systems), and dealing with those who will not reform

As a grand goal, I think that the tooling should continue to evolve to assist elected moderators in carrying out those tasks, and to reduce our involvement in everything else. It's the only way that a small group of elected volunteers will be able to continue to support a site the size of Stack Overflow (and hopefully many of the other sites as they grow).

One example where I think this has worked well is in the offloading of close vote flags from moderators to the community. Despite all the wailing and gnashing of teeth over the size of the close vote review queue, things were worse before that, but no one saw it. Close votes languished on terrible questions for lack of visibility, and people turned to flagging moderators to short-circuit this process. Bad questions weren't getting dealt with, and the moderator flag queue piled up flags that had a very low degree of urgency. The close vote review queue, even while people were raging about it on Meta, led to a significant increase in the number of questions closed per day while eliminating that class of flags for moderators to handle. The recent change to focus the review queue on questions requiring the most urgent attention first has improved things further.

Personally, I hated handling close vote flags because I always felt like I was being used as a proxy by someone to override the community with my binding vote, and because of the tremendous amount of time it took to deal with the hundreds of these we got every day. It's also a lot easier to claim the site is run by a bunch of jackbooted thugs when you see a question closed by a single person than by a vote of five average community members.

I think it's worth investing the effort to think of ways to do the same and offload other classes of flags that don't serve the above-mentioned moderator-only tasks.

Comments are an easy target. While we've always treated comments as second-class citizens, I think it's time to improve the way we moderate these. Is it really all that urgent for us to review the dozens of "Thanks, that answered my question" comments that get flagged via searches every day? On the flip side, I just cleaned up 180 spam comments left by a single user which sat on the site since December before they were flagged. Extremely abusive comments are being removed before we can get to them and see the troubling pattern of behavior from certain users. We're spending a lot of time chasing the trivial, while bigger problems go unnoticed.

"Other" flags are being used to ask us to migrate things to other sites, reopen duplicates that people claim aren't really duplicates, and other tasks that aren't quite covered by standard close flags. These still seem like tasks that could somehow be handled by the community.

The flag bans need to kick in earlier for people on their second or third declined "plz answer thz now" flag, or even better, find a way to detect and stop that earlier through keyword matching and flag warnings.

By removing the flags we shouldn't be worrying about, we can devote more time to larger-scale problems. Tool improvements could increase the signal-to-noise ratio in tracking down sock puppets and voting rings, and give us early warning of their behavior. We can have better ways of identifying when reviewers are approving spam or vandalism, and letting us know when to step in. Somehow, the identification of plagiarism can be improved, along with the workflow around that. We can better identify networks of recurring spammers and trolls, and take care of them before anyone notices. We can better notice patterns of rudeness or hostility and try to prevent that.

If you've stuck with me this long, these are my broader thoughts on how I'd like to see moderation evolve on the site. As Shog9's history lesson shows, it's come a long way already, and this site has scaled far beyond the point where every other mailing list, newsgroup, or forum I've participated in broke down. I believe that leaning even harder into letting the community moderate itself is the key to keeping the site healthy.

8
  • 52
    "the amount of effort they put into tools that only a few dozen people will ever know about" The solution therefore is to tell everybody else about them. For example, did you know that we – BoltClock Apr 3 '14 at 1:35
  • 1
    How did you clean up the 180 comments? Did you have to delete each one by hand or were you able to zap the user and therefore all their posted activity automagically got zapped too? Just curious. – Simon O'Hanlon Jun 2 '14 at 11:49
  • 1
    @SimonO'Hanlon - Delete all the comments by hand. It took a little while. Deleting the user wouldn't remove the comments, unfortunately, only anonymize them. That would actually make things even worse, since at least I could track those when the user existed. – Brad Larson Jun 2 '14 at 14:17
  • 5
    @BradLarson I applaud your dedication to the site and keeping it relevant to folks like me who use it daily. Perhaps a suggestion for a mod-tool then? To auto-delete every contribution ever made by spam-a-lots If a user has had warnings/or a scroll through their history shows 180+ spam comments, it should be easier to delete & have all posts/comments/chat entries deleted as part of the process. Of course that has to be balanced against the possibility of losing a valuable contribution by erasing (but that could be protected against, e.g. keeping accepted answers & comments with 1+ upvotes). – Simon O'Hanlon Jun 2 '14 at 14:24
  • 1
    Maybe moderators should be able to cast non-binding close votes. – tbodt Aug 1 '14 at 0:24
  • @BoltClock I thought you were going to say give everyone mod powers when I first read that. :D – Anonymous Penguin Aug 1 '14 at 16:48
  • 1
    It almost seems to me like the mods simply should be given a help screen and an SQL console, with a process that enforces a preview (dry-run) of the query before it's accepted and made visible to everyone. Perhaps with an alias mechanism where you can give each parametrized query an alias and can say zap-comments-by foo to mean run "DELETE FROM comments WHERE USER=$1". It almost seems that the "nice" UI is counterproductive for anything but browsing the flags. – Kuba hasn't forgotten Monica Sep 3 '14 at 21:05
  • 3
    I may or may not have once tried using jQuery to delete a long string of comments. – Nathan Osman May 12 '16 at 6:18
45

This sentence struck a nerve:

The same could be said of probably 90% of comment flags - they're not really a good use of moderators' (or anyone's) time, we just haven't been able to figure out how they should be handled automatically.

To get an idea of how much time was spent handling various types of flag, I tabulated how many flags were raised on each type of content and each moderator decision:

time   N       flag_result content_flagged 
------ ------- ----------- --------------- 
130.63 4164687 Helpful     question        
  7.01 1031461 Helpful     answer          
 11.44  194600 Helpful     comment         
 24.91   44123 Declined    question        
   9.4   43952 Declined    comment         
  4.45   31877 Declined    answer          

The time column requires some explanation. I set the average handling time for all flags ever handled on Stack Overflow = 100. So helpful question flags take ~30% longer to handle than other flags. It's important to not imagine this is the time a moderator actually spends making the decision. Rather, it includes the time is spent in the flag queue plus the time to ponder and act on the flag. Note that some flags are handled by 10k users; I didn't filter out those cases. The flag queue is substantially more complicated than a FIFO, so the time metric is a very rough estimate of how costly flags are to process.

On to the analysis: there are 4 times as many helpful question flags as helpful answer flags and 5 times as many of those as helpful comment flags. And yet, there are nearly as many declined comment flags as declined question flags and more comment flags are declined than answer flags. To put it in perspective, 18% of comment flags were a clear waste of moderator time compared to 3% of answer flags and 1% of question flags.

More troubling, it takes nearly as long to decline a comment flag as to clear one as helpful. This might seem strange if you haven't moderated comments before. But as jmac noted:

That is the difficulty with comments. They are plentiful, connected, and require grokking of context.

When evaluating an flag on a post, the text of the post itself is often sufficient to make a judgement. Occasionally, you'll want to look at answers to flagged questions or the question of a flagged answer just to be sure. But evaluating comments requires looking at the comment in context of other comments, the question, all the answers, and perhaps some of the other comments the user made in the past. As a result, moderators often prefer to purge an entire comment thread than to meticulously sort the good from the bad.

When you finish evaluating a flag, the actions you can take to fix the problem vary widely by content. For questions, you might close/open, edit, comment, delete, protect, lock, etc. Answers are limited to edits, deletion, comments, post notices, locks, and voting. Realistically, comments present a binary choice: delete or leave be. It's like replacing this toolbox, with this one. Even in the best case, our moderators' talents are wasted on comment flags.

Let's divide the comment flags by type:

time  N     flag_type                             flag_result length score weight
----- ----- ------------------------------------- ----------- ------ ----- ------
 8.25 67664 Comment Not Constructive Or Off Topic Helpful      92    1.14   7.1 
 5    22939 Comment Not Constructive Or Off Topic Declined    127    4.17  12.5
 7.65 44558 Comment Obsolete                      Helpful     109    0.66   7.3
 4.91  2394 Comment Obsolete                      Declined    127    2.79  10.5
27.8  42361 Comment Rude Or Offensive             Helpful     118    0.77   7.9
24.38 10609 Comment Rude Or Offensive             Declined    147    2.52  11.8
 2.4  28094 Comment Too Chatty                    Helpful      67    1.14   5.5
 3.02  3285 Comment Too Chatty                    Declined    121    6.32  14.1
 6.86 11923 Comment Other                         Helpful     129    0.67   8.6
 3.87  4725 Comment Other                         Declined    144    2.23  11.6

length and score refer to the average length and upvotes of the flagged comment. weight is length/15 + score, which is loosely based on hiding scheme. I included those because in every case the comments with declined flags (presumably because the moderator thought they were worth keeping) are longer and have a higher score than the helpfully flagged comments (which were likely deleted).

The second thing to notice is that there's a pronounced difference between flag types in terms of declined rate:

25%  Comment Not Constructive Or Off Topic
 5%  Comment Obsolete
20%  Comment Rude Or Offensive
10%  Comment Too Chatty
28%  Comment Other

The accuracy of obsolete comments flagging is still worse than for questions and answers, but it's at least in the right county. There seems to be wide disagreement between flaggers and ♦ moderators about what counts as "Not Constructive Or Off Topic". I was surprised that "Rude Or Offensive" flags were so often declined. Nor do I understand why they take so much longer to process than other comment flags.

Summary

Our moderators do a first-class job of handling flag about our second-class citizens. But it's frustrating that their time is so often wasted looking into trivia. The mechanics of handling comment flags are about as efficient as possible, but evaluating comment flags is unlikely to scale. Therefore, we need to either find ways to automate comment moderation or open it up to the masses.

10
  • 10
    For context on the high decline rates for "rude and offensive" and "other" comment flags, for the former we see a lot of people using "rude" flags on any comments that disagree with their answers or comments, even if those comments were civil and factual. For "other" flags, we often see people flagging comments as "wrong", "false", or somehow doubting the technical correctness of a comment. It's not our place to singlehandedly delete comments based on someone's assertion of correctness, so I decline all of those I see. – Brad Larson Apr 2 '14 at 1:21
  • 25
    I think that tossing the comments in a queue of any sort is going to end miserably. I think the only long-term scalable way to deal with comments is to give users of a certain rep the ability to vote on them in line when reading the question. Make it as simple as clicking an 'x' below the flag icon or the like. After X votes, it is hidden from normal users, and greyed out for users above a certain rep, and over Y votes it just gets deleted. Flags could then be limited to spam/offensive, saving headaches for the mods too. – jmac Apr 2 '14 at 3:33
  • 2
    It looks to me like a table of Helpful with a weight > 10 and Declined with a weight < 9 would be worth seeing. Given the weight column shown, it looks like a good predictor of the result of a flag. – Mark Hurd Apr 4 '14 at 0:46
  • 4
    Jon, how about replacing the flag button with a downvote button? That's close to what @jmac is suggesting. It could be worth doing an A/B test on... – Benjol Apr 9 '14 at 8:42
  • 3
    IMO mods spending too much time on Offensive comment flags is a mod issue. Comments are not meant to be fragile pearls so carefully considered. If you have a good point, but name-call, too bad; delete. – Matthew Read Apr 23 '14 at 15:59
  • 2
    @MatthewRead: I think the issue is that if you present what appears to be a civil site devoid of crappy comments, then people will want to visit more often. If there is a lot of arguing and name-calling, then it scares people way. I call it the Janitor Principle. If a janitor picks up small gum wrappers off the floor all the time, people may not notice. But the moment he stops picking up the minor gum wrappers, eventually they build up. He may still clean the bathrooms and do the major jobs, but the minor problems build up into big problems. – Brian May 6 '14 at 12:07
  • 9
    @staticx: Ideally everyone would be enabled to clean up gum wrappers. – Jon Ericson May 6 '14 at 16:35
  • Useful data but the 'time' column is utterly misleading, it is not an estimate of effort, more '(wallclock_)time_to_process', but even that is merely a reflection of the differential backlog across different queues (and ratio of users working on them), not anything vaguely related to effort. (I raised a plagiarism flag over a month ago, it will take the moderator like 10 seconds to process, but they haven't gotten around to it yet. so for that 'time' should be '10 seconds' not '>1mth'). – smci Apr 19 '18 at 0:11
  • 1
    @smci: Very few tags take less than 10 seconds. You might better off just editing in proper references than going to a moderator. The main reason to flag is if a user has done nothing but copied content from someone else. In that case, the moderator would need to verify the accusation and compose a moderator message. Templates help streamline the process, but it can still take closer to 10 minutes than 10 seconds. In any case, I think I explained the problems with that number in the answer, no? – Jon Ericson Apr 19 '18 at 0:23
  • @JonEricson: no you misunderstood. I was not accused of plagiarism. User B blatantly plagiarized their answer (which was posted after mine), to incorporate mine, after their answer got accepted. It literally takes 10 seconds looking at the log and timeline of dates to verify that. As to whether it takes longer to compose a moderator message about a decision than to take the decision, that sounds like it could be improved but I have no visibility. – smci Apr 19 '18 at 0:39
5

This second bullet really caught my eye (focus on the italic part on the second bullet)

  1. Unicorns are great and they make my day happier...

  2. Tools that allow decoupling moderation from communication without separating moderators and users. While Stack Overflow does have a powerful "moderator class" elected by the community, a fairly large portion of the actual moderation is performed by individual members of the site, those who've participated enough to demonstrate sufficient familiarity with the community. While this has been a fundamental part of the system for a very long time, I didn't fully appreciate how it relates to scale until I started working with very small Stack Exchange sites: the proportional cost of moderation is much higher, even though the total volume of work is lower. Many hands make (relatively) light work... As long as the system puts tools in those hands.

The proportion of moderation work to users? I'd say so. Stack Overflow has 17 moderators for somewhere around 3,060,936 users. Arduino SE (where I'm a moderator) has 3 moderators for slightly under 2,400 users! That's about a bit less than 800 users per moderator vs. over 180,000 users per moderator! No wonder SO moderators don't like using a lot of their time to respond to flags like plz delete this comment: it says "thanks" and that's bad!

Quick note: Why it's okay for smaller SE sites to be way "under-powered?" It's efficiency. Sure, it's not that smaller site mods are inefficient, there's just a million things that haven't ever been worked with. Sometimes it's like an endless cycle: whenever there's something not clear, go to meta!

Anyway, the issue with comments is they're like spaghetti; they're all over the place, which makes it really hard for the system to group together. The way they were designed probably wasn't the way they are used now. They probably were designed to be like comments on a blog: they very rarely have full conversations. It's usually between the blogger and the commenter, not a third party.

So, what can be done? Not much really. The only thing that I can think of is try to give moderators more relevant results by linking comments when one of these is triggered:

  • An @username reply
  • Mentioning a name (I agree with Jeff's comment but I think Penguin also brings up a good point...)
  • Quoting a part of the comment (*I don't think that [xyz] on your post is right.* Actually, he is right. Look at [link])

Now, there are obvious problems with this approach:

  • You'll have to use RegEx (except for the first one) and other ways to figure out if this is a match. RegEx always makes something messed up that no one will think about.
  • (In reply to Annonomus Penguin's comment):

    I agree with the penguin about Jeff Atwood is a human, but controlled by Jon Skeet.

    You could realize "second part of his name." However, things could bork if I renamed myself RegEx...

It might have to be worked with some. I realize this will never be 100% accurate, but the filter should just try to make it easier for mods to find relevant content.

The only other option is to add a "reply" button to a comment (like in chat) so you can have asynchronous conversations at the same time.

2
  • Yeah! - there is no way to start a question-or-answer-specific chat without commenting a lot and having a mod move it to chat - is there? – Menasheh Sep 20 '17 at 17:35
  • @Menasheh anyone should be able to create a chat room (over a certain rep) if I remember right off the top of my head. You could create one yourself and then comment with a link. – Anonymous Penguin Sep 20 '17 at 22:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .
https://meta.stackoverflow.com/questions/251175/stack-overflow-is-not-yet-a-vast-wasteland-a-history-of-moderator-tooling/251176
meta.stackoverflow.com
crawl=CC-MAIN-2021-21
229545
application/http; msgtype=response
sha1:YHYJP2LJNQCJYOVYUK3EY5RPXEQ2LRTC
<urn:uuid:c86a85d2-0a72-4eac-96ad-350be7fe4dae>
2021-05-11T08:21:11Z
151.101.193.69
text/html
sha1:AQ2WZT55VUNM4LA6EY46UA4ADD7HSMBL
<urn:uuid:5883e771-d56f-4a88-b2a4-8f1241774589>
https://meta.stackoverflow.com/questions/251175/stack-overflow-is-not-yet-a-vast-wasteland-a-history-of-moderator-tooling/251176
response
<urn:uuid:8b7bd5f6-7c0c-40f9-bd41-46200932290d>
isPartOf: CC-MAIN-2021-21
publisher: Common Crawl
description: Wide crawl of the web for May 2021
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-178.ec2.internal
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.2-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/