38

This is a site that gets a lot of off-topic comments, comments that really are answers, chatty comments that should be posted in chat, etc. I frequently flag these comments, but it's like I'm fighting an uphill battle. Every time I flag one comment thread, I find another. It really would be nice if there's a simple trick that we could use to get people to stop leaving off-topic comments.

Normally, there are no simple fixes. But in this case there is one. One of the community managers changed the text of the "add comment" button on Area51 to "suggest improvements", which is what comments are supposed to be used for. And, surprise surprise, off-topic comments on Area51 have all but vanished.

Is there any chance that we could get this feature? It would take at most a day to implement, we need it, and we could test it out to see if it should be implemented on the rest of the Stack Exchange sites.

4
  • 1
    Naturally, they will implement it everywhere possible, but after they test it out on less visited sites first. IPS isn't a less visited site anymore (psst.. 8 HNQs per day)
    – NVZ
    Commented Aug 26, 2017 at 15:31
  • 8
    @NVZ implementing this on a less visited site makes absolutely no sense, because less visited sites usually don't have problems with comments. We have a problem with comments and would like a solution. Let's try this out and see if it works.
    – user288
    Commented Aug 26, 2017 at 15:34
  • That I agree. But I'm saying what I think SE is doing about it. If they wanted to, they would have tested it on SO first. But did they?
    – NVZ
    Commented Aug 26, 2017 at 15:35
  • 2
    The clickbait structure of your title made me chuckle, nicely done.
    – J A
    Commented May 9, 2018 at 19:20

6 Answers 6

25

A quick update on this request:

Yes, the Community Team has submitted a request to have this feature change implemented using Interpersonal SE to test it.

Acceptance by the dev team is still pending, but with the pent up demand and backlog of feature-requests, it is still unclear how this will be prioritized. If it is accepted, the date of completion is still to be determined so I do not have any further information at this time.

Stay tuned.

4
  • 3
    Thank you for this. Fingers crossed.
    – Catija
    Commented Oct 12, 2017 at 22:29
  • 4
    What happened to The Workplace's request to test this? Commented Oct 13, 2017 at 3:43
  • 1
    Poke, poke... is there any update?
    – Tinkeringbell Mod
    Commented Apr 10, 2018 at 14:25
  • 2
    @Tinkeringbell Not that I'm aware of. It's been about 1-1/2 years since renewed Q&A development has been just around the corner, so I have to back away from this and let someone more in-the-know answer. Commented Apr 10, 2018 at 14:48
38

June 25: Experiment has ended

Ok! It's been about two months since this started, and I've turned off the altered comment text. This should be enough to give us a reasonable idea of what effect was had by this change.

With less to watch on TV, I've had time to fix a few serious errors in my initial query and also adjusted the method for associating edits and comments to avoid weird effects from edits made long after an irrelevant comment. With that done, I've updated the results in the section below, and now for a wrap-up I'll add another 30 days fore and aft.

The table below, contrasts 4 time periods: 60-30 days prior to the change, the 30 days before the change, the 30 days after the change, and the 30 days after that:

Messages
Description                     2/22-3/24  3/24-4/23   4/23-5/23  5/23-6/23   
------------------------------- ---------- ----------  ---------- ----------
Comments / post ratio                 4.45       4.31        3.40       3.79
Flags / comment ratio                 0.21       0.37        0.30       0.29
Pct comments deleted                 49.05      51.14       48.89      48.48
Comment upvotes / comment ratio       2.02       2.05        1.90       1.81
Pct comments followed by edits       11.16      11.00       14.13      13.15

The most noticeable effect here for me remains the last: a higher percentage of comments are followed by edits. That does not seem to be caused by an increase in constructive comments or edits - rather, it's caused by the reduction in comments overall (as indicated by the first metric). With fewer non-constructive comments being posted (and deleted), the portion of useful comments represents a larger portion of the total.

This clearly isn't a dramatic improvement. An awful lot of comments are still getting flagged and deleted... But, it is still an improvement.

The next question we need to answer... Can we replicate this success elsewhere?

June 3rd: Experiment results

Just as promised, I've collected the resul...

...Ok, so I said "In 30 days" when I probably should have said "after 30 days". It was a busy 30 days, ok? I had a LOT of stuff on NetFlix to watch. Clearly, the first thing learned from this experiment is "be more vague about schedules".

Anyway. Let's see how far we got in terms of achieving those 5 goals Jon suggested. Here are the numbers for each in the 30 days after April 23rd, contrasted with those for the 30 days leading up to April 23rd:

Description                     Before     After      PctChange  
------------------------------- ---------- ---------- ---------- 
Comments / post ratio                 4.31       3.40     -21.12 
Flags / comment ratio                 0.37       0.30     -20.27 
Pct comments deleted                 51.14      48.89      -4.40 
Comment upvotes / comment ratio       2.05       1.90      -7.21 
Pct comments followed by edits       11.00      14.13      28.38 

So...

  1. Fewer comments. Yes!
  2. Fewer comment flags. Yup
  3. Lower ratio of deleted comments. Indeed.
  4. More upvotes per comment. ABJECT FAIL.
  5. More edits following comments. YES!

In conclusion: this is not a dramatic change; there are noticeably fewer comments, and they would appear to be aimed more at generating improvement, but an awful lot of them still get flagged and deleted. Also, they seem to be less popular with readers... Which, in spite of failing one of the goals, may not actually be a bad sign: if folks are spending less time soapboxing and more time suggesting improvements, there may just be fewer polarizing opinions to garner votes.

It's worth keeping in mind that some of this may be due entirely to the novelty of the change. So with that in mind, I'm going to leave the experiment to run for another month and see if any noticeable change in behavior is still detectable at the end of that time. This may also be a good time to try and replicate the results on another site.

April 23rd: Ok, this experiment is live:

suggest improvements

From now until the end of the experiment, "add a comment" will no longer be available here on Interpersonal Skills; authors and those with the "comment everywhere" privilege will only be able to "suggest improvements". Kudos to Tim for reviving this request and Kasra for whipping up the functionality needed to toggle it!

There are no doubt tons of rough edges here, as my reference to the privilege illustrates; the help center, flags, user profiles and of course tons of meta posts will all still reference "comments". The goal here is just to see if this one weird trick actually works - if it turns out that it does, then we can see about making it less... weird.

Tests for success

So, how do we know if it works? Well, Jon Ericson suggested the following goals:

  1. Fewer comments.
  2. Fewer comment flags.
  3. Lower ratio of deleted comments.
  4. More upvotes per comment.
  5. More edits following comments.

All of those seem like good things to me, so let's call it a success if during the duration of this experiment any of them improve significantly over the current status quo... Unless, of course, the ones that don't improve somehow get significantly worse.

We picked this site to test on because... Well, because y'all post a LOT of comments. And a significant volume of those comments are not useful. Heck, let's be honest: an awful lot of the comments here are awful, just awful, just awfully, awfully awful. I guess what I'm saying is, there's TONS of room for improvement - so if we're gonna see this little change have an effect anywhere, it's gonna be here.

In 30 days, we'll end this experiment and analyze the results.

18
  • 1
    Since I'm still seeing "add a comment" here on meta, I'm going to assume this is main only?
    – Catija
    Commented Apr 24, 2018 at 2:58
  • 1
    Correct, main-site only
    – Shog9
    Commented Apr 24, 2018 at 2:59
  • 6
    Sounds great. Thanks :) I'd also really be interested in seeing how it affects the behavior of established IPS users vs 101 rep users. :)
    – Catija
    Commented Apr 24, 2018 at 3:01
  • It was mentioned by John Erickson that there is a relationship with upvoted comments and negativity, so I think we should be cautious how we interpret an increase in upvotes per comment. Examining the sentiment score before and after the change might be a good metric to track.
    – sphennings
    Commented Apr 24, 2018 at 3:11
  • 2
    Maybe. I'll leave it to Jon to get fancy with that data science AI magic; I trust the mods here aren't shy about deleting even upvoted comments if they're disruptive / unnecessary.
    – Shog9
    Commented Apr 24, 2018 at 3:25
  • The hover over text is still the old text. Is that going to be changing as part of the experiment too in order to better explain what acceptable comments are? Commented Apr 27, 2018 at 16:38
  • 1
    No other changes until the experiment concludes, @thunderforge
    – Shog9
    Commented Apr 27, 2018 at 17:00
  • It looks like this experiment is proving successful enough to continue beyond the 30 days. It would be fascinating to get an indication of just what impact it's had.
    – Mark Booth
    Commented May 30, 2018 at 14:33
  • 4
    I'll post the results sometime this week, @Mark
    – Shog9
    Commented May 30, 2018 at 14:40
  • 3
    It'd be interesting to see how a more established site like TWP reacts to the change. We have in the past deleted a lot of very highly upvoted but not helpful comments, so I'm not convinced that reduced comment upvoting is bad. I'm also somewhat curious if there was any skewing of the numbers if you compare the 30 days with the prior 30 days and the previous 30 days. There was a lot of crazy activity leading up to this change.
    – Catija
    Commented Jun 4, 2018 at 15:39
  • 1
    Also, with the addition of mods on the other side of the planet, comments may get deleted more quickly, giving them less time to accumulate upvotes.
    – Catija
    Commented Jun 4, 2018 at 15:40
  • 1
    There are a LOT of potential influencers here, @Daniel; new mods (as Cat noted), some notable absences from the site, renewed interest in proactively dealing with problematic questions and answers... That's why I'm going to revisit in another month & see if the results - particularly WRT comments-motivating-edits - hold for a longer period of time. I also need to try this somewhere else & see if it can be reproduced on a more established site.
    – Shog9
    Commented Jun 6, 2018 at 14:06
  • 4
    I would submit that fewer upvotes per comment is an abject SUCCESS since the comments that used to get most upvotes were those that soapboxed in the comments or made witty (but useless in a comment) quips on an answer or question.
    – Magisch
    Commented Jun 7, 2018 at 8:41
  • 1
    So... interesting... side effect(?)... interpersonal.stackexchange.com/questions/15653/… OP can't find "comment" button which leads to an edit rather than comment... this is good... but leads to some confusion... so not sure what to think. Was there a reason (technical or otherwise) for using "suggest improvement" for all cases? I know the original concept was to use something like "request clarification" for questions but I'm not sure if that's technically possible.
    – Catija
    Commented Jun 19, 2018 at 2:31
  • 2
    There are fewer comments in general, and fewer comments followed by edits @1006a. But the reduction in comments followed by edits is less than that of comments overall, so they make up a larger portion of the whole. It's hard to be certain if the reduction in comments is due to the change; even as a ratio, that can vary. But the change in the effect of those comments is pretty big.
    – Shog9
    Commented Jun 26, 2018 at 22:08
12

Yes Yes Yes. This is a great idea.

I know your pain, I have 232 comment flags, 28 alone today and yesterday are waiting for review and 188 helpful in 27 days! This is madness. I feel it's even more important on this website because everyone has some form of interpersonal skills at all different levels and have opinions on other people's interpersonal skills in comparison to their own.

This is only a small way of improving the way people comment! So yes, I would love for this to be implemented. There is a lot of noise on this SE and this would not only help towards keeping this ever so strange addition to the exchange's Q&A format (and lack thereof discussion), it'll alleviate the mods a little on flags, so they can do the important stuff.

8
  • 1
    Are you flagging individual comments or are you flagging posts with a custom mod flag asking for a purge?
    – apaul
    Commented Aug 26, 2017 at 15:38
  • @apaul34208 flagging individual comments mainly and I have done numerous times asking for a purge if the stream is just noise. Commented Aug 26, 2017 at 15:38
  • 2
    You may save the mods a little time and aggravation by flagging the post when there's several comments that need to be removed. They tend to be pretty good about separating the wheat from the chaff...
    – apaul
    Commented Aug 26, 2017 at 15:41
  • @apaul34208 Duly noted, cheers for the heads up. (although I have done some older questions today and yesterday, that doesn't take too much away from the site being noisy still) Commented Aug 26, 2017 at 15:43
  • 1
    @apaul34208 as a mod on another site, if there are comment flags on a post, I'll look through all the comments, delete those that need to be deleted, and then look again to see what flags I missed. So I personally wouldn't be aggravated by multiple flags.
    – user288
    Commented Aug 26, 2017 at 15:45
  • @Hamlet I guess that makes sense on smaller scales. I cut my teeth over on SO where the volume was much higher and it could be easy to burn through your daily allotment of flags.
    – apaul
    Commented Aug 26, 2017 at 15:48
  • 3
    @apaul34208 as a flagger, I wouldn't use multiple comment flags for one thread (too much work). As a moderator, it wouldn't make my life harder if people do.
    – user288
    Commented Aug 26, 2017 at 15:50
  • 1
    @apaul34208 it depends on how obvious the comments are. If they're "+1 this is awesome", individual flags are better. If it's a long chain of crap that needs more context, flagging the entire post is better... but each mod has their own process. I tend to click through to the post unless it's clearly a chatty comment. Not everyone does that. I know a lot of mods who prefer individual flags rather than having to hunt through 30 comments for the ones that need to die.
    – Catija
    Commented Aug 26, 2017 at 17:49
5

If Robert wants to test it here, I'm willing to be a guinea pig but I'm not going to hold my breath that it will work.

Area 51 does not reflect an actual site, the types of users, and the types of comments. Until it's actually tested on a regular site, I don't have any reason to think it will actually do anything.

Changing the wording on A51 and having it work does not mean that it will actually work here. People on the actual SE sites have been using comments for so long that it's unlikely that they'll even notice the verbiage change. It may reduce the number of new users utilizing comments incorrectly but old users have been writing comments for (potentially) years, even though they usually know that comments are 1. not for answers and 2. not for off-topic chatter.

The verbiage may help but I doubt it.

I think that other MSE solutions like comment locks or requiring local reputation to comment on protected questions could be more effective - though I'd prefer they have expiration dates if they were to be implemented. These would actually make it impossible to comment either at all or with insufficient local reputation.

7
  • 1
    I think it's actually very likely that it would work. From personal experience, when I remember to ask the question "is this a suggestion for how to improve an answer", I often find myself not leaving comments that I would have left otherwise, or changing the tone of my comments. But I agree, let's test it out and see if it works. It really wouldn't be hard to implement.
    – user288
    Commented Aug 26, 2017 at 18:05
  • You're a moderator... which already implies that you have been trusted to have some modicum of common sense... many users don't have that. Many of the people who are posting comments just want to argue about something. Those people will not care whether the text says "add a comment" or "suggest improvement".
    – Catija
    Commented Aug 26, 2017 at 18:10
  • 1
    Not every comment is the result of trolling. My impression is that a lot of the comments aren't malicious, they're just jokes or answers or off-topic or a conversation. And even when comments do veer towards violating be nice... I'm pretty sure that everyone on SE has violated be nice at some point. (The difference is that unlike trolls, people apologize and learn from their mistakes). For ordinary members, a little reminder in the interface might stop people from getting carried away. So yes, trolls would still need moderation, but this feature request isn't necessarily about trolls.
    – user288
    Commented Aug 27, 2017 at 1:02
  • @Hamlet who said anything about trolls?
    – Catija
    Commented Aug 27, 2017 at 3:59
  • "Many of the people who are posting comments just want to argue about something."
    – user288
    Commented Aug 27, 2017 at 16:30
  • Well, I guess that it's useful to know that your definition of "troll" is "someone who wants to argue about something".
    – Catija
    Commented Aug 27, 2017 at 16:40
  • my definition of troll is someone who rather than participating constructively, is there in bad faith and wants to do things that are not constructive. People who are here for the sole purpose of getting into arguments in comments, rather than say, using comments to point out inaccuracies, or asking questions, would qualify, right?
    – user288
    Commented Aug 28, 2017 at 17:50
0

Are we taking this just a bit too far?

There are many times when the comments are as engaging and delightful as the answers, even though they aren't direct questions/suggestions on how to improve the answer. Things like "Ha, that reminds me of a time when..." I'll suggest that this sort of tangential marginalia does add value in an indirect way.

Honestly, I know I'm bonking my head against a brick wall on this, but I have to say it. I predict that relentlessly removing tangential and non answer-improving comments will make the IPSE experience fussy and tedious.

Therefore ...

Let's do it. Prove me wrong! "Life is the best teacher", as they say.

0

Great idea!

I just looked at this and with mouse over the following text is visible. I guess it would be good to change the wording on this as well. Otherwise people will ask where the comments are - "Use comments"

enter image description here

4
  • 4
    See shog's post above. This is an experiment to see if this changes the commenting behavior on the main site. If it proves successful then they're going to see about changing the language elsewhere.
    – sphennings
    Commented Apr 25, 2018 at 11:57
  • 1
    @sphennings: I understand, my idea was to prevent users from asking "where can I comment" is they see that text but no "comment button".
    – user8838
    Commented Apr 25, 2018 at 15:39
  • 4
    We're going to have many sources of confusion until this experiment is over. Even longer if we decide that we want to implement the change permanently. There are a lot of spots where comments are mentioned on this site and all of them will need to be changed. If you see users who are confused let us know.
    – sphennings
    Commented Apr 25, 2018 at 15:48
  • @sphennings: I will try to open all those cameras in front of confused users...
    – user8838
    Commented Apr 26, 2018 at 0:01

You must log in to answer this question.
https://interpersonal.meta.stackexchange.com/questions/1528/this-one-weird-trick-gets-rid-of-all-off-topic-comments-lets-use-it/2839
interpersonal.meta.stackexchange.com
CC-MAIN-2024-38
224015
application/http; msgtype=response
sha1:QSUAOSD2Y2FV2Y3WZ3OFP73PA5E3RXNQ
<urn:uuid:9893233a-5dee-4d19-b5e4-787b583af6ae>
2024-09-10T15:13:37Z
172.64.144.30
text/html
sha1:MG3M6XIONK6564C3GKI54336I4TJEG4W
<urn:uuid:49fc03a4-fc0b-49d2-b7b4-ba0365f386b4>
https://interpersonal.meta.stackexchange.com/questions/1528/this-one-weird-trick-gets-rid-of-all-off-topic-comments-lets-use-it/2839
response
<urn:uuid:bc8b6735-15a5-4cb6-b0a8-9c3fd2f0da84>
isPartOf: CC-MAIN-2024-38
publisher: Common Crawl
description: Wide crawl of the web for September 2024
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-5
software: Apache Nutch 1.20 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.5-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/