  Stack Overflow has accumulated millions of answers. Informative comments can
strengthen their associated answers (e.g., providing additional information).
Currently, Stack Overflow hides comments that are ranked beyond the top 5.
Stack Overflow aims to display more informative comments (i.e., the ones with
higher scores) and hide less informative ones using this mechanism. As a
result, 4.4 million comments are hidden under their answer threads. Therefore,
it is very important to understand how well the current comment hiding
mechanism works. In this study, we investigate whether the mechanism can
effectively deliver informative comments while hiding uninformative comments.
We find that: 1) Hidden comments are as informative as displayed comments; more
than half of the comments (both hidden and displayed) are informative (e.g.,
providing alternative answers, or pointing out flaws in their associated
answers). 2) The current comment hiding mechanism tends to rank and hide
comments based on their creation time instead of their score in most cases due
to the large amount of tie-scored comments (e.g., 87% of the comments have
0-score). 3) In 97.3% of answers that have hidden comments, at least one
comment is hidden while there is another comment with the same score is
displayed (i.e., we refer to such cases as unfairly hidden comments). Among
such unfairly hidden comments, the longest unfairly hidden comment is more
likely to be informative than the shortest unfairly displayed comments. Our
findings suggest that Stack Overflow should consider adjusting their current
comment hiding mechanism, e.g., displaying longer unfairly hidden comments to
replace shorter unfairly displayed comments. We also recommend that users
examine all comments, in case they would miss informative details such as
software obsolescence, code error reports, or notices of security vulnerability
in hidden comments.
