  Popularity systems, like Twitter retweets, Reddit upvotes, and Pinterest pins
have the potential to guide people toward posts that others liked. That,
however, creates a feedback loop that reduces their informativeness: items
marked as more popular get more attention, so that additional upvotes and
retweets may simply reflect the increased attention and not independent
information about the fraction of people that like the items. How much
information remains? For example, how confident can we be that more people
prefer item A to item B if item A had hundreds of upvotes on Reddit and item B
had only a few? We investigate using an Internet game called GuessTheKarma that
collects independent preference judgments (N=20,674) for 400 pairs of images,
approximately 50 per pair. Unlike the rating systems that dominate social media
services, GuessTheKarma is devoid of social and ranking effects that influence
ratings. Overall, Reddit scores were not very good predictors of the true
population preferences for items as measured by GuessTheKarma: the image with
higher score was preferred by a majority of independent raters only 68% of the
time. However, when one image had a low score and the other was one of the
highest scoring in its subreddit, the higher scoring image was preferred nearly
90% of the time by the majority of independent raters. Similarly, Imgur view
counts for the images were poor predictors except when there were orders of
magnitude differences between the pairs. We conclude that popularity systems
marked by feedback loops may convey a strong signal about population
preferences, but only when comparing items that received vastly different
popularity scores.
