Australian Twitter News Index: Hunting Down Spammers

We start this week’s ATNIX with a brief detour – please bear with me, or just skip down to the next section if you’re only interested in this week’s results.

The approach to researching the uses of Twitter which we’re taking with the Australian Twitter News Index is a somewhat unusual one. Much recent Twitter research – including plenty of the work we’ve been doing in the Mapping Online Publics project – has focussed on hashtag datasets. Such datasets are useful because they’re essentially self-selecting: users mark their tweets as relating to certain topics by including hashtags, and this makes those tweets as easy to find for other users as they are easy to track and analyse for researchers. Hashtag datasets are topically unified and generally well-behaved, therefore.

Our ATNIX datasets, on the other hand, represent a very different cross-section of Twitter activity. Here, we’re dealing with a collection of all tweets that happened to link to a particular news site, across a wide range of topics and from users who are for the most part almost certainly unaware of one another. Posting to a hashtagged conversation, you’re already aware that the hashtag exists, and can easily see what other users have posted to the hashtag. Posting a link to, say, the ABC News site, you’re not usually aware at all of how many other Twitter users may be doing the same at any one point, and you don’t have an easy way of finding out.

This is why the daily and weekly patterns we find in ATNIX are so interesting – they represent a more or less instant response to the news of the day across the Australian Twittersphere which isn’t usually driven by the unifying forces of hashtags; for the most part, ATNIX tracks genuine reactions to the news of the day, which aren’t artificially orchestrated or coordinated.

My point in spelling this out is this: the comprehensive, whole-of-Australia nature of the ATNIX data means that when we do see significant short-term spikes or longer-term movements in the level of activity around particular news sites, they tend to be meaningful. Week by week, I have been able to highlight the key news events on Twitter in this column by pointing to the most widely shared stories and reactions. In turn, this has made it all the more perplexing that – once we filtered out the hair growth spam and dogged oversharing of columnists’ critiques of the PM – we haven’t been able to find a meaningful explanation of the inexorable growth in the number of tweets linking to news.com.au.

Until now.

Regular readers of my ATNIX updates will recall that a few weeks back, there was an unusual spike in news.com.au links because they were used by spammers promoting hair growth products to mask their posts with legitimate-looking content. We filtered out those tweets. More recently, one very active anti-Gillard campaigner took it on themselves to tweet about every last article by blogs.news.com.au columnists which was critical of Julia Gillard, often multiple times a day, thereby substantially boosting the number of links received by that opinion section. This week, that account posted some 1,200 such links (and is the most active account linking to that site) – but obsessive as they may be, those tweets are legitimate, and we’ve retained them in the blogs.news.com.au count. They don’t much affect news.com.au itself, at any rate.

And still, this week’s numbers are the best for news.com.au yet. At just over 20,000 tweets linking to it, the site received its best-ever result, well above its long-term average of 14,000 links per week. A closer look reveals that there’s a substantial number of links to news.com.au which redirect to the site via news.com.au.feedsportal.com (8,200 of the 20,000 this week). In itself, that’s legit – it’s just another redirection service, in essence; plenty of the Sydney Morning Herald links come in via sites like feedproxy.google.com or feeds.feedburner.com as well. Feedsportal does display interstitial ads before redirecting, though.

What’s very odd, though, is that in the feedsportal.com links, we very frequently get blocks of tweets from different users, who each link to the same story within a few seconds of each other. Not retweets, but original tweets, e.g.

Firebrand who renamed Bombay dies: BAL Thackeray, founder of the right-wing Hindu party Shiv Sena, was a firebra… http://t.co/[…]

(so, not exactly breaking news). The t.co link (which is different in each of the tweets in the block) resolves to a bit.ly short URL which is identical across the whole group – but as the tweets are from different users, made at the same time, that makes no sense. I’ve further spotchecked a few of the accounts which appear frequently in these tweet blocks – and of the five I checked, three had been suspended already. And the size of these tweet blocks is largely consistent as well – this week, we get several blocks of 16 tweets at first, then we go up to several blocks of 20, and several blocks of 26, then that drops back down to several consecutive blocks of 23 (probably as a few accounts are banned), and so on.

There’s only one conclusion: we’re dealing with yet another case of spam, if a little more sophisticated than the hair growth spammers; this lot appear to be looking to make money from the interstitial ads at Feedsportal. The issue for me now is that some other users (though not a lot, by comparison) are using feedsportal.com legitimately – so while I can exclude any feedsportal.com redirects from the dataset from now on, that will also undercount legitimate links to news.com.au to some extent…

Yeech, spammers.

ATNIX Week 47: 19-25 Nov. 2012

But let’s move on to the weekly numbers, then. Having removed the Feedsportal links from the news.com.au data, what’s left this week are some 147,000 tweets linking to our Australian news Websites: a result which is just slightly above the long-term average. Most notably, the news sections of the ABC Website come out ahead of the Sydney Morning Herald this week, and news.com.au is back a its more realistic fourth place. The remaining sites are relatively unchanged from the previous week.

ATNIX 47/2012: News Sites http://mappingonlinepublics.net/

Amongst the opinion and commentary sites and sections, by contrast, the Fairfax flagships are back in the lead, pushing The Conversation into third place; overall, too, at close to 22,000 tweets, the number of links to opinion articles has increased by some 3,000 tweets. blogs.news.com.au is in a strong fourth place, but again it must also be noted that a good half of its links come from a single, very committed right-wing Twitter account – if and when that user goes on summer holidays, blogs.news.com.au should fall back to a level which would be just above that of independent opinion site New Matilda.

ATNIX 47/2012: Opinion Sites http://mappingonlinepublics.net/

While the ranking of opinion sites always tends to fluctuate slightly, here, too, there’s a certain amount of stability in the overall picture; for the long-struggling Global Mail, I’m afraid that’s bad news, incidentally, as it seems to have returned to its pre-redesign baseline of some 500 tweets per week.

On to the daily patterns, then: for the news sites, we clearly see how ABC News has pulled ahead of the usually closely-matched Sydney Morning Herald this week, at least on the weekdays. Oddly, however, there’s no obvious driver for this lead: major weekday stories cover the discovery of a fire tornado in Australia (560 tweets), an Amnesty International report on the detention camp in Nauru (470 tweets), the death of author Bryce Courtenay (320 tweets), as well as, very oddly, a 2004 piece on the health of gladiators in Ancient Rome. Leading stories in the Sydney Morning Herald, by contrast, cover the strange disappearance of a Pacific island (470 tweets) and the eruption of a New Zealand volcano (420 tweets).

ATNIX 47/2012: News Sites http://mappingonlinepublics.net/

The daily volume of tweets linking to opinion and commentary sites and sections shows a greater level of fluctuation, as usual. The overall lift in links to blogs.news.com.au, courtesy of a lone fan, is plainly evident over the past five weeks, while the Sydney Morning Herald and The Age score some notable wins late in the week.

The Saturday spike for the Sydney Morning Herald is caused by a strongly-worded editorial on the nation’s policy on asylum seekers, which boldly notes that “Australia does not have an asylum-seeker problem; Australia has a political leadership problem.” Syndicated across the Fairfax titles, it is nonetheless the SMH which picks up the majority of the links, with more than 430 tweets referencing the article.

ATNIX 47/2012: Opinion Sites http://mappingonlinepublics.net/

The Thursday spike for The Age is less obvious, driven as it is by a range of stories (of which a good number are related either to the Opposition’s continuing attack on Julia Gillard, or the government’s hardening policy on asylum seekers). No clear frontrunner emerges here, however.

But this, in itself, is a notable story, too: what’s largely absent from ATNIX this week is any indication that Australian Twitter users care about, nor feel compelled to share with others, the column miles which have by now been devoted to the allegations about the PM’s conduct in her previous career. Recent opinion polls seem to provide little indication that other Australians think any differently.

Join the conversation

4 Comments sorted by

  1. Sean Lamb

    Science Denier

    "Australian Twitter users care about, nor feel compelled to share with others, the column miles which have by now been devoted to the allegations about the PM’s conduct in her previous career. "

    Thats because Twitterers have a strong left bias and the unfolding Gillard trainwreck simply fills them with despondency. Who wants to tweet stories about the inexorable crumbling of the beloved deliverer of the great Misogyny speech?

    Ahh remember those halycon days, how the Twittersphere went wild with delight. Blissed was it in that dawn to be alive, but to be young was very heaven.

    report
    1. Craig Minns

      Self-employed

      In reply to Sean Lamb

      Twits have a strong left bias because they're young. The events being described occurred when many of them were in nappies and to them they're ancient history. Moreover, there is a strong female bias in the user base, as I understand it, and Gillard has made much of her "grrrrl power" of late,so naturally these young women will be offering sororal solidarity - that's what young women do, after all.

      An interesting article spoilt by the writer's need to make his own political point in the closing sentence. It's fascinating how many reporters of news feel so compelled to be art of it...

      report
    2. Craig Minns

      Self-employed

      In reply to Sean Lamb

      A small correction - my own bias is showing - apparently the user base is fairly even divided on gender grounds. It seems that our current generation of young men is just as motivated by social vacuities as their female counterparts. A genuine feminist triumph!

      report
  2. Yoron Hamber

    Thinking

    Impressive use of statistics and human deduction. It seems those spammers are becoming more and more sophisticated, don't they? Soon to be incorporated in all NSA 'lookalike' digital defences and, naturally, attacks :)

    I confidently predict that this kind of use of statistics will grow in importance the more some 'groups' want us to 'believe' in the importance of some 'beliefs/opinions'. Keep it up..

    report
http://theconversation.com/australian-twitter-news-index-hunting-down-spammers-11040
theconversation.com
crawl=CC-MAIN-2013-20
48819
application/http; msgtype=response
sha1:7B2LXDPH6K7IPISNYZ3WG5XOFKIN7MRO
<urn:uuid:7384e566-a3f6-4b4b-a95d-ba38df2a6651>
2013-06-19T22:44:48Z
91.186.19.133
sha1:DZTQIGZNG56YYBXC4NOKFH6O5ISAPQ72
<urn:uuid:c39ed890-f1f6-4dc6-8d06-927c7e8d9b45>
http://theconversation.com/australian-twitter-news-index-hunting-down-spammers-11040
response
<urn:uuid:f8b8d3ec-7a9c-4fb9-b65b-47f570f229d9>
robots: classic
hostname: ip-10-60-113-184.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2013-20
operator: CommonCrawl Admin
description: Wide crawl of the web with URLs provided by Blekko for Spring 2013
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf