OUseful.Info, the blog…

Trying to find useful things to do with emerging technologies in open education

Searching for Twitter Hashtags and Finding Hashtag Communities

Over the last few weeks I’ve been messing around more than I should with Twitter, and in particular trying to get a feel for how we might use hashtag communities as a well of identifying and growing community structures in a particular topic area (see posts all over OUseful.info for more details).

A couple of days ago, @clarileia raised the question of how you find new hashtags, so I had a little tinker today putting together a couple of hacks (Twitter hashtag search pipe and Twitter my network hashtags) that let you identify recently used Twitter hashtags associated with a particular search term, or with a specified user’s recent friends or followers.

Twiitter hashtag search http://pipes.yahoo.com/ouseful/twitterhashtagsearch

[Note: at the time of writing, Pipes appears to be running a little slow... if the Pipe appears to stall, it does work, honest... try it again later ;-)]

At the core of all the hacks is a clunky hashtag tokeniser pipe that takes a Twitter status update and pulls out the hashtags:

This utility pipe works by taking the status update, extracting the hashtags using a hacked together regular expression, splits the separate hashtags into separate feed items, and then filters them to emit only legitimate hashtags.

The utility pipe is then used in a search powered pipe, which searches twitter for the 100 most recent tweets containing the search terms and then scans those for hashtags; and a ‘personal network hashtags’ pipe that takes a Twitter username, pulls back the tweets from their one hundred most recent friends, and their one hundred most recently followers, and then scans those tweets for hashtags.

For example, here’s the search pipe:

Both pipes have a common output routine – the list of hashtags is filtered through the Unique block, which also returns a count of how many times each hashtag has appeared. The hashtags are then ordered and filtered according to the minimum number of required occurrences in the sample. A regular expression adds the number of occurrences of each hashtag.

The pipes could be extended to pull in more search results, or more followers/friends (maybe the first hundred friends/followers as well as the most recent hundred?) but that’s left as an exercise for the reader. As for the use case – I dunno? Maybe integration with the OUseful TwitterMyHashtag apps? Or perhaps @clarileia had a use case in mind?!;-)

PS thanks to PJ on the Yahoo Pipes team for getting back to me earlier today when I was struggling with a slow running pipes editor… I’m now totally reliant on Pipes for many apps, and especially for rapid the majority of my prototyping, so when Pipes is slow, I feel as happy as if I’ve lost an unbacked up server… Brian Kelly would probably tell me I need to do a risk assessment… I’ve already done one: What Happens If Yahoo! Pipes Dies? – but I haven’t made a start on the contingency stuff that was considered there…

Written by Tony Hirst

September 23, 2009 at 9:17 am

5 Responses

Subscribe to comments with RSS.

  1. I was really just commenting on the impact of creating a list of twittering librarians and then thinking that now I’m set up using Twitter I don’t go out looking for new people to follow, other than those who start to follow me. I didn’t expect anyone to take it up and work out how to find people using particular hashtags as a way of doing that, so thanks and I’ll have to think about how to use it!

    clari

    September 23, 2009 at 9:56 am

  2. [...] Searching for Twitter Hashtags and Finding Hashtag Communities for details about the [...]

  3. [...] use that to generate a custom search engine over the hashtaggers’ linked to homepages); – find other hashtags being used by the hashtaggers (that is, hashtags they may be using in arbitrary [...]

  4. [...] and Event based Twittering and Searching for Twitter Hashtags and Finding Hashtag Communities – [...]

  5. [...] exactly, using Yahoo Pipes to find what hashtags are being used around a particular search term (Searching for Twitter Hashtags and Finding Hashtag Communities) or by members of a particular list (What’s Happening Now: Hashtags on Twitter Lists; that post [...]


Comments are closed.

Follow

Get every new post delivered to your Inbox.

Join 795 other followers

%d bloggers like this:
http://blog.ouseful.info/2009/09/23/finding-hashtag-communities/?orderby=ID&order=ASC
blog.ouseful.info
crawl=CC-MAIN-2014-41
64910
application/http; msgtype=response
sha1:4RYQ5ZGEIMZWPPDJS5LBNQTB6XQ2AZK5
<urn:uuid:afd10074-4c33-41f1-944d-e48ef4a79dc7>
2014-09-19T15:48:58Z
66.155.11.238
sha1:EHAFCJG5WCGIWKURUARAFDZ7H37RGJCE
<urn:uuid:614e1d37-9689-4bae-b2fc-075c9bac6740>
http://blog.ouseful.info/2009/09/23/finding-hashtag-communities/?orderby=ID&order=ASC
length
response
<urn:uuid:014f9ce9-0616-4b59-bcf0-fd4755fe6083>
robots: classic
hostname: ip-10-196-40-205.us-west-1.compute.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2014-41
operator: CommonCrawl Admin
description: Wide crawl of the web with URLs provided by Blekko for September 2014
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf