Twitter  Is the river behind your house rising? A British Twitter bot will tell you nie.mn/1kQfI9f  
Nieman Journalism Lab
Pushing to the future of journalism — A project of the Nieman Foundation at Harvard
61freshscreenshot

The Hub of the Twitterverse: The Boston Globe has built a localized, tweet-powered news aggregator

61Fresh is the latest experiment from Globe Lab, the paper’s R&D department, to find ways to harness social signals to see what readers are talking about.

61freshscreenshotIn the Boston area, the passing of a Dunkin Donuts franchise is news. So are the finer points of Tom Brady’s hair. But in this instance, we know the closing of the Mass Ave. Dunkin reached a certain level of…prominence? virality? social lift? — because it made an appearance on 61Fresh, the experimental news aggregator built by the Globe Lab, the research and development team inside The Boston Globe.

This tweet, in particular, caught fire:

(In fairness, Matt Damon, a napkin, and apples also made that Dunkin newsworthy.)

What, exactly, is 61Fresh? And what’s with the name?

“Half the people who respond to it ask, ‘What’s up with the name?’ And I say, ‘Oh, it’s like 617, but six-one fresh,” said Chris Marstall, a creative technologist in the Globe Lab. (Out-of-towners: 617 is the primary Boston area code.)

The name, like the site, is an experiment. Think of 61Fresh as a close cousin to something like Techmeme, Mediagazer, or our very own heat-seeking Twitter bot, Fuego. Where those algorithmic link aggregators are topic-based, 61Fresh instead has a geographic focus: “It’s philosophically the same,” Marstall says. “It’s using a network to find the most relevant stories in a narrow domain for now.”

Every 10 minutes, 61Fresh takes a spin around a localized slice of Twitter searching for rising news items. On any given day, you’re likely to find stories on apartment fires, startup companies, or the Bruins. Unlike some aggregators that mix in some human control, 61Fresh is 100 percent algorithm-driven — which means both that it runs without the need for ongoing staff resources and that the end product can be a little raw, with different versions of the same story repeated during breaking news.

The project is still in early stages, and Marstall said they’re still figuring out the best way for users to interact with 61Fresh, which for now exists as a website and Twitter feed. “The stories that pop up are stories drawing interest on Twitter,” he said. “It’s sort of an open question if this can be a standalone site in the future or if this is something that is a compelling part of Boston.com.”

But how does 61Fresh figure out what’s news, what’s talk, and what’s noise? The algorithm built by Globe Lab looks for tweets that link to stories or updates from just over 500 pre-selected websites in the Boston area. Aside from recognizable names like Boston.com, and the Boston Herald, the universe also includes stories from radio, TV, town government sites, universities, and blogs, including the growing network of online outlets that cover the particular cross-section of education, technology, and culture unique to the Boston area.

“We got three daily newspapers, a bunch of weekly newspapers, we got TV and radio, and all these blogs. That’s just the tip of the iceburg,” Marstall said. “It shows there’s a lot of vibrancy here and a lot of smart people fighting to get attention for their stories.”

Here’s how they explain it:

We’re downloading a real time stream of the local Twitter posts (originating roughly within Route 128) that link to stories on one of those 500 sites. Every tweet represents a vote. Our algorithm mixes popularity with freshness and a couple other ingredients, discounting the value of ‘official’ accounts and bots. We group similar articles together, generating a diverse feed that changes frequently throughout the day.

What you get with 61Fresh is a cascade of headlines, but in the form 140 characters, as the links that are pulled in use the text of the most prominent tweet on a particular story. “We want this to look like a conversation, we don’t want this to look like a bunch of headlines,” Marstall said. Keeping in that theme, each story pulled into 61Fresh shows a roster of people who have tweeted the link, as well as Twitter prompts to reply, retweet, or favorite.

One interesting feature on 61Fresh is the “mute sports” button, which, as the name implies, lets users excise anything related to sports from their feed. “Early on, we realized the sports stories could really dominate,” Marstall said. “I like it — I’m a sports fan — but I know for a lot of people it’s total noise.”

Ali Hashmi, a former Knight fellow at the Globe Lab who is now at MIT, created a natural language processing system that helps the aggregator parse sports from the rest of the news. Using a base set of 1,000 sports stories, the system learns how to spot a sports article and pluck it out of the stream of stories, Marstall said, if you want neither pucks, punts, nor pitchers.

61Fresh is one of the latest projects to come out of the partnership between the Globe and MIT’s Center for Civic Media. Originally, the lab’s creative technologists and Knight fellows from MIT were working on a way to archive Globe reporters’ tweets from the Boston Marathon bombing. Lessons and code from that were put into use for 61Fresh, Marstall said.

This is not the first time the Globe Lab has experimented with social media feeds as a means of monitoring life in Boston. Previously the lab developed Snap, a tool that maps Instgram photos to their geolocated spots around the city.

Jeff Moriarty, the Globe’s vice president of digital products and general manager of Boston.com, said 61Fresh “fits in with the idea that we’re moving Boston.com to include more of what people are saying and what people are doing right now in Boston.” With the split between the two Globe sites, Boston.com has been focused on news and information, but also digital tools and online discussion, Moriarty said. The central idea is to be able to give readers a sense of what they need to know in a format that works best for them. “By combining both the news and what’s popular on Twitter, you get this pretty unique view of what’s happening right now,” Moriarty said.

At the moment, 61Fresh is a little rough around the edges, the design is not particularly eye catching, and the team at Globe Lab is catching problems as they arise. But the lessons from the project will be applied in the future, either for a stand-alone aggregator or built into parts of Boston.com. Moriarty said he thinks the project can have a lot of value for niche topics like sports or technology — pulling in tweets and headlines for a Boston audience interested in the NFL or biotech, for instance. As Moriarty sees it, the job of Boston.com isn’t just to deliver the news, but to provide context, conversation, and the other pieces to help readers see the bigger picture.

“I think it really needs to be what we do as a media organization: People want someone to help them make sense of what is going on around them,” Moriarty said.

                                   
What to read next
RevealCIRlogo
Justin Ellis    July 18, 2014
With $3.5 million in grant funding and an eye for collaboration, the Center for Investigative Reporting and PRX aim to bring deep investigations to radio and podcasting.
  • Laura Oliver

    Very interesting. Will be watching to see how this is used e.g. as an internal newsgathering tool by the Globe or embedded on a certain section of the site. The latter may throw up questions around verification and a healthy discussion about who sets the news agenda.
http://www.niemanlab.org/2013/09/the-hub-of-the-twitterverse-the-boston-globe-has-built-a-localized-tweet-powered-news-aggregator/
www.niemanlab.org
crawl=CC-MAIN-2014-23
71661
application/http; msgtype=response
sha1:GJLF6V6IO35NIOJTLO4T4WMPVLX5Q3OI
<urn:uuid:454e90c5-e297-4b5b-9b19-445e504dccd1>
2014-07-23T11:37:25Z
216.92.13.127
sha1:GZE27QK7D7PDX4PGNFAGQU6FM72ANQQ3
<urn:uuid:42a12e5f-b34f-4da1-b320-8b1d0a8fc2c1>
http://www.niemanlab.org/2013/09/the-hub-of-the-twitterverse-the-boston-globe-has-built-a-localized-tweet-powered-news-aggregator/
length
response
<urn:uuid:605d48c6-dbf1-450b-afcb-a5ff0674d8d3>
robots: classic
hostname: ip-10-33-131-23.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2014-23
operator: CommonCrawl Admin
description: Wide crawl of the web with URLs provided by Blekko for July 2014
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf