You can now read 10 articles each month for free on BostonGlobe.com.

The Boston Globe

Nation

Library of Congress finds archiving tweets no small task

WASHINGTON — In the few minutes it will take you to read this story, 3 million new tweets will have flitted across the publishing platform Twitter and ricocheted across the Internet.

The Library of Congress is busy archiving the sprawling and frenetic Twitter canon — with some key exceptions — dating back to the site’s 2006 launch. That means saving for posterity more than 170 billion tweets and counting, with an average of more than 400 million new tweets sent each day, according to Twitter.

Continue reading below

But in the two years since the library announced this unprecedented acquisition project, few details have emerged about how its unwieldy corpus of 140-character bursts will be made available to the public.

That’s because the library hasn’t figured it out yet.

‘‘People expect fully indexed — if not online searchable — databases, and that’s very difficult to apply to massive digital databases in real time,’’ said Robert Dizard Jr., deputy librarian of Congress. ‘‘The technology for archival access has to catch up with the technology that has allowed for content creation and distribution on a massive scale.’’

“Twitter is focused on creating and distributing content; that’s the model,’’ he said. “Our focus is on collecting that data, archiving it, stabilizing it, and providing access; a very different model.’’

Colorado-based data company Gnip is managing the transfer of tweets to the archive, which is populated by a fully automated system that processes tweets from across the globe.

Continue reading below

Each archived tweet comes with more than 50 fields of metadata — where the tweet originated, how many times it was retweeted, who follows the account that posted the tweet, and so on — although content from links, photos, and videos attached to tweets are not included. For security’s sake, there are two copies of the complete collection.

But the library hasn’t started the task of sorting or filtering its 133 terabytes of Twitter data, which it receives from Gnip in chronological bundles, in any meaningful way.

‘‘It’s pretty raw,’’ Dizard said. ‘‘You often hear a reference to Twitter as a fire hose, that constant stream of tweets going around the world. What we have here is a large and growing lake. What we need is the technology that allows us to both understand and make useful that lake of information.’’

For now, giving researchers access to the archive remains cost-prohibitive for the library, which has spent tens of thousands of dollars on the project so far, Dizard says. Like many federal agencies, the Library of Congress has been hit by budget cuts in recent years. Without a major overhaul to its computing infrastructure, it isn’t equipped to handle even the simplest queries.

‘‘We know from the testing we've done with even small parts of the data that we are not going to be able to, on our own, provide really useful access at a cost that is reasonable for us,’’ Dizard said. ‘‘For even just the 2006 to 2010 [portion of the] archive, which is about 21 billion tweets, just to do one search could take 24 hours using our existing servers.’’

Instead, the library is exploring whether it might be able to afford to pay a third party to provide public access to the archive. But for those who have immediate research interests — and many people have contacted the library, Dizard says — the wait is maddening.

You have reached the limit of 10 free articles a month

Stay informed with unlimited access to Boston’s trusted news source.

  • High-quality journalism from the region’s largest newsroom
  • Convenient access across all of your devices
  • Today’s Headlines daily newsletter
  • Subscriber-only access to exclusive offers, events, contests, eBooks, and more
  • Less than $1 a week
https://www.bostonglobe.com/news/nation/2013/01/19/separating-tweet-from-chaff-not-easy-for-congress/LVjpLUa9Fr0tOuu6pwKczI/story.html
www.bostonglobe.com
crawl=CC-MAIN-2014-15
48508
application/http; msgtype=response
sha1:44DODNHBJO435D5X4CEI6BBEYAR6BUMX
<urn:uuid:fc0150e3-4524-4045-ba66-2f17f32c5d30>
2014-04-25T03:28:55Z
23.218.148.136
sha1:34C653KSBVO6ZQU4W4VOWH4HBAXKGIP6
<urn:uuid:2039e9c1-2446-49aa-9f4d-b25d9458075c>
https://www.bostonglobe.com/news/nation/2013/01/19/separating-tweet-from-chaff-not-easy-for-congress/LVjpLUa9Fr0tOuu6pwKczI/story.html
response
<urn:uuid:8a3c25f3-45a7-400d-85f3-e41e7ecc1093>
robots: classic
hostname: ip-10-147-4-33.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2014-15
operator: CommonCrawl Admin
description: Wide crawl of the web with URLs provided by Blekko for April 2014
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf