Nieman Foundation at Harvard
HOME
          
LATEST STORY
Jeff Israely: Five years in, our news startup is seeing the pace of change slow
ABOUT                    SUBSCRIBE
July 19, 2012, 10:38 a.m.
Twitter preserved

That plan to archive every tweet in the Library of Congress? Definitely still happening

It has turned out to be quite an undertaking, but the Library plans to make good on its promise to America.

Twitter preserved

A little more than two years ago, the Library of Congress announced it would preserve every public tweet, ever, for future generations.

That’s right. Every public tweet, ever, since Twitter’s inception in March 2006, will be archived digitally at the Library of Congress. That’s a LOT of tweets, by the way: Twitter processes more than 50 million tweets every day, with the total numbering in the billions.

Fifty million tweets a day. How cute. That number is now 400 million, according to Twitter CEO Dick Costolo. (The first comment on the project’s FAQ page sums up much of the Internet’s reaction: “It’s critical the future generations know what flavor burrito I had for lunch.”)

We hadn’t heard about this project in some time. Last week a story on Canada.com quoted a social-media researcher as saying the LoC “has quietly backed away from the commitment.”

False, said Library spokeswoman Jennifer Gavin; the project is very much still happening. Good librarianship, she said, moves more slowly than Twitter.

“The process of how to serve it out to researchers is still being worked out, but we’re getting a lot of closer,” Gavin told me. “I couldn’t give you a date specific of when we’ll be ready to make the announcement.”

The Library first revealed its plans in a tweet on April 14, 2010, but apparently that was before sorting out with Twitter the logistics of acquiring all that data. Petabytes of data.

“We began receiving the material, portions of it, last year. We got that system down. Now we’re getting it almost daily,” Gavin said. “And of course, as I think is obvious to anyone who follows Twitter, it has ended up being a very large amount of material.”

Gavin said the archive will be made available to anyone with a library card, but only on the premises in Washington. “My understanding is that at this time we do not intend to make it available by web,” she said, but that may be subject to change. It’s not meant to be the Ultimate Twitter Search Box we’ve always dreamed of.

In fact, there will be a six-month embargo on fresh tweets (even though, obviously, the data is publicly available — if you can find it). That agreement has been in place since the deal was struck. Twitter said then the tweets could be used only “for internal library use, for non-commercial research, public display by the library itself, and preservation.”

The challenge now is finding ways to refine the raw data in useful ways. Sort by keywords? Date? Sentiment? Burrito flavor? Gavin said the Library is still figuring out the user interface.

POSTED     July 19, 2012, 10:38 a.m.
SHARE THIS STORY
   
Show comments  
Show tags
 
Join the 15,000 who get the freshest future-of-journalism news in our daily email.
Jeff Israely: Five years in, our news startup is seeing the pace of change slow
“The future is already here, and we have to hustle every day to survive. And succeed.”
This: Vox.com hires Andrew Golis as its first general manager
“He is going to be tasked with thinking about what are the big swings that we want to take in the next few years.”
Slate, now 20 years old, reflects on the value of taking the long view and not chasing digital media trends
“One of the things you’ve seen across the marketplace for the last five years is a lot of companies are chasing the same kind of traffic from the same social distribution mechanisms…It’s not a recipe for producing a distinctive media brand.”
http://www.niemanlab.org/2012/07/that-plan-to-archive-every-tweet-in-the-library-of-congress-definitely-still-happening/
www.niemanlab.org
crawl=CC-MAIN-2016-40
102723
application/http; msgtype=response
sha1:SIQRK6CXK2NREK2BYGLGY5XGGQKJYHXM
<urn:uuid:e9ad56ac-22c6-4f6c-8c0f-474f173ea50e>
2016-09-26T15:40:11Z
216.92.13.127
sha1:TWMFWZN6MYQNQ4BSPWQRJKNQMPL4E3LE
<urn:uuid:2922d05c-f6d9-4818-b4fd-abadbc350b73>
http://www.niemanlab.org/2012/07/that-plan-to-archive-every-tweet-in-the-library-of-congress-definitely-still-happening/
length
response
<urn:uuid:67ca266b-1d08-4dc4-9532-1416bd5b4823>
robots: classic
hostname: ip-10-143-35-109.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2016-40
operator: CommonCrawl Admin
description: Wide crawl of the web for September 2016
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf