The Stanford WebBase Project Architecture
Wall Script
Follow Me:
Friday, October 10, 2008

The Stanford WebBase Project Architecture

The Stanford WebBase project has been collecting topic focused snapshots of Web sites. All the resulting archives are available to the public via fast download streams. For example, we collected pages from 350 sites every day for several weeks after the Katrina hurricane disaster. We also collect pages from government Web sites on a regular basis.

In addition, the project examines how our archives can be explored by historians, sociologists, and public policy professionals.



WebBase was originally funded by the Digital Library Initiatives I and II. During that time the focus of the project was crawling, indexing, clustering and searching technology. The current Google company spun out into the commercial sphere during this phase.



The focus on providing generally accessible, topically coherent archive snapshots was later supported by NSF Infrastructure grant EIA-0322975. An exploratory effort to learn from political scientists about their needs for tools that would allow them to analyze large Web archives was supported by NSF grant IIS-0624725. The project collaborates with the California Digital Library (CDL), and with Stanford's Communications Department.

For a list of available content, please see our access instructions page. A Web page for initiating downloads is also available.
Was this article helpful?
Thanks! Your feedback helps us to improve 9lessons.info


0 comments:

Make in India
http://www.9lessons.info/2008/10/stanford-webbase-project-architecture.html
www.9lessons.info
crawl=CC-MAIN-2017-26
82785
application/http; msgtype=response
sha1:C6JFC3EXGXBHDODHACIE2PNWLFOMN4UT
<urn:uuid:730b0b97-fd4f-483e-bcfa-cbf5140e95d8>
2017-06-24T14:05:39Z
172.217.7.147
application/xhtml+xml
sha1:SMDKXXYM5HDDGMP72ACAWSWO3XZQIX3G
<urn:uuid:66a0fd23-ab41-443c-a1e2-0b15b28a9e23>
http://www.9lessons.info/2008/10/stanford-webbase-project-architecture.html
response
<urn:uuid:ae9a2b66-f9f2-4b44-aed9-97704a33c8f8>
robots: classic
hostname: ip-10-145-172-11.ec2.internal
software: Nutch 1.6 (CC)
isPartOf: CC-MAIN-2017-26
operator: Common Crawl Admin
description: Wide crawl of the web for June 2017
publisher: Common Crawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf