Category Posts Navigation

Common Crawl – Open Repository of Web Crawl Data Composed Of Over 5 Billion Freely Available Web Pages

Posted by Marcus Zillman

Common Crawl – Open Repository of Web Crawl Data Composed Of Over 5 Billion Freely Available Web Pages
http://www.CommonCrawl.org/

The Common Crawl Foundation is a California 501(c)(3) registered non-profit founded by Gil Elbaz with the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable. Their vision is of a truly open web that allows open access to information and enables greater innovation in research, business and education. They level the playing field by making wholesale extraction, transformation and analysis of web data cheap and easy. This will be added to Statistics Resources and Big Data Subject Tracer™. This will be added to Bot Research Subject Tracer™. This has been added to the tools section of Research Resources Subject Tracer™ Information Blog.

Leave a Reply

Facebook Comments

Browse Categories

AwarenessWatch Newsletter
https://www.zillman.us/common-crawl-open-repository-of-web-crawl-data-composed-of-over-5-billion-freely-available-web-pages/
www.zillman.us
crawl=CC-MAIN-2022-27
76178
application/http; msgtype=response
sha1:BT3IAA2566OOR4GFSB7HTO6P6JIQIZDZ
<urn:uuid:ae2da205-ee28-4439-94c4-1ebdf176354d>
2022-06-29T09:50:22Z
172.67.136.181
text/html
sha1:CV3SDHK6ARO66QRTCICI7UQA6JHZ52UC
<urn:uuid:17c86212-8678-4094-adc3-ec5cb58f8111>
https://www.zillman.us/common-crawl-open-repository-of-web-crawl-data-composed-of-over-5-billion-freely-available-web-pages/
response
<urn:uuid:90956c89-6444-4100-b3c4-24c598264e2c>
isPartOf: CC-MAIN-2022-27
publisher: Common Crawl
description: Wide crawl of the web for June/July 2022
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-87
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.3-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/