[ZWeb] link cataloguing policy

ethan mindlace fremen mindlace@digicool.com
Fri, 13 Oct 2000 06:18:48 -0400


Zopatistas,

I've been getting a lot of requests to catalog links that are not
directly related to zope and not zope powered, like people's homepages
(not zope powered) and places like Andover.net (not zope powered, or
about zope).

Should I catalog these kind of links?

~ethan
https://mail.zope.org/pipermail/zope-web/2000-October/000177.html
mail.zope.org
crawl=CC-MAIN-2019-09
2437
application/http; msgtype=response
sha1:Q77QXMLJ4A5FT66FSQ3DBZLTU4RBFVPJ
<urn:uuid:33e4a3ba-7630-42f5-8320-8c5dea49c5ff>
2019-02-17T11:40:13Z
46.4.103.124
text/html
sha1:PI7M5VI7I5YGHI6VUMEVAXYK3CC4FLQ7
<urn:uuid:9facf4fa-84ca-49da-b717-beec788c5bca>
https://mail.zope.org/pipermail/zope-web/2000-October/000177.html
response
<urn:uuid:9563a869-c8f6-4af6-b311-c968616477ca>
isPartOf: CC-MAIN-2019-09
publisher: Common Crawl
description: Wide crawl of the web for February 2019
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-148-30-93.ec2.internal
software: Apache Nutch 1.15 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 0.11-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/