Page MenuHomePhabricator

Ability to create pages per Wikipedia Library partner
Closed, ResolvedPublic

Description

It is useful to be able to track multiple URL patterns for any given partner. Reasons vary, but for one partner a particular link pattern is being used for Wikipedia library editors specifically, and for others it's simply for tracking different collections.

If we want an interface that we can send to partners to give them an overview of link additions on Wikipedia, it should be capable of showing each kind of URL pattern and the relevant changes/lists over time.
https://phabricator.wikimedia.org/T152320
phabricator.wikimedia.org
crawl=CC-MAIN-2020-45
55140
application/http; msgtype=response
sha1:77PYSPB2ZWSHZEEF2HHCEEW3G4QXCMGI
<urn:uuid:25071fc1-bee7-4e03-898b-be392fd0f4fd>
2020-10-23T13:00:28Z
208.80.154.224
text/html
sha1:JSD4G26I76W6KFFQUN3FLTOV32NLEWXE
<urn:uuid:e7b35786-d9f8-4881-9792-c93aa2dd7e62>
https://phabricator.wikimedia.org/T152320
response
<urn:uuid:97ec3c92-4ca7-4aae-a602-3864709f6066>
isPartOf: CC-MAIN-2020-45
publisher: Common Crawl
description: Wide crawl of the web for October 2020
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-141.ec2.internal
software: Apache Nutch 1.17 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.2-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/