Um grupo de investigadores olham para um bastidor de servidores

CDXJ index files are available to support bulk access

Last updated on May 5th, 2023 at 01:39 pm

The research and education community has been requesting to support the bulk download of web-archived data and index files (CDXJ), for instance, to feed AI training models, optimize routing of web archive requests or recover information from selected websites (e.g. news).

Arquivo.pt begun making all its CDXJ index files publicly available in real-time to facilitate the bulk download of web-archived data. Learn how at:

Your feedback with comments or suggestions is most welcome to improve this service!

Please disseminate this information among potentially interested parties.
https://sobre.arquivo.pt/en/cdxj-index-files-are-available-to-suppport-bulk-access/
sobre.arquivo.pt
CC-MAIN-2024-10
85675
application/http; msgtype=response
sha1:H3E2ZYREIL36JQBCDESZIONVCK55JHUZ
<urn:uuid:be202273-1e34-4a3e-a4da-d4f13a0a2bc0>
2024-02-29T22:38:57Z
194.210.235.11
text/html
sha1:DRIW2OHQCKZJX5VXAILIBQTRAGY7HLXG
<urn:uuid:aaa8b89b-f475-43b8-96c2-da47b8d75a4f>
https://sobre.arquivo.pt/en/cdxj-index-files-are-available-to-suppport-bulk-access/
response
<urn:uuid:e0baab37-9481-439f-8b0a-4634e0ceefe2>
isPartOf: CC-MAIN-2024-10
publisher: Common Crawl
description: Wide crawl of the web for February/March 2024
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-208
software: Apache Nutch 1.19 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.5-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/