archive.org tools and stuff

http://machawk1.github.io/wail/

Upload a site to archive.org

http://archiveteam.org/index.php?title=Internet_Archive#Uploading_to_archive.org

1.crawl save each webpage seperately

 wget http://web.archive.org/save/

2.https://github.com/n0tan3rd/wail

make warc file and upload to archive.org

http://archiveteam.org/index.php?title=Frequently_Asked_Questions

I uploaded a WARC file but why doesn’t it show up in Wayback Machine?

To ensure content integrity, items with WARC files must have the mediatype set to “web” and be under the Archive Team collection in order for it to be ingested by the Wayback Machine.

 


 

Locally browsing with warc file –

https://github.com/webrecorder/webrecorderplayer-electron

 

Tools 

https://webrecorder.io/

http://rhizome.org/software/

https://ws-dl.cs.odu.edu/Main/Software

http://archive.org/help/abouts3.txt – s3 api key -> http://archive.org/account/s3.php


 

Download a site/page from archive.org for given time

modify and use this script -> https://gist.github.com/lazanet/872f88c9874e4a7a78fd

 

Bulk Downloading Collections from Archive.org

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.
https://kanihal.wordpress.com/2017/08/20/archive-org-tools-and-stuff/
kanihal.wordpress.com
crawl=CC-MAIN-2019-43
54556
application/http; msgtype=response
sha1:LNSKS7DKFK6BM5R5K3DMDZIDG53OQUIY
<urn:uuid:be40e7c9-cea8-4825-b5fe-822b68614bd8>
2019-10-22T14:37:21Z
192.0.78.13
text/html
sha1:O5URG7QTHIMGR33UIGXIQ5C5FJ5NCEMX
<urn:uuid:ac235be2-3b5d-4d8d-b298-36430beed275>
https://kanihal.wordpress.com/2017/08/20/archive-org-tools-and-stuff/
response
<urn:uuid:333aab71-5740-4897-869f-f47a0ba7f55d>
isPartOf: CC-MAIN-2019-43
publisher: Common Crawl
description: Wide crawl of the web for October 2019
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-166.ec2.internal
software: Apache Nutch 1.16 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.1-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/