User:Trumad

From Archiveteam
Jump to: navigation, search

I'm Thanael. It's high time I joined this wiki.

Here's a tip:

Steps for enumerating (sub)domains:

  • Visit Rapid7's Sonar project
  • Download the large dataset ending with "fdns_any.json.gz"
    • This can be done in linux/mac terminal with wget -c LINK_TO_FILE
  • Use zgrep in terminal to find subdomains related to the domain you're interested in
    • If you're looking for subdomains.blogspot.com, for example, use this command on the file:
    • zgrep .blogspot.com R7Date_fdns_any.json.gz > blogspot.txt
  • Wait quite a while until zgrep finishes, and then let's get to archiving all the subdomains!
https://archiveteam.org/index.php?title=User:Trumad&printable=yes
archiveteam.org
crawl=CC-MAIN-2020-50
14564
application/http; msgtype=response
sha1:WHTOO5PSYBYKDDBFUKFBPC2WJSFDCZQV
<urn:uuid:4e88e546-fa95-4663-8238-5929e46251ab>
2020-11-25T17:09:54Z
204.109.60.27
text/html
sha1:LJRDSMZEOCPJBJBQJFBIMYBRIAI5YSXU
<urn:uuid:ba99f531-4ce5-451a-a77d-6fa0416a59dd>
https://archiveteam.org/index.php?title=User:Trumad&printable=yes
response
<urn:uuid:15189109-3b49-42d0-85d5-2b87bcf71ca7>
isPartOf: CC-MAIN-2020-50
publisher: Common Crawl
description: Wide crawl of the web for November/December 2020
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-240.ec2.internal
software: Apache Nutch 1.17 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.2-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/