View Single Post
Old May 2, 2012, 12:24 PM   #36
Mike Irwin
Staff
 
Join Date: April 13, 2000
Location: Northern Virginia
Posts: 36,912
"The discrepancies in those numbers are quite consistent, over time."

So, how much time are you observing?

The simple fact that they are consistent is a pretty solid indication that these are, in fact, regularly scheduled crawlers and not humans. Human patterns would tend to vary considerably.

They will target the forums that are most active.

At least that's how I understand them to work.
__________________
"The gift which I am sending you is called a dog, and is in fact the most precious and valuable possession of mankind" -Theodorus Gaza

Baby Jesus cries when the fat redneck doesn't have military-grade firepower.
Mike Irwin is offline  
 
Page generated in 0.04167 seconds with 7 queries
http://thefiringline.com/forums/showpost.php?p=5067742&postcount=36
thefiringline.com
crawl=CC-MAIN-2015-22
15172
application/http; msgtype=response
sha1:SZERR6WRUXIWFLA6FFY4M6LW2DKMM4OV
<urn:uuid:f596867f-36a2-4c95-91a2-bbca28cea29b>
2015-05-27T18:05:42Z
162.243.102.188
sha1:Z6WOEZLAJRXQTLBCF4OHDT7ADDRZZFIY
<urn:uuid:ffec4db0-0138-40c9-8ab5-d8caff07b458>
http://thefiringline.com/forums/showpost.php?p=5067742&postcount=36
response
<urn:uuid:b76eae40-a7b3-438b-b9fb-a164c2093b4a>
robots: classic
hostname: ip-10-180-206-219.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2015-22
operator: CommonCrawl Admin
description: Wide crawl of the web for May 2015
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf