[greenstone-devel] Greenstone Processing 1 out of 761 documents (was RE: Missing HTML documents)

From Gregory S. Williamson
DateTue, 16 Nov 2004 12:25:00 -0800
Subject [greenstone-devel] Greenstone Processing 1 out of 761 documents (was RE: Missing HTML documents)
Sorry for the report but this is a serious show stopper and I haven't heard anything since posting this 3 days ago ... any help would be welcome.

Thanks

-----Original Message-----
From:Gregory S. Williamson
Sent:Sat 11/13/2004 6:59 PM
To:greenstone-devel&#64;list.scms.waikato.ac.nz
Cc:
Subject:Missing HTML documents
Dear long-suffering developers & list lurkers,

I have recently built a collection from HTML documents (this is on a red hat linux using gsdl 2.40); the build summary shows 761 documents were considered for and processed into the collection.

When I look in the build log I see all of the HTML files being copied and then processed (with an occasional complaint telling me that the language could not be extracted and it has assumed "en" as the encoding, which is fine).

But only the last document processed by HTMLPlug seems to be showing; the subsequent steps in all import all show only some 4700 words (and some warnings for not having any data or very little data).

I don't see any warnings elsewhere save perhaps a clue in the apache warning log:
[Sat Nov 13 17:49:21 2004] [error] [client 127.0/0/1] (70007) The timeout specified has expired: ap

oops

The requested page could not be found. Please use your browsers 'back' button or the above home button to return to the Greenstone Digital Library.length_filter: apr_bucket_read() failed,
referrer: http://127.0.0.1/gsdl/cgi-bin/library

(the server is not connected to a network yet so this is a hand transcription; the time stamnp puts it about 2 minutes before then of the build process, so after most of the copying is done, I think)

Any suggestions as to what might be the wrong ? I sort of had my heart set on doing the other 760 files in the same collection ;-}

Thanks,

Greg Williamson
http://www.nzdl.org/gsdlmod?e=q-00000-00---off-0gsarch--00-0----0-10-0---0---0direct-10---4-----dfr--0-1l--11-en-50---20-preferences-s.yeates%26%2364%3Bcs.waikato.ac.nz--00-0-21-00-0--4----0-0-11-10-0utfZz-8-00&a=d&c=gsarch&srp=0&srn=0&cl=search&d=71E37EF6B7DCC1499CEA0316A256832801D4BB11-loki-wc-globexplorer-net
www.nzdl.org
crawl=CC-MAIN-2020-34
12201
application/http; msgtype=response
sha1:6QSFESG7WBOKUYX2BEWTWKNWDHLJE6CE
<urn:uuid:b9eaad90-ac36-4dd8-9141-cefbf69a834d>
2020-08-13T17:59:04Z
130.217.208.20
text/html
sha1:NRZHNSQHFMCYJ6YPISCBLL2NTPOPSCEQ
<urn:uuid:f3bb6db0-d827-47b2-8fbe-702da072c3dc>
http://www.nzdl.org/gsdlmod?e=q-00000-00---off-0gsarch--00-0----0-10-0---0---0direct-10---4-----dfr--0-1l--11-en-50---20-preferences-s.yeates%26%2364%3Bcs.waikato.ac.nz--00-0-21-00-0--4----0-0-11-10-0utfZz-8-00&a=d&c=gsarch&srp=0&srn=0&cl=search&d=71E37EF6B7DCC1499CEA0316A256832801D4BB11-loki-wc-globexplorer-net
response
<urn:uuid:b8bc7843-0278-466f-8c2b-c88b668df02b>
isPartOf: CC-MAIN-2020-34
publisher: Common Crawl
description: Wide crawl of the web for August 2020
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-20.ec2.internal
software: Apache Nutch 1.17 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.2-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/