We gratefully acknowledge support from
the Simons Foundation
and member institutions
Full-text links:

Download:

Current browse context:

cs.DL

Change to browse by:

cs

References & Citations

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Digital Libraries

Title: Routing Memento Requests Using Binary Classifiers

Abstract: The Memento protocol provides a uniform approach to query individual web archives. Soon after its emergence, Memento Aggregator infrastructure was introduced that supports querying across multiple archives simultaneously. An Aggregator generates a response by issuing the respective Memento request against each of the distributed archives it covers. As the number of archives grows, it becomes increasingly challenging to deliver aggregate responses while keeping response times and computational costs under control. Ad-hoc heuristic approaches have been introduced to address this challenge and research has been conducted aimed at optimizing query routing based on archive profiles. In this paper, we explore the use of binary, archive-specific classifiers generated on the basis of the content cached by an Aggregator, to determine whether or not to query an archive for a given URI. Our results turn out to be readily applicable and can help to significantly decrease both the number of requests and the overall response times without compromising on recall. We find, among others, that classifiers can reduce the average number of requests by 77% compared to a brute force approach on all archives, and the overall response time by 42% while maintaining a recall of 0.847.
Comments: 10 pages, 6 figures, 11 tables, accepted to be published at JCDL 2016
Subjects: Digital Libraries (cs.DL)
ACM classes: H.3.7; H.3.5
DOI: 10.1145/2910896.2910899
Cite as: arXiv:1606.09136 [cs.DL]
  (or arXiv:1606.09136v1 [cs.DL] for this version)

Submission history

From: Herbert Van De Sompel [view email]
[v1] Wed, 29 Jun 2016 14:54:43 GMT (1163kb,D)
https://arxiv.org/abs/1606.09136
arxiv.org
crawl=CC-MAIN-2018-43
13640
application/http; msgtype=response
sha1:HTOGOXFJDPRBQMP4G2XO6N5WB5UEFATW
<urn:uuid:63a6b33c-635b-4604-9963-37063d4e1f91>
2018-10-22T06:30:23Z
128.84.21.199
application/xhtml+xml
sha1:JSAQOXONUORUD4IA4UYIIUIYIIEOW5EY
<urn:uuid:35a31c87-1440-4e49-8957-c6b312c1169c>
https://arxiv.org/abs/1606.09136
response
<urn:uuid:552a9866-f23b-4a1a-b706-15996fb065a7>
isPartOf: CC-MAIN-2018-43
publisher: Common Crawl
description: Wide crawl of the web for October 2018
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-79-243-28.ec2.internal
software: Apache Nutch 1.15 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 0.11-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: http://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/