City Research Online

Parallel methods for the update of partitioned inverted files

MacFarlane, A., McCann, J. A. & Robertson, S. E. (2007). Parallel methods for the update of partitioned inverted files. Aslib Proceedings; New Information Perspectives, 59(4-5), pp. 367-396. doi: 10.1108/00012530710817582

Abstract

Purpose – An issue which tends to be ignored in information retrieval is the issue of updating inverted files. This is largely because inverted files were devised to provide fast query service, and much work has been done with the emphasis strongly on queries. In this paper we study the effect of using parallel methods for the update of inverted files in order to reduce costs, by looking at two types of partitioning for inverted files: document identifier and term identifier.

Design/methodology/approach – Raw update service and update with query service are studied with these partitioning schemes using an incremental update strategy. We use standard measures used in parallel computing such as speedup to examine the computing results and also the costs of reorganising indexes while servicing transactions.

Findings – Empirical results show that for both transaction processing and index reorganisation the document identifier method is superior. However, there is evidence that the term identifier partitioning method could be useful in a concurrent transaction processing context.

Practical implications – There is an increasing need to service updates which is now becoming a requirement of inverted files (for dynamic collections such as the Web), demonstrating that a shift in requirements of inverted file maintenance is needed from the past.

Originality/value – The paper is of value to database administrators who manage large-scale and dynamic text collections, and who need to use parallel computing to implement their text retrieval services.

Publication Type: Article
Additional Information: This article is (c) Emerald Group Publishing and permission has been granted for this version to appear here http://openaccess.city.ac.uk/. Emerald does not grant permission for this article to be further copied/distributed or hosted elsewhere without the express permission from Emerald Group Publishing Limited.
Publisher Keywords: Information retrieval, Parallel programming, Query languages, Databases
Subjects: Z Bibliography. Library Science. Information Resources > Z665 Library Science. Information Science
Departments: School of Mathematics, Computer Science & Engineering > Computer Science > Human Computer Interaction Design
Related URLs:
[img]
Preview
PDF - Accepted Version
Download (327kB) | Preview

Export

Downloads

Downloads per month over past year

View more statistics

Actions (login required)

Admin Login Admin Login
https://openaccess.city.ac.uk/id/eprint/4456/
openaccess.city.ac.uk
crawl=CC-MAIN-2022-21
39412
application/http; msgtype=response
sha1:HM75T3GCQG7JRBK2F6C7KDKEQQOAKM2U
<urn:uuid:a1919c8e-1d37-433f-928d-a75b1eeae0b6>
2022-05-16T18:32:18Z
46.22.140.49
application/xhtml+xml
sha1:OTJIA4DUAXMH3GHAL2BNNQRON27ZAMDV
<urn:uuid:f2c49c1e-30c9-4873-bca1-2eabddb1705a>
https://openaccess.city.ac.uk/id/eprint/4456/
response
<urn:uuid:58d25e56-155c-422b-a1dc-fa18a1874006>
isPartOf: CC-MAIN-2022-21
publisher: Common Crawl
description: Wide crawl of the web for May 2022
operator: Common Crawl Admin (info@commoncrawl.org)
hostname: ip-10-67-67-28
software: Apache Nutch 1.18 (modified, https://github.com/commoncrawl/nutch/)
robots: checked via crawler-commons 1.3-SNAPSHOT (https://github.com/crawler-commons/crawler-commons)
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/