Parallel methods for the update of partitioned inverted files

Abstract

Purpose

An issue that tends to be ignored in information retrieval is the issue of updating inverted files. This is largely because inverted files were devised to provide fast query service, and much work has been done with the emphasis strongly on queries. This paper aims to study the effect of using parallel methods for the update of inverted files in order to reduce costs, by looking at two types of partitioning for inverted files: document identifier and term identifier.

Design/methodology/approach

Raw update service and update with query service are studied with these partitioning schemes using an incremental update strategy. The paper uses standard measures used in parallel computing such as speedup to examine the computing results and also the costs of reorganising indexes while servicing transactions.

Findings

Empirical results show that for both transaction processing and index reorganisation the document identifier method is superior. However, there is evidence that the term identifier partitioning method could be useful in a concurrent transaction processing context.

Practical implications

There is an increasing need to service updates, which is now becoming a requirement of inverted files (for dynamic collections such as the web), demonstrating that a shift in requirements of inverted file maintenance is needed from the past.

Originality/value

The paper is of value to database administrators who manage large‐scale and dynamic text collections, and who need to use parallel computing to implement their text retrieval services.

Citation

MacFarlane, A., McCann, J.A. and Robertson, S.E. (2007), "Parallel methods for the update of partitioned inverted files", Aslib Proceedings, Vol. 59 No. 4/5, pp. 367-396. https://doi.org/10.1108/00012530710817582