Parallelising Harvesting

Suleman, Hussein (2006) Parallelising Harvesting, Proceedings of 9th International Conference on the Asian Digital Library, 27-30 November 2006, Kyoto, Japan.

[img] PDF

Download (261kB)


Metadata harvesting has become a common technique to transfer a stream of data from one metadata repository or digital library system to another. As collections of metadata, and their associated digital objects, grow in size, the ingest of these items at the destination archive can take a significant amount of time, depending on the type of indexing or post-processing that is required. This paper discusses an approach to parallelise the post-processing of data in a small cluster of machines or a multi-processor environment, while not increasing the burden on the source data provider. Performance tests have been carried out on varying architectures and the results indicate that this technique is indeed promising for some scenarios and can be extended to more computationally-intensive ingest procedures. In general, the technique presents a new approach for the construction of harvest-based distributed or component-based digital libraries, with better scalability than before.

Item Type: Conference paper
Subjects: Computer systems organization > Dependable and fault-tolerant systems and networks
Computer systems organization > Architectures > Distributed architectures
Date Deposited: 07 Nov 2006
Last Modified: 10 Oct 2019 15:35

Actions (login required)

View Item View Item