UCT CS Research Document Archive

Parallelising Harvesting

Suleman, Hussein (2006) Parallelising Harvesting. In Proceedings 9th International Conference on the Asian Digital Library, Kyoto, Japan.

Full text available as:
PDF - Requires Adobe Acrobat Reader or other PDF viewer.


Metadata harvesting has become a common technique to transfer a stream of data from one metadata repository or digital library system to another. As collections of metadata, and their associated digital objects, grow in size, the ingest of these items at the destination archive can take a significant amount of time, depending on the type of indexing or post-processing that is required. This paper discusses an approach to parallelise the post-processing of data in a small cluster of machines or a multi-processor environment, while not increasing the burden on the source data provider. Performance tests have been carried out on varying architectures and the results indicate that this technique is indeed promising for some scenarios and can be extended to more computationally-intensive ingest procedures. In general, the technique presents a new approach for the construction of harvest-based distributed or component-based digital libraries, with better scalability than before.

EPrint Type:Conference Paper
Subjects:C Computer Systems Organization: C.4 PERFORMANCE OF SYSTEMS
C Computer Systems Organization: C.2 COMPUTER-COMMUNICATION NETWORKS
ID Code:328
Deposited By:Suleman, Hussein
Deposited On:07 November 2006