Suleman, Hussein (2006) Parallelising Harvesting, Proceedings of 9th International Conference on the Asian Digital Library, 27-30 November 2006, Kyoto, Japan.
![[img]](http://pubs.cs.uct.ac.za/style/images/fileicons/application_pdf.png) |
PDF
icadl_2006_oaihpc.pdf Download (261kB) |
Abstract
Metadata harvesting has become a common technique to transfer a stream of data from one metadata repository or digital library system to another. As collections of metadata, and their associated digital objects, grow in size, the ingest of these items at the destination archive can take a significant amount of time, depending on the type of indexing or post-processing that is required. This paper discusses an approach to parallelise the post-processing of data in a small cluster of machines or a multi-processor environment, while not increasing the burden on the source data provider. Performance tests have been carried out on varying architectures and the results indicate that this technique is indeed promising for some scenarios and can be extended to more computationally-intensive ingest procedures. In general, the technique presents a new approach for the construction of harvest-based distributed or component-based digital libraries, with better scalability than before.
| Item Type: | Conference paper |
|---|---|
| Subjects: | Computer systems organization > Dependable and fault-tolerant systems and networks Computer systems organization > Architectures > Distributed architectures |
| Date Deposited: | 07 Nov 2006 |
| Last Modified: | 10 Oct 2019 15:35 |
| URI: | http://pubs.cs.uct.ac.za/id/eprint/328 |
Actions (login required)
 |
View Item |