UCT CS Research Document Archive

Dynamic Role Allocation for Small Search Engine Clusters

Nakashole, Ndapandula, Hussein Suleman and Calvin Pedzai (2007) Dynamic Role Allocation for Small Search Engine Clusters. In Barnard, Lynette and Reinhard A Botha, Eds. Proceedings Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists (SAICSIT 2007), Port Alfred, South Africa.

Full text available as:
PDF - Requires Adobe Acrobat Reader or other PDF viewer.

Abstract

Search engines facilitate efficient discovery of information in large information environments such as the Web. As the amount of information rapidly increases, search engines require greater computational resources. Similarly, as the user base increases search engines need to handle increasing numbers of user requests. Existing solutions to these scalability problems are often designed for large computer clusters. This paper presents a flexible solution that is deployable also on small clusters. The solution is based on the allocation and dynamic re-adjustment of indexing and querying roles to cluster nodes in order to optimize cluster utilisation. By allocating cluster machines to the job that requires the most computational power, indexing and querying may both realize performance gains, while neither overwhelms the limited resources available. A prototype system was built and tested on a small cluster using a dataset of over 100 000 Web pages from the uct.ac.za domain. Initial results confirm an improved system resource utilisation, which warrants further investigation

EPrint Type:Conference Paper
Keywords:indexing ,querying ,small search engine cluster,dynamic role allocation
Subjects:H Information Systems: H.3 INFORMATION STORAGE AND RETRIEVAL
C Computer Systems Organization: C.4 PERFORMANCE OF SYSTEMS
ID Code:431
Deposited By:nakashole, ndapandula
Deposited On:09 October 2007