Dynamic Role Allocation for Small Search Engine Clusters

Nakashole, Ndapandula and Suleman, Hussein and Pedzai, Calvin (2007) Dynamic Role Allocation for Small Search Engine Clusters, Proceedings of Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists (SAICSIT 2007), 2-3 October 2007, Port Alfred, South Africa, ACM.

PDF dynamic_role.pdf Download (368kB)

Abstract

Search engines facilitate efficient discovery of information in large information environments such as the Web. As the amount of information rapidly increases, search engines require greater computational resources. Similarly, as the user base increases search engines need to handle increasing numbers of user requests. Existing solutions to these scalability problems are often designed for large computer clusters. This paper presents a flexible solution that is deployable also on small clusters. The solution is based on the allocation and dynamic re-adjustment of indexing and querying roles to cluster nodes in order to optimize cluster utilisation. By allocating cluster machines to the job that requires the most computational power, indexing and querying may both realize performance gains, while neither overwhelms the limited resources available. A prototype system was built and tested on a small cluster using a dataset of over 100 000 Web pages from the uct.ac.za domain. Initial results confirm an improved system resource utilisation, which warrants further investigation

Item Type:	Conference paper
Uncontrolled Keywords:	indexing ,querying ,small search engine cluster,dynamic role allocation
Subjects:	Information systems > Information retrieval Computer systems organization > Dependable and fault-tolerant systems and networks
Date Deposited:	09 Oct 2007
Last Modified:	10 Oct 2019 15:35
URI:	http://pubs.cs.uct.ac.za/id/eprint/431

Actions (login required)

View Item