An assessment of orthographic similarity measures for several African languages

Keet, C. Maria (2016) An assessment of orthographic similarity measures for several African languages, arxiv 1608.03065, Department of Computer Science, University of Cape Town.

PDF 1608.03065v1.pdf Download (706kB)

Abstract

Natural Language Interfaces and tools such as spellcheckers and Web search in one's own language are known to be useful in ICT-mediated communication. Most languages in Southern Africa are under-resourced, however. Therefore, it would be very useful if both the generic and the few language-specific NLP tools could be reused or easily adapted across languages. This depends on the notion, and extent, of similarity between the languages. We assess this from the angle of orthography and corpora. Twelve versions of the Universal Declaration of Human Rights are examined, showing clusters of languages, and which are thus more or less amenable to cross-language adaptation of NLP tools, which do not match with Guthrie zones. To examine the generalisability of these results, we zoom in on isiZulu both quantitatively and qualitatively with four other corpora and texts in different genres. The results show that the UDHR is a typical text document orthographically. The results also provide insight into usability of typical measures such as lexical diversity and genre, and that the same statistic may mean different things in different documents. While NLTK for Python could be used for basic analyses of text, it, and similar NLP tools, will need considerable customization.

Item Type:	Technical report
Uncontrolled Keywords:	natural language processing
Subjects:	Applied computing > Document management and text processing
Alternate Locations:	http://www.meteck.org/files/1608.03065v1.pdf, http://arxiv.org/abs/1608.03065
Date Deposited:	18 Aug 2016
Last Modified:	10 Oct 2019 15:32
URI:	http://pubs.cs.uct.ac.za/id/eprint/1088

Actions (login required)

View Item