Quality Assessment in Crowdsourced Indigenous Language Transcription

Munyaradzi, Ngoni and Suleman, Hussein (2013) Quality Assessment in Crowdsourced Indigenous Language Transcription, Proceedings of Theory and Practice of Digital Libraries (TPDL) 2013, 22-26 September 2013, Malta, Springer.

PDF tpdl_2013_paper104.pdf Download (274kB)

Abstract

The digital Bleek and Lloyd Collection is a rare collection that contains artwork, notebooks and dictionaries of the indigenous people of Southern Africa. The notebooks, in particular, contain stories that encode the language, culture and beliefs of these people, handwritten in now-extinct languages with a specialised notation system. Previous attempts have been made to convert the approximately 20000 pages of text to a machine-readable form using machine learning algorithms but, due to the complexity of the text, the recognition accuracy was low. In this paper, a crowdsourcing method is proposed to transcribe the manuscripts, where non-expert volunteers transcribe pages of the notebooks using an online tool. Experiments were conducted to determine the quality and consistency of transcriptions. The results show that volunteeers are able to produce reliable transcriptions of high quality. The inter-transcriber agreement is 80% for |Xam text and 95% for English text. When the |Xam text transcriptions produced by the volunteers are compared with a gold standard, the volunteers achieve an average accuracy of 64.75%, which exceeded that in previous work. Finally, the degree of transcription agreement correlates with the degree of transcription accuracy. This suggests that the quality of unseen data can be assessed based on the degree of agreement among transcribers.

Item Type:	Conference paper
Uncontrolled Keywords:	crowdsourcing, transcription, cultural heritage
Subjects:	Information systems > Information systems applications Information systems > Data management systems Information systems > Data management systems > Database design and models Human-centered computing Information systems > Information retrieval
Date Deposited:	28 Oct 2013
Last Modified:	10 Oct 2019 15:33
URI:	http://pubs.cs.uct.ac.za/id/eprint/909

Actions (login required)

View Item