MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages

Lombard, Anri and Aina, Temi and Wolff, Ethan and Norvick, Elan and Gumede, Sbonelo and Mawere, Simbarashe and Meyer, Francois and Buys, Jan (2026) MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages, Proceedings of The Fifteenth Language Resources and Evaluation Conference (LREC 2026), 11-16 May 2026, Palma, Mallorca, Spain, 1394-1408, European Language Resources Association (ELRA).

Text 2026.lrec2026-1.109.pdf Download (335kB)

Abstract

Decoder-only language models can be adapted to diverse tasks through instruction finetuning, but the extent to which this generalizes at small scale for low-resource languages remains unclear. We focus on the languages of South Africa, where we are not aware of a publicly available decoder-only model that explicitly targets all eleven official written languages, nine of which are low-resource. We introduce MzansiText, a curated multilingual pretraining corpus with a reproducible filtering pipeline, and MzansiLM, a 125M-parameter language model trained from scratch. We evaluate MzansiLM on natural language understanding and generation using three adaptation regimes: monolingual task-specific finetuning, multilingual task-specific finetuning, and general multi-task instruction finetuning. Monolingual task-specific finetuning achieves strong performance on data-to-text generation, reaching 20.65 BLEU on isiXhosa and competing with encoder-decoder baselines over ten times larger. Multilingual task-specific finetuning benefits closely related languages on topic classification, achieving 78.5% macro-F1 on isiXhosa news classification. While MzansiLM adapts effectively to supervised NLU and NLG tasks, few-shot reasoning remains challenging at this model size, with performance near chance even for much larger decoder-only models. We release MzansiText and MzansiLM to provide a reproducible decoder-only baseline and clear guidance on adaptation strategies for South African languages at small scale.

Item Type:	Conference paper
Subjects:	Computing methodologies > Artificial intelligence > Natural language processing
Date Deposited:	03 Sep 2026 07:13
Last Modified:	03 Sep 2026 07:13
URI:	https://pubs.cs.uct.ac.za/id/eprint/1795

Actions (login required)

View Item