The Geometry of Low-Resource Language Representations

Meyer, Francois and Buys, Jan (2026) The Geometry of Low-Resource Language Representations, Proceedings of 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP 2026), 24-29 October, Budapest, Hungary, Association for Computational Linguistics.

Text 6623_The_Geometry_of_Low_Resou.pdf Download (426kB)

Abstract

The performance gap between low- and high-resource languages in LLMs is widely known, but it remains unclear which internal model factors drive these disparities. In this paper, we characterise this gap through the lens of representational geometry. Comparing the geometric properties of hidden representations across 30 languages reveals that LLM geometry is systematically related to language data availability. The most consistent effect is in final layers, where low-resource languages exhibit representational degeneration. To counter this, we investigate the effectiveness of regularisation terms to penalise degeneration during continued pretraining (CPT). Experiments monolingually adapting 9 base LLMs to 10 African languages show that geometric regularisation successfully reduces representational degeneration during CPT. For larger models, cosine similarity-based regularisation marginally improves performance over vanilla CPT, with more consistent gains on the most challenging tasks. We establish that the representational geometry of low- and high-resource languages in LLMs is measurably distinct, and that targeted geometric intervention is a viable strategy for improving CPT for low-resource languages.

Item Type:	Conference paper
Subjects:	Computing methodologies > Artificial intelligence > Natural language processing
Date Deposited:	03 Sep 2026 07:14
Last Modified:	03 Sep 2026 07:14
URI:	https://pubs.cs.uct.ac.za/id/eprint/1797

Actions (login required)

View Item