An evaluation of template and ML-based generation of user-readable text from a knowledge graph

Mahlaza, Z and Keet, CM and Dunn, J and Poulter, M (2021) An evaluation of template and ML-based generation of user-readable text from a knowledge graph, arXiv:2106.14613v1, Computer Science, UCT.

[thumbnail of 2106.14613.pdf] Text
2106.14613.pdf

Download (1MB)

Abstract

Typical user-friendly renderings of knowledge graphs are visualisations and natural language text. Within the latter HCI solution approach, data-driven natural language generation systems receive increased attention, but they are often outperformed by template-based systems due to su ering from errors such as content dropping, hallucination, or repetition. It is unknown which of those errors are associated signi cantly with low quality judgements by humans who the text is aimed for, which hampers addressing errors based on their impact on improving human evaluations. We assessed their possible association with an experiment availing of expert and crowdsourced evaluations of human authored text, template generated text, and sequence-to-sequence model generated text. The results showed that there was no significant association between human authored texts with errors and the low human judgements of naturalness and quality. There was also no significant association between machine learning generated texts with dropped or hallucinated slots and the low human judgements of naturalness and quality. Thus, both approaches appear to be viable options for designing a natural language interface for knowledge graphs

Item Type: Technical report
Subjects: Computing methodologies > Artificial intelligence > Natural language processing > Natural language generation
Alternate Locations: https://arxiv.org/abs/2106.14613
Date Deposited: 03 Dec 2021 11:01
Last Modified: 03 Dec 2021 11:01
URI: https://pubs.cs.uct.ac.za/id/eprint/1478

Actions (login required)

View Item View Item