Policy-based Reinforcement Learning for Generalisation in Interactive Text-based Environments

Toledo, Edan and Buys, Jan and Shock, Jonathan (2023) Policy-based Reinforcement Learning for Generalisation in Interactive Text-based Environments, Proceedings of 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2023), 2-6 May 2023, Dubrovnik, Croatia, 1230-1242, Association for Computational Linguistics.

Text 2023.eacl-main.88.pdf - Published Version Download (1MB)

Abstract

Text-based environments enable RL agents to learn to converse and perform interactive tasks through natural language. However, previous RL approaches applied to text-based environments show poor performance when evaluated on unseen games. This paper investigates the improvement of generalisation performance through the simple switch from a value-based update method to a policy-based one, within text-based environments. We show that by replacing commonly used value-based methods with REINFORCE with baseline, a far more general agent is produced. The policy-based agent is evaluated on Coin Collector and Question Answering with interactive text (QAit), two text-based environments designed to test zero-shot performance. We see substantial improvements on a variety of zero-shot evaluation experiments, including tripling accuracy on various QAit benchmark configurations. The results indicate that policy-based RL has significantly better generalisation capabilities than value-based methods within such text-based environments, suggesting that RL agents could be applied to more complex natural language environments.

Item Type:	Conference paper
Subjects:	Computing methodologies > Artificial intelligence > Natural language processing Computing methodologies > Machine learning > Learning paradigms > Reinforcement learning
Date Deposited:	10 Nov 2023 14:49
Last Modified:	10 Nov 2023 14:49
URI:	https://pubs.cs.uct.ac.za/id/eprint/1635

Actions (login required)

View Item