Leveraging Relative Position for Trajectory Learning in Text-based Games

Abstract

Text-based games (TBG) are complex environments that allow users or computer agents to make textual interactions and achieve game goals. The text-based gameplay experiences are comparable to smart computing agent task episodes, which are understanding user text input and achieving a target to get user satisfaction. Designing text-based game agents that autonomously learn from the games and reach the game goals is challenging. In order to understand and propose our way to tackle this challenge, we first need to systematically classify and analyze text-based game agent-related methods. Based on the encoder analysis and observations, we propose a new model to improve the text-based game agent's trajectory learning capability, allowing the agent to learn from its gameplay experience and improve performance.

In this thesis, we first review the related literature of text-based games and analyze associated methods. We implemented a standardized agent and performed ablation tests on selected encoder models, which allowed us to choose the best-performing encoder architecture. Next, with a Transformer-based model chosen as our base model, we propose a model that uses a GPT encoder to replace the widely adopted LSTM trajectory encoder to improve text-based game agent trajectory learning capability. To capture the state relationship of the text-based game environment, we extend the relative positional embedding used in traditional NLP tasks into state trajectory learning under the Reinforcement Learning framework. We analyze the critical differences between relative positional embedding used in traditional NLP tasks and reinforcement learning state observation trajectory learning. We propose our exploration trajectory embedding, which is the first relative positional embedding used in this field. Our experiment shows that our model, together with the newly introduced relative positional embedding, brings substantial improvement to text-based game trajectory learning.