Conversational agents for simulation applications and video games

Natural language processing (NLP) applications are becoming increasingly popular today, largely due to recent advances in theory (machine learning and knowledge representation) and the computational power required to train and store large language models and data. Since NLP applications such as Alexa, Google Assistant, Cortana, Siri, and chatGPT are widely used today, we assume that video games and simulation applications can successfully integrate NLP components into various use cases. The main goal of this paper is to show that natural language processing solutions can be used to improve user experience and make simulation more enjoyable. In this paper, we propose a set of methods along with a proven implemented framework that uses a hierarchical NLP model to create virtual characters (visible or invisible) in the environment that respond to and collaborate with the user to improve their experience. Our motivation stems from the observation that in many situations, feedback from a human user during the simulation can be used efficiently to help the user solve puzzles in real time, make suggestions, and adjust things like difficulty or even performance-related settings. Our implementation is open source, reusable, and built as a plugin in a publicly available game engine, the Unreal Engine. Our evaluation and demos, as well as feedback from industry partners, suggest that the proposed methods could be useful to the game development industry.