Just like how animals learn to adapt to different environments, scientists have developed a model that allows monkeys to learn a complex target search task. This task involves the agent gazing at one of four light spots, with two neighboring spots alternating as the correct target. The model uses a dynamic state space and a ‘history-in-episode architecture’ to make decisions based on past actions and results. In comparison to the conventional SARSA method, the model with the dynamic state space performed much better and achieved close to the theoretical optimum. This breakthrough in reinforcement learning paves the way for highly adaptable learning systems in complex environments. To dive deeper into this fascinating research, check out the full article!

Learning is a crucial basis for biological systems to adapt to environments. Environments include various states or episodes, and episode-dependent learning is essential in adaptation to such complex situations. Here, we developed a model for learning a two-target search task used in primate physiological experiments. In the task, the agent is required to gaze one of the four presented light spots. Two neighboring spots are served as the correct target alternately, and the correct target pair is switched after a certain number of consecutive successes. In order for the agent to obtain rewards with a high probability, it is necessary to make decisions based on the actions and results of the previous two trials. Our previous work achieved this by using a dynamic state space. However, to learn a task that includes events such as fixation to the initial central spot, the model framework should be extended. For this purpose, here we propose a “history-in-episode architecture.” Specifically, we divide states into episodes and histories, and actions are selected based on the histories within each episode. When we compared the proposed model including the dynamic state space with the conventional SARSA method in the two-target search task, the former performed close to the theoretical optimum, while the latter never achieved target-pair switch because it had to re-learn each correct target each time. The reinforcement learning model including the proposed history-in-episode architecture and dynamic state scape enables episode-dependent learning and provides a basis for highly adaptable learning systems to complex environments.

Read Full Article (External Site)