Publications

| Sorted by Date | Classified by Publication Type | Classified by Research Category |

Diverse Projection Ensembles for Distributional Reinforcement Learning

Moritz A. Zanger, Wendelin Böhmer, and Matthijs T. J. Spaan. Diverse Projection Ensembles for Distributional Reinforcement Learning. arXiv:2306.07124, 2023.

Download

pdf 

Abstract

In contrast to classical reinforcement learning, distributional reinforcement learning algorithms aim to learn the distribution of returns rather than their expected value. Since the nature of the return distribution is generally unknown a priori or arbitrarily complex, a common approach finds approximations within a set of representable, parametric distributions. Typically, this involves a projection of the unconstrained distribution onto the set of simplified distributions. We argue that this projection step entails a strong inductive bias when coupled with neural networks and gradient descent, thereby profoundly impacting the generalization behavior of learned models. In order to facilitate reliable uncertainty estimation through diversity, this work studies the combination of several different projections and representations in a distributional ensemble. We establish theoretical properties of such projection ensembles and derive an algorithm that uses ensemble disagreement, measured by the average 1-Wasserstein distance, as a bonus for deep exploration. We evaluate our algorithm on the behavior suite benchmark and find that diverse projection ensembles lead to significant performance improvements over existing methods on a wide variety of tasks with the most pronounced gains in directed exploration problems.

BibTeX Entry

@Misc{Zanger23pedqn,
  author =       {Moritz A. Zanger and Wendelin B{\"o}hmer and Matthijs T. J. Spaan},
  title =        {Diverse Projection Ensembles for Distributional Reinforcement Learning},
  howpublished = {arXiv:2306.07124},
  year =         2023
}

Note: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

Generated by bib2html.pl (written by Patrick Riley) on Thu Feb 29, 2024 16:15:45 UTC