Published April 1, 2020 | Version Accepted pre-print
Journal article Open

A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials

  • 1. Inria, CNRS, Universit´e de Lorraine, LORIA, F-54000 Nancy, France
  • 2. Inria, CNRS, Universit´e de Lorraine, LORIA, F-54000 Nancy, France and Research Centre on Interactive Media, Smart Systems and Emerging Technologies, Dimarcheio Lefkosias, Plateia Eleftherias, 1500, Nicosia, Cyprus
  • 3. German Aerospace Center (DLR), Institute of Robotics and Mechatronics, Wessling, Germany
  • 4. Idiap Research Institute, Rue Marconi 19, 1920 Martigny, Switzerland

Description

Most policy search algorithms require thousands of training episodes to find an effective policy, which is often infeasible with a physical robot. This survey article focuses on the extreme other end of the spectrum: how can a robot adapt with only a handful of trials (a dozen) and a few minutes? By analogy with the word “big-data”, we refer to this challenge as “microdata reinforcement learning”. We show that a first strategy is to leverage prior knowledge on the policy structure (e.g., dynamic movement primitives), on the policy parameters (e.g., demonstrations), or on the dynamics (e.g., simulators). A second strategy is to create data-driven surrogate models of the expected reward (e.g., Bayesian optimization) or the dynamical model (e.g., model-based policy search), so that the policy optimizer queries the model instead of the real system. Overall, all successful microdata algorithms combine these two strategies by varying the kind of model and prior knowledge. The current scientific challenges essentially revolve around scaling up to complex robots, designing generic priors, and optimizing the computing time.

Notes

This work has been partly supported by the project that has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 739578 (RISE – Call: H2020-WIDESPREAD-01-2016-2017-TeamingPhase2) and the Government of the Republic of Cyprus through the Directorate General for European Programmes, Coordination and Development.

Files

Chatzilygeroudis_et_al_TRO2019.pdf

Files (1.4 MB)

Name Size Download all
md5:14d080f6fb11fb6d3d31632dd1427b23
1.4 MB Preview Download

Additional details

Funding

MEMMO – Memory of Motion 780684
European Commission
ResiBots – Robots with animal-like resilience 637972
European Commission
An.Dy – Advancing Anticipatory Behaviors in Dyadic Human-Robot Collaboration 731540
European Commission
RISE – Research Center on Interactive Media, Smart System and Emerging Technologies 739578
European Commission