HackerPapers

arXiv:2012.06684cs.LG

Faster Policy Learning with Continuous-Time Gradients

Samuel Ainsworth, Kendall Lowrey, John Thickstun, Zaid Harchaoui, Siddhartha Srinivasa

Abstract

We study the estimation of policy gradients for continuous-time systems with known dynamics. By reframing policy learning in continuous-time, we show that it is possible construct a more efficient and accurate gradient estimator. The standard back-propagation through time estimator (BPTT) computes exact gradients for a crude discretization of the continuous-time system. In contrast, we approximate continuous-time gradients in the original system. With the explicit goal of estimating continuous-time gradients, we are able to discretize adaptively and construct a more efficient policy gradient estimator which we call the Continuous-Time Policy Gradient (CTPG). We show that replacing BPTT policy gradients with more efficient CTPG estimates results in faster and more robust learning in a variety of control tasks and simulators.

arXiv comments
Journal reference
L4DC 2021
DOI
Subjects
cs.LGMachine Learning stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 2 identifiable commenters in 3 comments across 2 threads, from Dec 2021 to Sep 2022.

Comments linking this paper, per year.

Discussed in comments2 threads

  1. Accelerate Python code by importing Taichi 1 comment

    • The only ones that I know of are in https://arxiv.org/abs/2012.06684 (with Julia DifferentialEquations.jl and DiffTaichi), but those are more algorithmic. There, Julia does extremely well, but the conclusion that I would draw from it is more: use a programming language with…

      ChrisRackauckasView on HNParent context

  2. Julia 1.7 Highlights 2 comments

    • >One of the domains where I hope Julia will excel is precisely as you comment on physics + ml (I check frequently if MuJoCo source code is available at last). Somebody already did a comparison of DifferentialEquations.jl's standard usage against MuJoCo and DiffTaichi and showed…

      ChrisRackauckasView on HNParent context

    • I manage the MuJoCo.jl wrapper, and having the source code wont (naively) help with AD. Internally, there's a number of iterative algorithms that you wouldn't want to automatically differentiate, and if analytical derivatives were apparent they would be in there (the MuJoCo…

      klowreyView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.