HackerPapers

arXiv:2106.01345cs.LG

Decision Transformer: Reinforcement Learning via Sequence Modeling

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, Igor Mordatch

Abstract

We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked Transformer. By conditioning an autoregressive model on the desired return (reward), past states, and actions, our Decision Transformer model can generate future actions that achieve the desired return. Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

arXiv comments
First two authors contributed equally. Last two authors advised equally
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.AIArtificial Intelligence

On Hacker News

Linked by 6 identifiable commenters in 7 comments across 7 threads, from Jul 2021 to Oct 2025. Found in 2 HN submissions from Jun 2021 to Nov 2022. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. Implicit Actor Critic Coupling via a Supervised Learning Framework for RLVR 1 comment

    • > By treating the outcome reward as a predictable label, we reformulate the RLVR problem into a supervised learning task over a score function parameterized by the policy model and optimized using cross-entropy loss. Isn't this how the Decision Transformer works? I don't see it…

      radarsat1View on HNParent context

  2. Q-learning is not yet scalable 1 comment

    • No mention of Decision Transformers or Trajectory Transformers? Both are offline approaches that tend to do very well at long-horizon tasks, as they bypass the credit assignment problem by virtue of having an attention mechanism. Most RL researchers consider these approaches not…

      isaacimagineView on HNParent context

  3. RLHF is just barely RL 1 comment

    • > if you include the desired score in your prompt, the model will now strive to produce an answer that is consistent with that score But you need a model to generate score from answer, and then fine-tune another model to generate answer conditioned on score. The first time the…

      visargaView on HNParent context

  4. MemGPT: Towards LLMs as Operating Systems 1 comment

    • >they've solved reinforcement learning? Transformers can do Reinforcement Learning yes. https://arxiv.org/abs/2106.01345 https://arxiv.org/abs/2205.14953 >they can handle continuous domains, like robot motion? Yes they can handle it just fine. Excellently in fact.…

      famouswafflesView on HNParent context

  5. The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games 1 comment

    • PPO is awesome, but so is GPT-style reward-trajectory prediction! http://arxiv.org/pdf/2106.01345v1. As a RL hobbyist, I'd love to see some sort of hybrid approach. Thoughts?

      isaacimagineView on HNParent context

  6. Deep reinforcement learning is a waste of time (2019) 1 comment

    • The new hotness this month just says 'screw it; treat RL as a supervised sequence problem with causal transformers.' https://arxiv.org/abs/2106.01345 'We consider the following shift in paradigm: instead of training a policy through conventional RL algorithms like temporal…

      sdenton4View on HNParent context

  7. Reinforcement Learning As One Big Sequence Modeling Problem 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.