Do language models plan ahead for future tokens? · HackerPapers discussions

Do language models plan ahead for future tokens? — HN submission

Sun, 19 Jul 2026 17:01:47 GMT

Do Language Models Plan Ahead for Future Tokens? (2024)

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Do language models plan ahead for future tokens? — mentioned by x312

Tue, 14 Jul 2026 13:22:00 GMT

It's been known for several years that LLM activations encode future tokens ahead of time (e.g. https://arxiv.org/abs/2404.00859). But this has only been shown on simple tasks, so I think this paper is still quite neat. The interesting thing is that they show "future horizon length" varies across models.

Paper and discussion history. A mention is not an endorsement.

Do language models plan ahead for future tokens? — mentioned by elcomet

Wed, 01 May 2024 14:14:33 GMT

I think you're not looking at this from the right perspective. A LLM is designed to sample text, that follows the training distribution. It is not designed to tell you the "most likely" text that follows, and we don't actually want that. This would mean you have no diversity in your outputs. In your example, sampling a 0 in 40% of cases and a 1 in 60% of cases does make sense for chat applications. For applications where we do care about the most likely sentence (e.g. question answering), then beam search helps, as others have mentioned. Another thing to consider is that the model can "look…

Paper and discussion history. A mention is not an endorsement.

Do language models plan ahead for future tokens? — HN submission

Thu, 04 Apr 2024 16:26:58 GMT

Do language models plan ahead for future tokens?

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.