Training Large Language Models to Reason in a Continuous Latent Space · HackerPapers discussions

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by tveita

Sun, 06 Sep 2026 16:42:40 GMT

The recurrent depth sounds a lot more like what is described in https://dnhkng.github.io/posts/rys/ - a way to add depth to a network without increasing the number of parameters. e: While the actual CoT in neuralese paper is Facebook's Coconut https://arxiv.org/abs/2412.06769 - not sure if any production models use that one.

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by supriyo-biswas

Thu, 03 Sep 2026 18:24:22 GMT

Is there a more technical source that tells what is being done? I assume it's about reasoning in latent space e.g. the coconut paper[1]? I disagree with the "experts" there, as if latent-space reasoning will only cause proper interpretability research rather than taking the CoT as gospel[2]. Give me that any day over the constant "you will be abolished to the permanent underclass!" talk coming from billionaires. Edit: (A better discussion is apparently here[3]) [1] https://arxiv.org/pdf/2412.06769 [2] https://thezvi.substack.com/p/the-most-forbidden-technique [3]…

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by entrope

Mon, 17 Aug 2026 09:25:58 GMT

"Latent" means it (the space) is there all the time but usually not exposed or apparent. Latent space is a high-dimensional space that is not legible to humans, so normally reasoning gets constrained to progress in tokens instead so that it can be evaluated. I think https://arxiv.org/abs/2412.06769 is a good description of the premise of reasoning in latent space, although https://arxiv.org/abs/2604.15726 argues it's already what really happens.

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by TeMPOraL

Mon, 22 Jun 2026 16:14:00 GMT

I saw that idea described as a step in AI 2027 (they call it "neuralese" and eyeballing the site, it's still labeled a hypothetical/future development), but AFAIK no one implemented/deployed this yet. EDIT: They link to a Meta paper from 2024/2025 though: https://arxiv.org/pdf/2412.06769/.

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — HN submission

Wed, 29 Apr 2026 11:33:34 GMT

Training Large Language Models to Reason in a Continuous Latent Space [pdf]

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by magicalist

Thu, 16 Apr 2026 19:44:55 GMT

> If that's true, then we're following the timeline Literally just a citation of Meta's Coconut paper[1]. Notice the 2027 folk's contribution to the prediction is that this will have been implemented by "thousands of Agent-2 automated researchers...making major algorithmic advances". So, considering that the discussion of latent space reasoning dates back to 2022[2] through CoT unfaithfulness, looped transformers, using diffusion for refining latent space thoughts, etc, etc, all published before ai 2027, it seems like to be "following the timeline of ai-2027" we'd actually need to verify that…

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by a-t-c-g

Wed, 01 Apr 2026 15:38:13 GMT

natural language may provide part of the scaffolding for reasoning, but the capability itself seems to depend more on learned transformations over internal representations than on language alone refs: https://arxiv.org/abs/2412.17819 https://arxiv.org/abs/2412.06769

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by krackers

Tue, 23 Dec 2025 06:09:39 GMT

> extra attention head were added that queried the KV data from lower layers Isn't this sort of similar to latent looping? E.g. [1]. But actually as [2] argues, even that wasn't a good experiment because it used the very last hidden state, which is too close to the logits and loses most of the rich embedding structure. Perhaps you don't even need access to the state of anything except the penultimate hidden layer, since based on my vague reading of [3] the residual stream doesn't "lose information" as it passes deeper down the attention layers, so each block maybe manipulates a different…

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by supriyo-biswas

Wed, 16 Jul 2025 20:58:34 GMT

There’s the coconut paper by meta, which kind of does a similar thing by reasoning in the latent space instead of doing it in tokens: https://arxiv.org/abs/2412.06769 Given that anthropic’s interpretability work finds that CoT does not reliably predict the model’s internal reasoning process, I think approaches like the one above are more likely to succeed.

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by vonneumannstan

Wed, 16 Jul 2025 20:35:32 GMT

Unfortunately I think the competitive pressures to improve model performance makes this kind of monitor-ability is short-lived. It's just not likely that textual reasoning in english ismost optimal. Researchers are already pushing in this direction: https://arxiv.org/abs/2502.05171 "We study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space. Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test-time. This stands in contrast to mainstream reasoning models that scale up compute by…

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by danielmorozoff

Wed, 18 Jun 2025 15:20:55 GMT

For those interested in the paper, it follows on very interesting work out of FAIR: https://arxiv.org/pdf/2412.06769

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by woadwarrior01

Fri, 23 May 2025 18:17:48 GMT

> Just because your sampler picks a sequence of tokens that contain incorrect reasoning doesn't mean a useful reasoning trace isn’t also contained within the latent space. That's essentially the core idea in Coconut[1][2], to keep the reasoning traces in a continuous space. [1]: https://arxiv.org/abs/2412.06769 [2]: https://github.com/facebookresearch/coconut

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by voxgen

Sun, 06 Apr 2025 08:52:50 GMT

> It's interesting that there are no reasoning models yet This may be merely a naming distinction, leaving the name open for a future release based on their recent research such as coconut[1]. They did RL post-training, and when fed logic problems it appears to do significant amounts of step-by-step thinking[2]. It seems it just doesn't wrap it in <thinking> tags. [1] https://arxiv.org/abs/2412.06769 "Training Large Language Models to Reason in a Continuous Latent Space" [2] https://www.youtube.com/watch?v=12lAM-xPvu8 (skip through this - it's recorded in real time)

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by eightysixfour

Mon, 17 Feb 2025 19:47:23 GMT

> I suppose that raises another questions: Do LLMs "think" in language? Or do they think in a more abstract space, then translate it to language later? My money is on the latter. The processing happens in latent space and then is converted to tokens/token space. There is research into reasoning models which can spend extra compute in latent space instead of in token space: https://arxiv.org/abs/2412.06769

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by scribu

Mon, 03 Feb 2025 16:27:51 GMT

Would be curious to know how this stacks up against Coconut [1] which also uses latent space for reasoning. [1] https://arxiv.org/abs/2412.06769

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by kevinventullo

Sun, 26 Jan 2025 17:13:25 GMT

This recent paper is relevant: https://arxiv.org/abs/2412.06769

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by avodonosov

Wed, 01 Jan 2025 09:06:37 GMT

Explicit <eot> is only used in training. At inference time, the paper says: > A challenge lies in determining when to switch between latent and language modes. As we focus on the problem-solving setting, we insert a <bot> token immediately following the question tokens. For <eot>, we consider two potential strategies: a) train a binary classifier on latent thoughts to enable the model to autonomously decide when to terminate the latent reasoning, or b) always pad the latent thoughts to a constant length. We found that both approaches work comparably well. Therefore, we use the second option…

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by davidhowlett

Tue, 31 Dec 2024 09:31:39 GMT

The official pdf for the paper is at https://arxiv.org/pdf/2412.06769 I can find "Coconut" 54 times in the PDF. The movie does not look made up.

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by CGamesPlay

Tue, 31 Dec 2024 05:05:14 GMT

Paper: https://arxiv.org/abs/2412.06769 The link is in the OP, hidden away in an image caption fir some reason.

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by exhaze

Sun, 22 Dec 2024 22:41:09 GMT

I think the wildest thing is actually Meta’s latest paper where they show a method for LLMs reasoning not in English, but in latent space https://arxiv.org/pdf/2412.06769 I’ve done research myself adjacent to this (mapping parts of a latent space onto a manifold), but this is a bit eerie, even to me.

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by padolsey

Sat, 21 Dec 2024 14:30:57 GMT

I think you mean this: https://arxiv.org/abs/2412.06769 From what I can see, presuming o3 is a progression of o1 and has good level of accountabiltiy bubbling up during 'inference' (i.e. "Thinking about ___") then I'd say it's just using up millions of old-school tokens (the 44 million tokens that are referenced). So not latent thinking per se.

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — mentioned by verdverm

Wed, 11 Dec 2024 07:27:19 GMT

Large language models (LLMs) are restricted to reason in the “language space”, where they typically express the reasoning process with a chain-of-thought (CoT) to solve a complex reasoning problem. However, we argue that language space may not always be optimal for reasoning. For example, most word tokens are primarily for textual coherence and not essential for reasoning, while some critical tokens require complex planning and pose huge challenges to LLMs. To explore the potential of LLM reasoning in an unrestricted latent space instead of using natural language, we introduce a new paradigm…

Paper and discussion history. A mention is not an endorsement.

Training Large Language Models to Reason in a Continuous Latent Space — HN submission

Tue, 10 Dec 2024 16:26:17 GMT

Training LLMs to Reason in a Continuous Latent Space

283 points; 114 discussion replies on HN. Paper and source history. A submission is not an endorsement.