Training Large Language Models to Reason in a Continuous Latent Space

The recurrent depth sounds a lot more like what is described in https://dnhkng.github.io/posts/rys/ - a way to add depth to a network without increasing the number of parameters. e: While the actual CoT in neuralese paper is Facebook's Coconut https://arxiv.org/abs/2412.06769…

tveitaSep 6, 2026View on HNParent context

Is there a more technical source that tells what is being done? I assume it's about reasoning in latent space e.g. the coconut paper[1]? I disagree with the "experts" there, as if latent-space reasoning will only cause proper interpretability research rather than taking the CoT…

supriyo-biswasSep 3, 2026View on HNParent context

"Latent" means it (the space) is there all the time but usually not exposed or apparent. Latent space is a high-dimensional space that is not legible to humans, so normally reasoning gets constrained to progress in tokens instead so that it can be evaluated. I think…

entropeAug 17, 2026View on HNParent context

I saw that idea described as a step in AI 2027 (they call it "neuralese" and eyeballing the site, it's still labeled a hypothetical/future development), but AFAIK no one implemented/deployed this yet. EDIT: They link to a Meta paper from 2024/2025 though…

TeMPOraLJun 22, 2026View on HNParent context

> If that's true, then we're following the timeline Literally just a citation of Meta's Coconut paper[1]. Notice the 2027 folk's contribution to the prediction is that this will have been implemented by "thousands of Agent-2 automated researchers...making major algorithmic…

magicalistApr 16, 2026View on HNParent context

natural language may provide part of the scaffolding for reasoning, but the capability itself seems to depend more on learned transformations over internal representations than on language alone refs: https://arxiv.org/abs/2412.17819 https://arxiv.org/abs/2412.06769

a-t-c-gApr 1, 2026View on HNParent context

> extra attention head were added that queried the KV data from lower layers Isn't this sort of similar to latent looping? E.g. [1]. But actually as [2] argues, even that wasn't a good experiment because it used the very last hidden state, which is too close to the logits and…

krackersDec 23, 2025View on HNParent context

Unfortunately I think the competitive pressures to improve model performance makes this kind of monitor-ability is short-lived. It's just not likely that textual reasoning in english ismost optimal. Researchers are already pushing in this direction…

vonneumannstanJul 16, 2025View on HNParent context

There’s the coconut paper by meta, which kind of does a similar thing by reasoning in the latent space instead of doing it in tokens: https://arxiv.org/abs/2412.06769 Given that anthropic’s interpretability work finds that CoT does not reliably predict the model’s internal…

supriyo-biswasJul 16, 2025View on HNParent context

For those interested in the paper, it follows on very interesting work out of FAIR: https://arxiv.org/pdf/2412.06769

danielmorozoffJun 18, 2025View on HNParent context

> Just because your sampler picks a sequence of tokens that contain incorrect reasoning doesn't mean a useful reasoning trace isn’t also contained within the latent space. That's essentially the core idea in Coconut[1][2], to keep the reasoning traces in a continuous space. [1]…

woadwarrior01May 23, 2025View on HNParent context

> It's interesting that there are no reasoning models yet This may be merely a naming distinction, leaving the name open for a future release based on their recent research such as coconut[1]. They did RL post-training, and when fed logic problems it appears to do significant…

voxgenApr 6, 2025View on HNParent context

> I suppose that raises another questions: Do LLMs "think" in language? Or do they think in a more abstract space, then translate it to language later? My money is on the latter. The processing happens in latent space and then is converted to tokens/token space. There is…

eightysixfourFeb 17, 2025View on HNParent context

Would be curious to know how this stacks up against Coconut [1] which also uses latent space for reasoning. [1] https://arxiv.org/abs/2412.06769

scribuFeb 3, 2025View on HNParent context

This recent paper is relevant: https://arxiv.org/abs/2412.06769

kevinventulloJan 26, 2025View on HNParent context

Paper: https://arxiv.org/abs/2412.06769 The link is in the OP, hidden away in an image caption fir some reason.

CGamesPlayDec 31, 2024View on HNParent context

The official pdf for the paper is at https://arxiv.org/pdf/2412.06769 I can find "Coconut" 54 times in the PDF. The movie does not look made up.

davidhowlettDec 31, 2024View on HNParent context

Explicit <eot> is only used in training. At inference time, the paper says: > A challenge lies in determining when to switch between latent and language modes. As we focus on the problem-solving setting, we insert a <bot> token immediately following the question tokens. For…

avodonosovJan 1, 2025View on HNParent context

I think the wildest thing is actually Meta’s latest paper where they show a method for LLMs reasoning not in English, but in latent space https://arxiv.org/pdf/2412.06769 I’ve done research myself adjacent to this (mapping parts of a latent space onto a manifold), but this is a…

exhazeDec 22, 2024View on HNParent context

I think you mean this: https://arxiv.org/abs/2412.06769 From what I can see, presuming o3 is a progression of o1 and has good level of accountabiltiy bubbling up during 'inference' (i.e. "Thinking about ___") then I'd say it's just using up millions of old-school tokens (the 44…

padolseyDec 21, 2024View on HNParent context

Large language models (LLMs) are restricted to reason in the “language space”, where they typically express the reasoning process with a chain-of-thought (CoT) to solve a complex reasoning problem. However, we argue that language space may not always be optimal for reasoning.…

verdvermDec 11, 2024View on HNParent context