Large Concept Models: Language Modeling in a Sentence Representation Space · HackerPapers discussions

Large Concept Models: Language Modeling in a Sentence Representation Space — mentioned by aspenmartin

Fri, 09 Jan 2026 17:55:32 GMT

I don’t think for this approach it sounds like, this is related to the large concept model: https://arxiv.org/abs/2412.08821, where the latent space is SONAR, which is very much designed for this purpose. You learn SONAR embeddings so that every sentence with the same semantic meaning gets mapped to the same latent representation. So you can have e.g. a French SONAR encoder and a Finnish SONAR encoder, trained separately with large scale corpi of paired sentences with the same meaning (basically the same thing you would use for learning translation models directly, but for SONAR you don’t…

Paper and discussion history. A mention is not an endorsement.

Large Concept Models: Language Modeling in a Sentence Representation Space — mentioned by naasking

Fri, 13 Jun 2025 16:30:58 GMT

> the behavior of LLMs is not thanks to a learned world model, but to brute force memorization of incomprehensibly abstract rules governing the behavior of symbols, i.e. a model of syntax. I think reinforcing this distinction between syntax and semantics is wrong. I think what LLMs have shown is that semantics reduces to the network of associations between syntax (symbols). So LLMs do learn a world model if trained long enough, past the "grokking" threshold, but they are learning a model of the world that we've described linguistically, and natural language is ambiguous, imprecise and not…

Paper and discussion history. A mention is not an endorsement.

Large Concept Models: Language Modeling in a Sentence Representation Space — mentioned by AlexCoventry

Sat, 15 Mar 2025 01:53:19 GMT

Not an insider, but: I don't know about you, but I certainly don't generate text autoregressively, token by token. Also, pretty sure I don't learn by global updates based on taking the derivative of some objective function of my behavior with respect to every parameter defining my brain. So there's good biological reason to think we can go beyond the capabilities of current architectures. I think probably an example of the kind of new architectures he supports is FB's Large Concept Models [1]. It's still a self-attention, autoregressive architecture, but the unit of regression is a sentence…

Paper and discussion history. A mention is not an endorsement.

Large Concept Models: Language Modeling in a Sentence Representation Space — mentioned by sgk284

Tue, 18 Feb 2025 00:08:40 GMT

Another take on a similar idea from FAIR is a Large Concept Model: https://arxiv.org/pdf/2412.08821

Paper and discussion history. A mention is not an endorsement.