Language Models Represent Space and Time · HackerPapers discussions

Language Models Represent Space and Time — mentioned by ashdksnndck

Sat, 14 Jun 2025 04:54:14 GMT

There is already research in the literature showing that LLMs have neurons that model the gender [1], personality [2], ideology [3], and historic era [4] of the author. There’s also evidence that they model the distinction between the beliefs of the author and other characters, which has been summarized as “theory of mind” [5]. And we have only scratched the surface, with most research using small open-weight models that lag behind frontier model capabilities. [1] Z. Yu & S. Ananiadou, “Understanding and Mitigating Gender Bias in LLMs via Interpretable Neuron Editing,” arXiv:2501.14457…

Paper and discussion history. A mention is not an endorsement.

Language Models Represent Space and Time — mentioned by naasking

Thu, 31 Oct 2024 15:49:47 GMT

> They are literally next token prediction machines normally trained on just text tokens. And in order to predict the next token well they have to build world models, otherwise they would just output nonsense. This has been proven [1]. This notion that just calling them "next token predictors" somehow precludes them being intelligent is based on a premise that human intelligence cannot be reduced to next token prediction, but nobody has proven any such thing! In fact, our best models for human cognition are literally predictive coding. LLMs are probably not the final story in AGI, but…

Paper and discussion history. A mention is not an endorsement.

Language Models Represent Space and Time — mentioned by agucova

Thu, 18 Jul 2024 18:42:23 GMT

> Now, we can see from this description that nothing about the modeling ensures that the outputs accurately depict anything in the world. There is not much reason to think that the outputs are connected to any sort of internal representation at all. This is just wrong. Accurate modelling of language at the scale of modern LLMs requires these models to develop rich world models during pretraining, which also requires distinguishing facts from fiction. This is why bullshitting happens less with better, bigger models: the simple answer is that they just know more about the world, and can also…

Paper and discussion history. A mention is not an endorsement.

Language Models Represent Space and Time — mentioned by edmara

Sat, 01 Jun 2024 13:55:34 GMT

https://arxiv.org/abs/2310.02207 https://transformer-circuits.pub/2024/scaling-monosemanticit...

Paper and discussion history. A mention is not an endorsement.

Language Models Represent Space and Time — mentioned by kromem

Mon, 20 Nov 2023 11:10:25 GMT

There's been a fair bit of research over the last year on this topic. An easy read on a Harvard/MIT study: https://thegradient.pub/othello/ A follow-up on more technical aspects of what's going on with it: https://www.lesswrong.com/posts/nmxzr2zsjNtjaHh7x/actually-o... Two more studies since showing linear representations of world models: https://arxiv.org/abs/2310.02207 (modeling space and time) https://arxiv.org/abs/2310.06824 (modeling truth vs falsehood) It's worth keeping in mind these are all on smaller toy models compared to something like GPT-4, so there's likely more complex versions…

Paper and discussion history. A mention is not an endorsement.

Language Models Represent Space and Time — mentioned by frabcus

Tue, 14 Nov 2023 09:51:16 GMT

This varies a lot by people - some people's conscious thought is in images, some in language, some in unsymbolised concepts. Hurlburt's "Descriptive Experience Sampling" is a good source on this - see for example table 2 of https://hurlburt.faculty.unlv.edu/heavey-hurlburt-2008.pdf Agreed that underneath that perceivable layer, there's lots else going on too. But to be fair, there's also lots going on the weights of GPT-4, e.g. see https://arxiv.org/abs/2310.02207

Paper and discussion history. A mention is not an endorsement.

Language Models Represent Space and Time — mentioned by lyu07282

Wed, 11 Oct 2023 11:20:09 GMT

https://arxiv.org/abs/2210.13382 https://arxiv.org/abs/2310.02207 among others

Paper and discussion history. A mention is not an endorsement.

Language Models Represent Space and Time — mentioned by richdougherty

Fri, 06 Oct 2023 09:56:01 GMT

https://arxiv.org/abs/2310.02207

Paper and discussion history. A mention is not an endorsement.

Language Models Represent Space and Time — mentioned by Nydhal

Thu, 05 Oct 2023 18:04:23 GMT

ArXiv link: https://arxiv.org/abs/2310.02207

Paper and discussion history. A mention is not an endorsement.

Language Models Represent Space and Time — HN submission

Wed, 04 Oct 2023 14:23:43 GMT

Language Models Represent Space and Time

123 points; 186 discussion replies on HN. Paper and source history. A submission is not an endorsement.