Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task

We understand the low level details of how they are constructed. But we do not fully understand how higher-level behavior emerges - it is a subject of active research. For example: https://arxiv.org/html/2210.13382v5 https://arxiv.org/abs/2109.06129

jeremyjhApr 30, 2026View on HNParent context

We understand the low level math quite well. We do not understand the source of emergent behavior. https://arxiv.org/html/2210.13382v5#abstract

jeremyjhApr 30, 2026View on HNParent context

Here's an article where they teach an LLM Othello and then probe its internal state to assess whether it is 'modelling' the Othello board internally https://thegradient.pub/othello/ Associated paper: https://arxiv.org/abs/2210.13382

codeulikeNov 22, 2024View on HNParent context

related : Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task https://arxiv.org/abs/2210.13382 Chess-GPT's Internal World Model https://adamkarvonen.github.io/machine_learning/2024/01/03/c... discussed here…

MiraltarNov 15, 2024View on HNParent context

When trained on simple logs of Othello's moves, the model learns an internal representation of the board and its pieces. It also models the strength of its opponent. https://arxiv.org/abs/2210.13382 I'd be more surprised if LLMs trained on human conversations don't create any…

thomashopOct 31, 2024View on HNParent context

Wrong. This is the common groupspeak on uninformed places like HN, but it is not what the current research says. See e.g. this: https://arxiv.org/abs/2210.13382 Most of what you wrote shows that you have zero education in modern deep learning, so I really wonder what makes you…

sigmoid10Jun 1, 2024View on HNParent context

I was on that side of the argument as well, but it seems that models can have emerging model states of a given situation https://arxiv.org/abs/2210.13382

DavidbrczMay 18, 2024View on HNParent context

I see many people get confused by this due to the widely spread (and false) "stochastic parrot" theme. But these models are much more than mere senzence-repeaters. In a way, the model is not learning that after A comes B. I mean, it could. With a lack of additional training data…

sigmoid10Mar 30, 2024View on HNParent context

https://arxiv.org/abs/2210.13382 https://arxiv.org/abs/2310.02207 among others

lyu07282Oct 11, 2023View on HNParent context

>Actual reasoning, or reconstruction of existing texts containing similar reasoning? The papers were linked in another comment. 3 of them don't even have anything to do with a existing dataset testing. so yeah, actual. for the world model papers https://arxiv.org/abs/2210.13382…

famouswafflesMay 22, 2023View on HNParent context

No world model ? A world model is so obvious papers like these are more confirmation than surprise https://arxiv.org/abs/2305.11169 https://arxiv.org/abs/2210.13382 There a certain sentiment that AGI however you wish to define it won't infact be a "We'll know it when we see it"…

famouswafflesMay 21, 2023View on HNParent context

There is in fact compelling evidence that language models can learn some world models. Here's a research paper on this and a blog post by the lead author summarizing the results. Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task.…

nopinsightMay 21, 2023View on HNParent context

It is not about what the model tells you. This paper shows an emergent world model in an LLM that was taught to play otello moves https://ar5iv.labs.arxiv.org/html/2210.13382 https://arxiv.org/pdf/2303.12712.pdf This paper discusses (among other things) how a GPT4 model…

patrickasMay 16, 2023View on HNParent context

LLMs demonstrably model their training data, which has a correspondence to the structure in the world captured in the data[1][2]. This goes far beyond just searching a database for the closest relevant response. The question is what does that mean regarding understanding? I go…

hackinthebochsApr 13, 2023View on HNParent context

LLMs demonstrably model their training data, which has a correspondence to the structure in the world[1][2]. The question is what does that mean regarding understanding? I go into that in some detail here[3]. [1] https://arxiv.org/abs/2210.13382 [2]…

hackinthebochsApr 10, 2023View on HNParent context

This paper from late last year shows that LLMs are not "just" stochastic parrots, but they actually build an internal model of the "world" that is not programmed in, just from trying to predict the next token. https://ar5iv.labs.arxiv.org/html/2210.13382 PS: More research has…

patrickasApr 6, 2023View on HNParent context

Have a look at this research paper[1] and follow-up post[2]. The upshot is that the researchers trained a model of the same shape that the main language models are to play Othello. The model started with no knowledge of the game, and was fed a bunch of move sequences (e.g. "c4…

JoshuaDavidApr 4, 2023View on HNParent context

LLMs clearly develop internal representations, this is an empirical fact. For example, one paper cited is https://arxiv.org/abs/2210.13382, which trained an LLM on Othello games, where an 8 layer model is fed one of 60 tokens (there are 60 possible moves in Othello, 8*8 minus 4…

sanxiynApr 4, 2023View on HNParent context

No there is reasoning in LLMs. This is actually proven through several scientific papers. I can't say whether it's using reasoning for specific answers. I'm sure many times it just regurgitates something that is known... But saying that LLMs in general can and do reason is…

Herval_freireMar 31, 2023View on HNParent context

> The 2 + 2 = 5 stuff is just an illustration that LLMs are fairly simple probability models and not, yet, understanding anything. See for example this research paper[1]. The researchers trained the model on sequences of moves in the game of Othello[2]. The model started with no…

JoshuaDavidMar 30, 2023View on HNParent context

>> LLMs have no knowledge of the underlying reality > They have no common sense & they can't plan their answer If his arguments are entirely based on this, then it's not fully correct: - GPT style language models try to build a model of the world…

HervalFreireMar 27, 2023View on HNParent context

Sure. A few below but far from exhaustive: - https://arxiv.org/abs/1909.07528 - https://arxiv.org/abs/2212.10403 - https://arxiv.org/abs/2201.11903 - https://arxiv.org/abs/2210.13382 There are also literally hundreds of articles and tweet threads about it. Moreover, as I said…

soraki_soladeadMar 19, 2023View on HNParent context

- GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples)…

HervalFreireMar 16, 2023View on HNParent context

- GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples)…

HervalFreireMar 16, 2023View on HNParent context

>That said, GPT has no model of the world. I don't think this is necessarily true. Here is an example where researchers trained a transformer to generate legal sequences of moves in the board game Othello. Then they demonstrated that the internal state of the model did, in fact…

sebzim4500Mar 14, 2023View on HNParent context

People have shown GPT has an internal model of the state of a game of Othello: Https://arxiv.org/abs/2210.13382

fancyfredbotMar 14, 2023View on HNParent context

I believe https://arxiv.org/pdf/2210.13382.pdf

JyBMar 18, 2023View on HNParent context

https://arxiv.org/pdf/2210.13382.pdf

JyBMar 18, 2023View on HNParent context

I have some good (or bad?) news for you then. Bing chat has no access to a Stockfish or other chess API. There is strong evidence that LLMs can learn and play games, including Chess. See this paper Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic…

nayrocladeMar 8, 2023View on HNParent context

GPTs can model simpler games like Othello https://thegradient.pub/othello/ https://arxiv.org/abs/2210.13382 with enough training data it could probably model chess - not well enough to win but well enough to make legal moves You think that I am misunderstanding whats going on…

codeulikeFeb 27, 2023View on HNParent context

- GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples)…

kilgnadFeb 27, 2023View on HNParent context

- GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples)…

kilgnadMar 3, 2023View on HNParent context

Here is just one recent example: https://arxiv.org/abs/2210.13382 If you actually follow the literature, you'll find that there is tons of evidence that the seemingly "simple" transformer architecture might actually work pretty similar to the way the human brain is believed to…

sigmoid10Feb 15, 2023View on HNParent context

The answer to this is: "we don't really know as its a very complex function automatically discovered by means of slow gradient descent, and we're still finding out" Here are some of the fun things we've found out so far: - GPT style language models try to build a model of the…

spionFeb 14, 2023View on HNParent context

It's clearly early technology, so it's not perfect. But what it is able to get right is clear proof it's more then what you think: https://www.engraved.blog/building-a-virtual-machine-inside/ Read to the end. The beginning and middle doesn't show off anything too impressive.…

kilgnadFeb 15, 2023View on HNParent context

I read the web page and the PDF paper quickly (https://arxiv.org/pdf/2210.13382.pdf) and it seems that there are no examples of the prompts they used. It would be great to see which technique they are using or is it obvious for experts of the field? It would help with…

pmontraJan 22, 2023View on HNParent context

Btw this post is based on a preprint - might be good to wait a lil https://arxiv.org/pdf/2210.13382.pdf

meow_mixJan 22, 2023View on HNParent context