Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task · HackerPapers discussions

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by jeremyjh

Thu, 30 Apr 2026 11:49:55 GMT

We understand the low level math quite well. We do not understand the source of emergent behavior. https://arxiv.org/html/2210.13382v5#abstract

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by jeremyjh

Thu, 30 Apr 2026 11:35:11 GMT

We understand the low level details of how they are constructed. But we do not fully understand how higher-level behavior emerges - it is a subject of active research. For example: https://arxiv.org/html/2210.13382v5 https://arxiv.org/abs/2109.06129

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by codeulike

Fri, 22 Nov 2024 12:24:53 GMT

Here's an article where they teach an LLM Othello and then probe its internal state to assess whether it is 'modelling' the Othello board internally https://thegradient.pub/othello/ Associated paper: https://arxiv.org/abs/2210.13382

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by Miraltar

Fri, 15 Nov 2024 09:25:42 GMT

related : Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task https://arxiv.org/abs/2210.13382 Chess-GPT's Internal World Model https://adamkarvonen.github.io/machine_learning/2024/01/03/c... discussed here https://news.ycombinator.com/item?id=38893456

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by thomashop

Thu, 31 Oct 2024 07:27:30 GMT

When trained on simple logs of Othello's moves, the model learns an internal representation of the board and its pieces. It also models the strength of its opponent. https://arxiv.org/abs/2210.13382 I'd be more surprised if LLMs trained on human conversations don't create any world models. Having a world model simply allows the LLM to become better at sequence prediction. No magic needed. There was another recent paper that shows that a language model is modelling things like age, gender, etc., of their conversation partner without having been explicitly trained for it

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by sigmoid10

Sat, 01 Jun 2024 07:32:56 GMT

Wrong. This is the common groupspeak on uninformed places like HN, but it is not what the current research says. See e.g. this: https://arxiv.org/abs/2210.13382 Most of what you wrote shows that you have zero education in modern deep learning, so I really wonder what makes you form such strong opinions on something you know nothing about.

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by Davidbrcz

Sat, 18 May 2024 19:45:27 GMT

I was on that side of the argument as well, but it seems that models can have emerging model states of a given situation https://arxiv.org/abs/2210.13382

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by sigmoid10

Sat, 30 Mar 2024 21:40:56 GMT

I see many people get confused by this due to the widely spread (and false) "stochastic parrot" theme. But these models are much more than mere senzence-repeaters. In a way, the model is not learning that after A comes B. I mean, it could. With a lack of additional training data it probably would, too. But with enough data, this kind of sentence completion based purely on existing sentences no longer works because it would saturate parameters. So to retain and improve accuracy during training, it will have to come up with a compression that essentially forms a model of the real world. Or at…

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by lyu07282

Wed, 11 Oct 2023 11:20:09 GMT

https://arxiv.org/abs/2210.13382 https://arxiv.org/abs/2310.02207 among others

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by famouswaffles

Mon, 22 May 2023 23:49:02 GMT

>Actual reasoning, or reconstruction of existing texts containing similar reasoning? The papers were linked in another comment. 3 of them don't even have anything to do with a existing dataset testing. so yeah, actual. for the world model papers https://arxiv.org/abs/2210.13382 https://arxiv.org/abs/2305.11169 >Lack of access to cameras or vehicle controls isn't why it can't drive a car. It would be best to wait till what you say can be evaluated. that is your hunch, not fact. >The existence of numerous ChatGPT jailbreaks is evidence to the contrary. No it's not. People fall for social…

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by nopinsight

Sun, 21 May 2023 05:36:34 GMT

There is in fact compelling evidence that language models can learn some world models. Here's a research paper on this and a blog post by the lead author summarizing the results. Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. https://arxiv.org/abs/2210.13382 Do Large Language Models learn world models or just surface statistics? https://thegradient.pub/othello/ ----- An argument based on common sense can also be made: Any system that possesses a wide range of capabilities, most of which it was not specifically trained to perform, cannot possibly…

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by famouswaffles

Sun, 21 May 2023 02:46:45 GMT

No world model ? A world model is so obvious papers like these are more confirmation than surprise https://arxiv.org/abs/2305.11169 https://arxiv.org/abs/2210.13382 There a certain sentiment that AGI however you wish to define it won't infact be a "We'll know it when we see it" situation but rather a "AGI will arrive long before consensus reaches its AGI". LLMs have made me believe this will 100% be the case, either way. It's one thing to argue over things we can't evaluate even now but man the 100th "They can't reason!" every week is pretty funny when you can basically take your pick of…

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by patrickas

Tue, 16 May 2023 19:13:02 GMT

It is not about what the model tells you. This paper shows an emergent world model in an LLM that was taught to play otello moves https://ar5iv.labs.arxiv.org/html/2210.13382 https://arxiv.org/pdf/2303.12712.pdf This paper discusses (among other things) how a GPT4 model navigated between rooms in a text adventure game and was able to create a map afterward. Literally building a model of the world as it was navigating and drawing a map of that afterwards

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by hackinthebochs

Thu, 13 Apr 2023 20:09:38 GMT

LLMs demonstrably model their training data, which has a correspondence to the structure in the world captured in the data[1][2]. This goes far beyond just searching a database for the closest relevant response. The question is what does that mean regarding understanding? I go into that in some detail here[3]. [1] https://arxiv.org/abs/2210.13382 [2] https://twitter.com/leopoldasch/status/1638848881558704129 [3] https://www.reddit.com/r/naturalism/comments/1236vzf/

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by hackinthebochs

Mon, 10 Apr 2023 19:54:17 GMT

LLMs demonstrably model their training data, which has a correspondence to the structure in the world[1][2]. The question is what does that mean regarding understanding? I go into that in some detail here[3]. [1] https://arxiv.org/abs/2210.13382 [2] https://twitter.com/leopoldasch/status/1638848881558704129 [3] https://www.reddit.com/r/naturalism/comments/1236vzf/on_larg...

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by patrickas

Thu, 06 Apr 2023 15:40:44 GMT

This paper from late last year shows that LLMs are not "just" stochastic parrots, but they actually build an internal model of the "world" that is not programmed in, just from trying to predict the next token. https://ar5iv.labs.arxiv.org/html/2210.13382 PS: More research has been done since that confirmed and strengthened the conclusion.

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by JoshuaDavid

Tue, 04 Apr 2023 23:30:53 GMT

Have a look at this research paper[1] and follow-up post[2]. The upshot is that the researchers trained a model of the same shape that the main language models are to play Othello. The model started with no knowledge of the game, and was fed a bunch of move sequences (e.g. "c4 c3 d3 e3"). The researchers were then able to take that model, and figure out, by looking only at the activations, what the board state was. Changing the activations so that they modeled a different board state caused the model to play moves that made sense with the new board state, and did not make sense with the…

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by sanxiyn

Tue, 04 Apr 2023 04:57:46 GMT

LLMs clearly develop internal representations, this is an empirical fact. For example, one paper cited is https://arxiv.org/abs/2210.13382, which trained an LLM on Othello games, where an 8 layer model is fed one of 60 tokens (there are 60 possible moves in Othello, 8*8 minus 4 initially occupied squares). You can probe this model at layer 5, and it computes Othello board state from moves, and you can be sure this board state is used, because if you patch activation so that board state is modified, the model generates moves for this modified board state.

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by Herval_freire

Fri, 31 Mar 2023 16:17:44 GMT

No there is reasoning in LLMs. This is actually proven through several scientific papers. I can't say whether it's using reasoning for specific answers. I'm sure many times it just regurgitates something that is known... But saying that LLMs in general can and do reason is categorically true. LLMs do attempt to extract actual mental models of the subject of the text. See here: - GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent…

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by JoshuaDavid

Thu, 30 Mar 2023 04:39:20 GMT

> The 2 + 2 = 5 stuff is just an illustration that LLMs are fairly simple probability models and not, yet, understanding anything. See for example this research paper[1]. The researchers trained the model on sequences of moves in the game of Othello[2]. The model started with no knowledge of the game, and was fed a bunch of move sequences (e.g. "c4 c3 d3 e3"). The researchers then were able to look at the model activations and figure out what it thought the board state was. When they updated those activations so that they represented a different board state, the model made moves that made…

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by HervalFreire

Mon, 27 Mar 2023 06:30:58 GMT

>> LLMs have no knowledge of the underlying reality > They have no common sense & they can't plan their answer If his arguments are entirely based on this, then it's not fully correct: - GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): https://arxiv.org/abs/2212.10559

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by soraki_soladead

Sun, 19 Mar 2023 20:27:45 GMT

Sure. A few below but far from exhaustive: - https://arxiv.org/abs/1909.07528 - https://arxiv.org/abs/2212.10403 - https://arxiv.org/abs/2201.11903 - https://arxiv.org/abs/2210.13382 There are also literally hundreds of articles and tweet threads about it. Moreover, as I said, you can test many of my claims above directly using readily available LLMs. GP has a much harder defense. They have to prove that despite all of these capabilities that LLMs are not intelligent. That the mechanisms by which humans possess intelligence is fundamentally distinct from a computer’s ability to exhibit the…

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by JyB

Sat, 18 Mar 2023 21:21:07 GMT

https://arxiv.org/pdf/2210.13382.pdf

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by JyB

Sat, 18 Mar 2023 21:19:46 GMT

I believe https://arxiv.org/pdf/2210.13382.pdf

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by HervalFreire

Thu, 16 Mar 2023 07:43:04 GMT

- GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): https://arxiv.org/abs/2212.10559

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by HervalFreire

Thu, 16 Mar 2023 06:03:11 GMT

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by fancyfredbot

Tue, 14 Mar 2023 18:44:42 GMT

People have shown GPT has an internal model of the state of a game of Othello: Https://arxiv.org/abs/2210.13382

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by sebzim4500

Tue, 14 Mar 2023 18:28:57 GMT

>That said, GPT has no model of the world. I don't think this is necessarily true. Here is an example where researchers trained a transformer to generate legal sequences of moves in the board game Othello. Then they demonstrated that the internal state of the model did, in fact, have a representation of the board. https://arxiv.org/abs/2210.13382

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by nayroclade

Wed, 08 Mar 2023 22:21:05 GMT

I have some good (or bad?) news for you then. Bing chat has no access to a Stockfish or other chess API. There is strong evidence that LLMs can learn and play games, including Chess. See this paper Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task by Kenneth Li et al.: https://arxiv.org/abs/2210.13382 or a more accessible write-up here: https://thegradient.pub/othello/

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by kilgnad

Fri, 03 Mar 2023 02:36:49 GMT

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by kilgnad

Mon, 27 Feb 2023 19:19:06 GMT

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by codeulike

Mon, 27 Feb 2023 08:59:31 GMT

GPTs can model simpler games like Othello https://thegradient.pub/othello/ https://arxiv.org/abs/2210.13382 with enough training data it could probably model chess - not well enough to win but well enough to make legal moves You think that I am misunderstanding whats going on and anthropomorphising ChatGPT. I know how ChatGPT works, my position is that we might be overestimating our selves, and underestimating the power of emergent phenomenon.

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — HN submission

Wed, 15 Feb 2023 11:55:42 GMT

Emergent World Representations in Neural Networks

3 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by kilgnad

Wed, 15 Feb 2023 03:28:59 GMT

It's clearly early technology, so it's not perfect. But what it is able to get right is clear proof it's more then what you think: https://www.engraved.blog/building-a-virtual-machine-inside/ Read to the end. The beginning and middle doesn't show off anything too impressive. It's the very end where chatGPT displays a sort of self awareness. Also here's a scientific paper showing that LLMs are more then a chinese room: https://arxiv.org/abs/2210.13382

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by sigmoid10

Wed, 15 Feb 2023 01:27:07 GMT

Here is just one recent example: https://arxiv.org/abs/2210.13382 If you actually follow the literature, you'll find that there is tons of evidence that the seemingly "simple" transformer architecture might actually work pretty similar to the way the human brain is believed to work.

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by spion

Tue, 14 Feb 2023 22:33:28 GMT

The answer to this is: "we don't really know as its a very complex function automatically discovered by means of slow gradient descent, and we're still finding out" Here are some of the fun things we've found out so far: - GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): https://arxiv.org/abs/2212.10559

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by meow_mix

Sun, 22 Jan 2023 18:43:46 GMT

Btw this post is based on a preprint - might be good to wait a lil https://arxiv.org/pdf/2210.13382.pdf

Paper and discussion history. A mention is not an endorsement.

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task — mentioned by pmontra

Sun, 22 Jan 2023 08:51:33 GMT

I read the web page and the PDF paper quickly (https://arxiv.org/pdf/2210.13382.pdf) and it seems that there are no examples of the prompts they used. It would be great to see which technique they are using or is it obvious for experts of the field? It would help with reproducibility too.

Paper and discussion history. A mention is not an endorsement.