Emergent Linear Representations in World Models of Self-Supervised Sequence Models · HackerPapers discussions

Emergent Linear Representations in World Models of Self-Supervised Sequence Models — mentioned by mitthrowaway2

Thu, 19 Dec 2024 19:37:04 GMT

Transformers, like other deep neural networks, have many hidden layers before the output. Are you certain that those hidden layers aren't modeling the world first before choosing an output token? Deep neural networks (including transformers) trained on board games have been found to develop an internal representation of the board state. (eg. https://arxiv.org/pdf/2309.00941)

Paper and discussion history. A mention is not an endorsement.

Emergent Linear Representations in World Models of Self-Supervised Sequence Models — mentioned by narush

Thu, 07 Nov 2024 11:15:42 GMT

I’ve replicated the OthelloGPT results mentioned in this paper personally - and it def felt like the next-move-only accuracy metric was not everything. Indeed, the authors of the original paper knew this, and so further validated the world model by intervening in a model’s forward pass to directly manipulate the world model (and check the resulting change in valid move predictions). I’d also recommend checking out Neel Nanda’s work on OthelloGPT, where he demonstrated the world model was actually linear: https://arxiv.org/abs/2309.00941

Paper and discussion history. A mention is not an endorsement.

Emergent Linear Representations in World Models of Self-Supervised Sequence Models — mentioned by bubblyworld

Sun, 21 Jul 2024 16:31:31 GMT

Here's the paper on OthelloGPT's internal models I mentioned: https://arxiv.org/abs/2309.00941 The references in that paper are also good reading!

Paper and discussion history. A mention is not an endorsement.