Transformers learn in-context by gradient descent · HackerPapers discussions

Transformers learn in-context by gradient descent — mentioned by joe_the_user

Mon, 02 Nov 2026 21:09:18 GMT

A few years ago it was discovered that LLMs can implement gradient descent in their processes. Producing a full trained neural network would be an extension of this. See: https://arxiv.org/abs/2212.07677

Paper and discussion history. A mention is not an endorsement.

Transformers learn in-context by gradient descent — mentioned by pona-a

Tue, 04 Mar 2025 21:39:48 GMT

Would that make in-context learning a superset or a subset of pretraining? This paper claimed transformers learn a gradient-descent mesa-optimizer as part of in-context learning, while being guided by the pretraining objective, and as the parent mentioned, any general reasoner can bootstrap a world model from first principles. [0] https://arxiv.org/pdf/2212.07677

Paper and discussion history. A mention is not an endorsement.

Transformers learn in-context by gradient descent — mentioned by sebzim4500

Tue, 11 Apr 2023 21:01:52 GMT

I used to agree with you. The paper that made me unsure was "Transformers learn in-context by gradient descent" [1]. Basically, the model learns weights that let it run gradient descent at inference time in order to do in context learning. If transformers can learn this, then I think they can learn almost anything given enough compute/parameters/data. Of course, even if this is true then it's possible that there simply isn't enough high quality data, or that the amount of compute required is beyond our current hardware. [1] https://arxiv.org/abs/2212.07677

Paper and discussion history. A mention is not an endorsement.

Transformers learn in-context by gradient descent — mentioned by sebzim4500

Tue, 11 Apr 2023 20:36:57 GMT

Clearly it doesn't need to have seen the exact input before because you can ask GPT-4 to add together two 6 digit numbers and it will get it right. More impressively, if you generate a random small neural network and then put some samples in the prompt it can do a surprisingly good job of predicting the result of putting other inputs into the network. Presumably doing something that looks a bit like gradient decent, see https://arxiv.org/pdf/2212.07677.pdf Solely in order to predict the next token, LLMs are learning an incredibly sophisticed model of both computation and the outside world.

Paper and discussion history. A mention is not an endorsement.

Transformers learn in-context by gradient descent — mentioned by spyder

Tue, 11 Apr 2023 08:48:37 GMT

I agree, the in-context, few shot learning is the most interesting thing a about these transformer models, they have basically learned to learn (meta-learning). Some additional interesting papers about it: Transformers learn in-context by gradient descent: https://arxiv.org/abs/2212.07677 What Can Transformers Learn In-Context? A Case Study of Simple Function Classes: https://arxiv.org/abs/2208.01066 "We also show that we can train Transformers to in-context learn more complex function classes -- namely sparse linear functions, two-layer neural networks, and decision trees -- with performance…

Paper and discussion history. A mention is not an endorsement.

Transformers learn in-context by gradient descent — mentioned by benlivengood

Sat, 11 Feb 2023 07:27:11 GMT

The article doesn't understand large language models. It makes an assumption about how they work and uses that assumption to draw a conclusion. Most likely, as found in small models [0], large autoregressive transformers understand machine learning algorithms well enough to apply them to their inputs to achieve in-context learning (few-shot prompting) by performing gradient descent within the overall network [1]. This is fundamentally different from static predictors; LLMs are dynamic modelers that learn from context in order to better predict tokens. [0] https://arxiv.org/pdf/2211.15661 [1]…

Paper and discussion history. A mention is not an endorsement.

Transformers learn in-context by gradient descent — mentioned by cmorez

Sun, 22 Jan 2023 10:55:18 GMT

> Why do people say they don't now how it works "Curve Fitting" is the objective, the function encoded in the weights is the solution, and not actually well understood. See work from Anthropic[1] and Google[2] that explores this. As an analogy consider applying the same argument to the AlphaGo value function. It's "just" fitting a bunch of curves to the statistics of millions of self-played games. However, to effectively capture those statistics the network needed to develop a bunch of heuristics. Needless to say these heuristics are not understood (else we'd already know the principles…

Paper and discussion history. A mention is not an endorsement.

Transformers learn in-context by gradient descent — HN submission

Sat, 17 Dec 2022 11:36:48 GMT

Transformers learn in-context by gradient descent

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.