What Can Transformers Learn In-Context? A Case Study of Simple Function Classes · HackerPapers discussions

What Can Transformers Learn In-Context? A Case Study of Simple Function Classes — mentioned by pama

Thu, 20 Nov 2025 19:42:09 GMT

LLMs have the ability to learn certain classes of algorithms from their datasets in order to reduce errors when compressing their pretraining data. If you are technically inclined, read the reference: https://arxiv.org/abs/2208.01066 (optionally followup work) to see how llms can pick up complicated algorithms from training on examples that could have been generated by such algorithms (in one of the cases the LLM is better than anything we know; in the rest it is simply just as good as our best algos). Learning such functions from data would not work with Markov chains at any level of…

Paper and discussion history. A mention is not an endorsement.

What Can Transformers Learn In-Context? A Case Study of Simple Function Classes — mentioned by pama

Fri, 03 Jan 2025 15:23:58 GMT

> I wish people would understand what a large language model is. I think your view of llm does not explain the learning of algorithms that these constructs are clearly capable of, see for example: https://arxiv.org/abs/2208.01066 More generally, the best way to compress information from too many different coding examples is to figure out how to code rather than try to interpolate between existing blogs and QA forums. My own speculation is that with additional effort during training (RL or active learning in the training loop) we will probably reach superhuman coding performance within two…

Paper and discussion history. A mention is not an endorsement.

What Can Transformers Learn In-Context? A Case Study of Simple Function Classes — mentioned by spyder

Tue, 11 Apr 2023 08:48:37 GMT

I agree, the in-context, few shot learning is the most interesting thing a about these transformer models, they have basically learned to learn (meta-learning). Some additional interesting papers about it: Transformers learn in-context by gradient descent: https://arxiv.org/abs/2212.07677 What Can Transformers Learn In-Context? A Case Study of Simple Function Classes: https://arxiv.org/abs/2208.01066 "We also show that we can train Transformers to in-context learn more complex function classes -- namely sparse linear functions, two-layer neural networks, and decision trees -- with performance…

Paper and discussion history. A mention is not an endorsement.

What Can Transformers Learn In-Context? A Case Study of Simple Function Classes — mentioned by spyder

Sun, 12 Feb 2023 12:08:23 GMT

Yea, this is an example of the transfomer models' ability of in-context learning, they just don't incorporate them into their long-term fixed memory yet, but I imagine it's the next step. Some interesting recent papers about the in-context learning: https://www.lesswrong.com/posts/firtXAWGdvzXYAh9B/paper-tran... What Can Transformers Learn In-Context? A Case Study of Simple Function Classes: https://arxiv.org/abs/2208.01066 Transformers learning to learn (meta-learning): https://openreview.net/forum?id=t6tA-KB4dO

Paper and discussion history. A mention is not an endorsement.