A Theory for Emergence of Complex Skills in Language Models · HackerPapers discussions

A Theory for Emergence of Complex Skills in Language Models — mentioned by neerajk

Tue, 28 Jul 2026 18:37:52 GMT

There is a Sanjeev Arora paper "A Theory for Emergence of Complex Skills in Language Models" (https://arxiv.org/pdf/2307.15936) on this subject. The key idea is there is cross entropy (how "surprised" the model is with the "correct" next token, lower is better), some of which is inherent in the language and therefore unavoidable, and the rest is model error, and that this portion of the cross entropy is reduced with scaling. And as scaling reduces a model's excess entropy, the model can become good at combinations of skills much faster than you would expect if it had to separately see and…

Paper and discussion history. A mention is not an endorsement.

A Theory for Emergence of Complex Skills in Language Models — mentioned by visarga

Tue, 23 Jan 2024 17:25:06 GMT

This article is 3 years old, right before GPT-3's launch. So it doesn't have all the recent info. We're not using meta-learning much, the problem of using few examples to train a model has been solved by few-shot prompting, zero shot, or fine-tuning with a small dataset. The problem with meta-learning is that it tries to back-propagate through the inner loop (the model) to the outer loop (meta-model), and this is hard to do. Since LLMs have such huge and diverse training sets, almost everything is in-distribution. LLMs are conditional probability distributions that can be steered with prompts…

Paper and discussion history. A mention is not an endorsement.

A Theory for Emergence of Complex Skills in Language Models — mentioned by svat

Mon, 22 Jan 2024 19:02:57 GMT

What I understood: they assume a hypothetical bipartite graph with text nodes (like hypothetical paragraphs / chunks of texts) on one side, and “skill” nodes on the other. The skills could be things like: • ability to understand the word “because” [causality] • ability to divide two numbers • ability to detect irony Each text node is connected to all the skill nodes that are required to understand it. (This is a hypothetical graph, so we don't need to be able to write down the graph precisely, or define what “understand” or “required” mean here.) Further, each text node is either “successful”…

Paper and discussion history. A mention is not an endorsement.

A Theory for Emergence of Complex Skills in Language Models — mentioned by blackbear_

Mon, 22 Jan 2024 18:19:39 GMT

Clearly no link to the actual research, so here it is: https://arxiv.org/abs/2307.15936

Paper and discussion history. A mention is not an endorsement.