A Theory of Emergent In-Context Learning as Implicit Structure Induction — mentioned by FeepingCreature

Wed, 05 Apr 2023 06:04:10 GMT

The relevant keyword you want is "zero-shot learning". (EDIT: Correction; "in-context learning". Sorry for that.) LLMs can pick up patterns from the context window purely at evaluation time using dynamic reinforcement learning. (This is one of those capabilities models seem to just pick up naturally at sufficient scale.) Those patterns are ephemeral and not persisted to memory, which I agree makes LLMs less general than humans, but that seems a weak objection to hang a fundamental difference in kind on. edit: Correction: I can't find a source for my claim that the model specifically picks up…

Paper and discussion history. A mention is not an endorsement.

A Theory of Emergent In-Context Learning as Implicit Structure Induction · HackerPapers discussions

A Theory of Emergent In-Context Learning as Implicit Structure Induction — mentioned by FeepingCreature