In-context Learning and Induction Heads — mentioned by HarHarVeryFunny

Fri, 26 May 2023 12:55:29 GMT

Another piece of the puzzle seems to be transformer "induction heads" where attention heads in consecutive layers work together to provide a mechanism that is believed to be responsible for much of in-context learning. The idea is that earlier instances of a token pattern/sequence in the context are used to predict the continuation of a similar pattern later on. In the most simple case this is a copying operation such that an early occurrence of AB predicts that a later A should be followed by B. In the more general case this becomes A'B' => AB which seems to be more of an analogy type…

Paper and discussion history. A mention is not an endorsement.

In-context Learning and Induction Heads · HackerPapers discussions

In-context Learning and Induction Heads — mentioned by HarHarVeryFunny