Transformers Learn Shortcuts to Automata · HackerPapers discussions

Transformers Learn Shortcuts to Automata — mentioned by bmc7505

Sat, 17 Jun 2023 15:44:53 GMT

The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. One way to think of why this is the case is because transformers are learning shortcuts to multistep reasoning tasks [1, 2], but do not learn to reason systematically and unsurprisingly fail to generalize when presented with more challenging tasks. The proper way to evaluate systematic reasoning is by training on a curriculum of problems of increasing average-case complexity. Think random k-SAT in the critical region, bisimulation games, or interpolation in…

Paper and discussion history. A mention is not an endorsement.

Transformers Learn Shortcuts to Automata — mentioned by fithisux

Mon, 05 Jun 2023 11:34:24 GMT

My 2 cents https://medium.com/mlearning-ai/understanding-and-coding-the... https://arxiv.org/pdf/2210.10749.pdf

Paper and discussion history. A mention is not an endorsement.

Transformers Learn Shortcuts to Automata — HN submission

Wed, 26 Apr 2023 04:42:24 GMT

Transformers Learn Shortcuts to Automata

37 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Transformers Learn Shortcuts to Automata — mentioned by sinenomine

Tue, 10 Jan 2023 01:22:27 GMT

Ironic, since the relatively recently discovered double descent makes it clear that bias-variance tradeoff as we know it from statistical learning theory simply doesn't apply to "overparameterized" deep models. Much of old theory is barely applicable and people are, understandably, bewildered and in denial. If someone were to be inclined to theory, I'd just recommend reading papers that don't try oversimplify the domain: https://arxiv.org/abs/2006.15191 https://arxiv.org/abs/2210.10749 https://arxiv.org/abs/2205.10343 https://arxiv.org/abs/2105.04026

Paper and discussion history. A mention is not an endorsement.