Saturated Transformers are Constant-Depth Threshold Circuits — mentioned by bmc7505

Sat, 17 Jun 2023 15:44:53 GMT

The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. One way to think of why this is the case is because transformers are learning shortcuts to multistep reasoning tasks [1, 2], but do not learn to reason systematically and unsurprisingly fail to generalize when presented with more challenging tasks. The proper way to evaluate systematic reasoning is by training on a curriculum of problems of increasing average-case complexity. Think random k-SAT in the critical region, bisimulation games, or interpolation in…

Paper and discussion history. A mention is not an endorsement.

Saturated Transformers are Constant-Depth Threshold Circuits — mentioned by shawntan

Tue, 18 Apr 2023 05:41:13 GMT

There's more detailed examination of Transformers from circuit complexity in this paper: https://arxiv.org/abs/2106.16213 There are a few references also in the blog post.

Paper and discussion history. A mention is not an endorsement.

Saturated Transformers are Constant-Depth Threshold Circuits · HackerPapers discussions

Saturated Transformers are Constant-Depth Threshold Circuits — mentioned by bmc7505

Saturated Transformers are Constant-Depth Threshold Circuits — mentioned by shawntan