Tighter Bounds on the Expressivity of Transformer Encoders · HackerPapers discussions

Tighter Bounds on the Expressivity of Transformer Encoders — mentioned by meltyness

Tue, 28 Jan 2025 03:13:31 GMT

What these guys are on about: https://arxiv.org/abs/2301.10743

Paper and discussion history. A mention is not an endorsement.

Tighter Bounds on the Expressivity of Transformer Encoders — mentioned by inciampati

Tue, 09 Jan 2024 12:05:14 GMT

Transformers with fixed precision and any amount of memory are not Turing complete. They're bounded by a variant of first order logic with counting quantifiers. https://arxiv.org/abs/2301.10743 This supports the intuitive idea that they "learn circuits for things" that I've heard a few others mention. If they have infinite precision then I suppose they can simulate a Turing machine in their infinitely precise states...

Paper and discussion history. A mention is not an endorsement.

Tighter Bounds on the Expressivity of Transformer Encoders — mentioned by meltyness

Fri, 15 Dec 2023 16:14:39 GMT

To that, what can these express, precisely, is an interesting question; so for transformer encoders: https://arxiv.org/abs/2301.10743 Another interesting research topic is the trusted generation of tasks for finetuning https://arxiv.org/abs/2306.08568 And I suppose too running these at the edge is terribly interesting too, if you can find analyses of "quantization" this is a highly active research are, and results are pretty incredible since it cuts resources by huge factors and no one quite knows why. This is one that's easy to dive into with consumer hardware, but don't know any great…

Paper and discussion history. A mention is not an endorsement.

Tighter Bounds on the Expressivity of Transformer Encoders — mentioned by bmc7505

Sat, 17 Jun 2023 15:44:53 GMT

The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. One way to think of why this is the case is because transformers are learning shortcuts to multistep reasoning tasks [1, 2], but do not learn to reason systematically and unsurprisingly fail to generalize when presented with more challenging tasks. The proper way to evaluate systematic reasoning is by training on a curriculum of problems of increasing average-case complexity. Think random k-SAT in the critical region, bisimulation games, or interpolation in…

Paper and discussion history. A mention is not an endorsement.

Tighter Bounds on the Expressivity of Transformer Encoders — mentioned by inciampati

Thu, 15 Jun 2023 10:44:36 GMT

This has been built on extensively over the past two years. For instance: Tighter Bounds on the Expressivity of Transformer Encoders https://arxiv.org/abs/2301.10743. I find it interesting that transformers are equivalent to first order logic on circuits with counters. Amazing what you can do even if you're not Turing complete!

Paper and discussion history. A mention is not an endorsement.

Tighter Bounds on the Expressivity of Transformer Encoders — HN submission

Tue, 25 Apr 2023 17:02:24 GMT

Tighter bounds on the expressivity of transformer encoders

76 points; 13 discussion replies on HN. Paper and source history. A submission is not an endorsement.