The Parallelism Tradeoff: Limitations of Log-Precision Transformers · HackerPapers discussions

The Parallelism Tradeoff: Limitations of Log-Precision Transformers — mentioned by logicchains

Tue, 17 Sep 2024 13:22:27 GMT

Only NNs of infinite size or precision. Under more realistic assumptions, transformers without chain of thought are actually limited in what they can solve: https://arxiv.org/abs/2207.00729

Paper and discussion history. A mention is not an endorsement.

The Parallelism Tradeoff: Limitations of Log-Precision Transformers — mentioned by logicchains

Tue, 17 Sep 2024 13:21:06 GMT

>How would that be remarkable, when it is exactly what he Universal Approximation Theorem already states Only with infinite precision, which is highly unrealistic. Under realistic assumptions, fixed depth transformer without chain-of-thought are very limited in what they can express: https://arxiv.org/abs/2207.00729 . Chain of thought increases the class of problems which fixed depth transformers can solve: https://arxiv.org/abs/2310.07923

Paper and discussion history. A mention is not an endorsement.

The Parallelism Tradeoff: Limitations of Log-Precision Transformers — mentioned by logicchains

Fri, 07 Jun 2024 17:29:28 GMT

If you read the rest of the paper it's clear what they mean. SSMs are theoretically no more powerful than transformers, meaning without chain of thought they can't efficiently solve state-tracking problems, unlike RNNs: https://arxiv.org/abs/2207.00729 .

Paper and discussion history. A mention is not an endorsement.

The Parallelism Tradeoff: Limitations of Log-Precision Transformers — mentioned by nyrikki

Mon, 25 Mar 2024 16:04:45 GMT

FFT is easy fairly easy for transformers to find as a pattern. Note that OpenAI admits that pre-training (pattern finding and matching) is where the abilities on professional tests comes from. Even proofs in presburger arithmetic have an upper bound on the second level of the polynomial hierarchy, which is way beyond the computational power of any LLM https://arxiv.org/abs/2207.00729 >The model’s capabilities on exams appear to stem primarily from the pre-training process and are not significantly affected by RLHF. On multiple choice questions, both the base GPT-4 model and the RLHF model…

Paper and discussion history. A mention is not an endorsement.

The Parallelism Tradeoff: Limitations of Log-Precision Transformers — mentioned by lambdaviking

Fri, 26 May 2023 21:22:19 GMT

Good question! A key use for this type of analysis is that it acts as an upper bound on transformers, rather just demonstrating "transformers are like logic". That is, it allows us to fairly easily identify problems that transformers cannot solve. Some examples are graph connectivity, linear programming, various forms of state tracking, etc. Definitely finding negative results like this is tough by just doing experiments, since it could be that the model might have succeeded if you had just trained it for more time/made it bigger/etc. Our prior paper discusses how this works more in depth…

Paper and discussion history. A mention is not an endorsement.

The Parallelism Tradeoff: Limitations of Log-Precision Transformers — mentioned by shawntan

Thu, 18 May 2023 18:30:24 GMT

But "don't try to codify 'insight' into the process" seems to suggest "don't try different approaches". I'm not sure how people can at once trot out the "Bitter Lesson" and interpret it as it is written, but still say "We're not saying not to think about new approaches". Is the idea then to work only on methods that allow for faster compute of more data? FWIW, the Transformer works faster on current methods of parallelisation, allowing for dramatic scaling that RNNs will find hard to compete on. But we do pay for that in terms of what can be computed (https://arxiv.org/pdf/2207.00729.pdf…

Paper and discussion history. A mention is not an endorsement.