The Expressive Power of Transformers with Chain of Thought · HackerPapers discussions

The Expressive Power of Transformers with Chain of Thought — mentioned by shawntan

Wed, 09 Sep 2026 17:49:05 GMT

For the research focused, there are some references in my blogpost here on what kinds of computational problems minimally require how much CoT to solve: https://blog.wtf.sg/posts/2023-02-03-the-new-xor-problem/ Notably Will Merrill's work: https://arxiv.org/abs/2310.07923 As for how universal transformers (looping transformers, but everyone has since forgotten prior work) will affect this, Will Merrill (again) has a paper here (https://arxiv.org/abs/2503.03961) that discusses exactly this. The original universal transformers is called "universal" because if you allow for per-token looping…

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by logicchains

Mon, 10 Aug 2026 19:15:02 GMT

>why wouldn't that same model be able to find ways to improve it's own algorithms beyond that of the capabilities of human mathematicians / ML researchers Because algorithms have lower bounds, and the computational characteristics of LLMs are well-characterized by papers like https://arxiv.org/abs/2310.07923 . No amount of intelligence can make something faster than a mathematically-proven lower bound, any more than it could make 1+1=3 (that's why every single successful production transformer architecture has some form of O(N^2) attention layers, because it's mathematically impossible to…

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by anonymid

Mon, 16 Feb 2026 04:02:14 GMT

Thanks for reading, and I really appreciate your comments! > who feed their produced tokens back as inputs, and whose tuning effectively rewards it for doing this skillfully Ah, this is a great point, and not something that I considered. I agree that the token feedback does change the complexity, and it seems that there's even a paper by the same authors about this very thing! https://arxiv.org/abs/2310.07923 I'll have to think on how that changes things. I think it does take the wind out of the architecture argument as it's currently stated, or at least makes it a lot more challenging. I'll…

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by red75prime

Tue, 30 Sep 2025 09:02:04 GMT

The term for research on whether LLMs can learn and execute this or that algorithm of program synthesis? I guess there's none. If we do know the algorithm, it's better to teach LLM to write a program that outputs a program. If we don't know the algorithm, we can't reason whether LLM can do it. > even if someone discovered that LLMs can only ever emit/recognize context-sensitive languages https://arxiv.org/abs/2310.07923 Their recognition abilities are limited by context-sensitive languages, if the number of steps in chain-of-thought is linear in the length of input. All in all, asymptotic…

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by logicchains

Wed, 03 Sep 2025 21:06:10 GMT

LLMs are not formally equivalent to Markov chains, they're more powerful; transformers with sufficient chain of thought can solve any problem in P: https://arxiv.org/abs/2310.07923.

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by logicchains

Tue, 24 Jun 2025 16:14:45 GMT

We can be sure via analysis based on computational theory, e.g. https://arxiv.org/abs/2503.03961 and https://arxiv.org/abs/2310.07923 . This lets us know what classes of problems a model is able to solve, and sufficiently deep transformers with chain of thought have been shown to be theoretically capable of solving a very large class of problems.

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by inciampati

Sun, 19 Jan 2025 13:11:18 GMT

Yes it is, because the post discussed this approach precisely because unrolling the actual chain of thought in interactive chat does not work. And it's doubly relevant because chain of thought let's transformers break out of TCO complexity and be UTM. This matters because TC0 is pattern matching while UTM is general intelligence. Forgetting what the model thought breaks this and (ironically) probably forces the model back into one-shot pattern matching. https://arxiv.org/abs/2310.07923

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by Isinlor

Thu, 02 Jan 2025 07:50:47 GMT

When Can Transformers Count to n? https://arxiv.org/html/2407.15160v2 The Expressive Power of Transformers with Chain of Thought https://arxiv.org/html/2310.07923v5 Transformer needs to retrieve letters per each token while forced to keep internal representation still aligned in length with the base tokens (each token also has finite embedding, while made out of multiple letters), and then it needs to count the letters within misaligned representation. Autoregressive mode completely alleviate the problem as it can align its internal representation with the letters and it can just keep…

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by logicchains

Sun, 15 Dec 2024 11:38:32 GMT

An LLM with chain of thought and unbounded compute/context can run any program in PTIME: https://arxiv.org/abs/2310.07923 , which is a huge class of programs.

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by logicchains

Sat, 14 Dec 2024 11:13:38 GMT

> that we are not guaranteed to be able to replicate all traits of human intelligence by brute forcing. We know from complexity theory that transformers with chain of thought are guaranteed to be able to reproduce a significant fraction of human reasoning, anything in the complexity class PTIME: https://arxiv.org/abs/2310.07923

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by aithrowawaycomm

Fri, 15 Nov 2024 00:44:28 GMT

I think the more obvious explanation has to do with computational complexity: counting is an O(n) problem, but transformer LLMs can’t solve O(n) problems unless you use CoT prompting: https://arxiv.org/abs/2310.07923

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by aithrowawaycomm

Fri, 15 Nov 2024 00:40:01 GMT

I am aware of errors in computations that can be fixed by better tokenization (e.g. long addition works better tokenizing right-left rather than L-R). But I am talking about counting, and talking about counting words, not characters. I don’t think tokenization explains why LLMs tend to fail at this without CoT prompting. I really think the answer is computational complexity: counting is simply too hard for transformers unless you use CoT. https://arxiv.org/abs/2310.07923

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by logicchains

Tue, 17 Sep 2024 13:21:06 GMT

>How would that be remarkable, when it is exactly what he Universal Approximation Theorem already states Only with infinite precision, which is highly unrealistic. Under realistic assumptions, fixed depth transformer without chain-of-thought are very limited in what they can express: https://arxiv.org/abs/2207.00729 . Chain of thought increases the class of problems which fixed depth transformers can solve: https://arxiv.org/abs/2310.07923

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by shawntan

Tue, 17 Sep 2024 02:11:08 GMT

Theoretical results exist that try to quantify the number of CoT tokens needed to reach different levels of computational expressibility: https://arxiv.org/pdf/2310.07923 TL;DR: Getting to Turing completeness can require polynomial CoT tokens, wrt the input problem size. For a field that constantly harps on parallelism and compute efficiency, this requirement seems prohibitive. We really need to get away from constant depth architectures.

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by logicchains

Wed, 04 Sep 2024 15:02:16 GMT

The theoretical foundation of transformers is well understood; they're able to approximate a very wide family of functions, particularly with chain of thought ( https://arxiv.org/abs/2310.07923 ). Training them on next-token-prediction is essentially training them to compress, and more optimal compression requires a more accurate model of the world, so they're being trained to model the world better and better. However you want to define intelligence, for practical purposes models with better and better models of the world are more and more useful.

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by logicchains

Mon, 02 Sep 2024 07:04:58 GMT

Transformers with chain of thought can use FOL: https://arxiv.org/abs/2310.07923

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by logicchains

Fri, 07 Jun 2024 19:48:15 GMT

Interestingly LLMs also do better with pen and paper reasoning: https://arxiv.org/abs/2310.07923 .

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — mentioned by logicchains

Wed, 05 Jun 2024 17:16:13 GMT

>For reasoning of any complexity, some sort of internal monologue and iteration is needed. From a formal perspective you're entirely correct. Transformers with chain-of-thought are strictly more powerful than transformers without it, and can efficiently solve classes of problems that would otherwise require exponentially increasing model depth: https://arxiv.org/abs/2310.07923

Paper and discussion history. A mention is not an endorsement.

The Expressive Power of Transformers with Chain of Thought — HN submission

Sun, 24 Mar 2024 09:01:46 GMT

The Expressive Power of Transformers with Chain of Thought (Revised)

20 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.