A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers · HackerPapers discussions

A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers — mentioned by shawntan

Wed, 09 Sep 2026 17:49:05 GMT

For the research focused, there are some references in my blogpost here on what kinds of computational problems minimally require how much CoT to solve: https://blog.wtf.sg/posts/2023-02-03-the-new-xor-problem/ Notably Will Merrill's work: https://arxiv.org/abs/2310.07923 As for how universal transformers (looping transformers, but everyone has since forgotten prior work) will affect this, Will Merrill (again) has a paper here (https://arxiv.org/abs/2503.03961) that discusses exactly this. The original universal transformers is called "universal" because if you allow for per-token looping…

Paper and discussion history. A mention is not an endorsement.

A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers — mentioned by shawntan

Wed, 08 Oct 2025 04:23:47 GMT

I should probably also add: It's long been known that Universal / Recursive Transformers are able to solve _simple_ synthetic tasks that vanilla transformers cannot. Just check out the original UT paper, or some of it's follow ups: Neural Data Router, https://arxiv.org/abs/2110.07732; Sparse Universal Transformers (SUT), https://arxiv.org/abs/2310.07096. There is even theoretical justification for why: https://arxiv.org/abs/2503.03961 The challenge is actually scaling them up to be useful as LLMs as well (I describe why it's a challenge in the SUT paper). It's hard to say with the way ARC-AGI…

Paper and discussion history. A mention is not an endorsement.

A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers — mentioned by NitpickLawyer

Thu, 14 Aug 2025 16:13:17 GMT

There's prior research that finds a connection between model depth and "reasoning" ability - https://arxiv.org/abs/2503.03961 A depth of 4 is very small. It is very much a toy model. It's ok to research this, and maybe someone will try it out on larger models, but it's totally not ok to lead with the conclusion, based on this toy model, IMO.

Paper and discussion history. A mention is not an endorsement.

A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers — mentioned by logicchains

Tue, 12 Aug 2025 09:41:26 GMT

The extrapolation doesn't work if the transformer is too shallow (too few layers) relative to sequence length, because of https://arxiv.org/abs/2503.03961 . A bunch of tasks become unfeasible when the layer count is too low, and 4 layers is way too low. I.e. linearly increasing the number of layers in a model can result in a superlinear increase in performance on tasks like reasoning.

Paper and discussion history. A mention is not an endorsement.

A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers — mentioned by logicchains

Tue, 24 Jun 2025 16:14:45 GMT

We can be sure via analysis based on computational theory, e.g. https://arxiv.org/abs/2503.03961 and https://arxiv.org/abs/2310.07923 . This lets us know what classes of problems a model is able to solve, and sufficiently deep transformers with chain of thought have been shown to be theoretically capable of solving a very large class of problems.

Paper and discussion history. A mention is not an endorsement.