Sparse Universal Transformer · HackerPapers discussions

Sparse Universal Transformer — mentioned by namibj

Wed, 09 Sep 2026 16:44:44 GMT

Oh, is the principle of sparse universal transformers finally in SoTA LLMs? I guess we did manage to eventually seriously crash into the wall "more compute than normal (non-looped/unique-weights) transformers can efficiently consume with the limited training data we have", plus massive focus on highly hands-off agentic tool use reasoning... https://arxiv.org/abs/2310.07096 Edit: read much of the article, it's brute force predecessor was explicitly called out as an almost-ancient example: > The looped transformer is nothing new, and the basic idea already appeared in the Universal Transformers…

Paper and discussion history. A mention is not an endorsement.

Sparse Universal Transformer — mentioned by shawntan

Wed, 08 Oct 2025 04:23:47 GMT

I should probably also add: It's long been known that Universal / Recursive Transformers are able to solve _simple_ synthetic tasks that vanilla transformers cannot. Just check out the original UT paper, or some of it's follow ups: Neural Data Router, https://arxiv.org/abs/2110.07732; Sparse Universal Transformers (SUT), https://arxiv.org/abs/2310.07096. There is even theoretical justification for why: https://arxiv.org/abs/2503.03961 The challenge is actually scaling them up to be useful as LLMs as well (I describe why it's a challenge in the SUT paper). It's hard to say with the way ARC-AGI…

Paper and discussion history. A mention is not an endorsement.