Universal Transformers · HackerPapers discussions

Universal Transformers — mentioned by omneity

Tue, 23 Dec 2025 12:51:46 GMT

Thanks, this was helpful! Reading the seminal paper[0] on Universal Transformers also gave some insights: > UTs combine the parallelizability and global receptive field of feed-forward sequence models like the Transformer with the recurrent inductive bias of RNNs. Very interesting, it seems to be an “old” architecture that is only now being leveraged to a promising extent. Curious what made it an active area (with the works of Samsung and Sapient and now this one), perhaps diminishing returns on regular transformers? 0: https://arxiv.org/abs/1807.03819

Paper and discussion history. A mention is not an endorsement.

Universal Transformers — mentioned by atrudeau

Wed, 08 Oct 2025 15:33:34 GMT

Very cool to see the effectiveness of recurrence on ARC. For those interested in recurrence, here are other works that leverage a similar approach for other types of problems: Language modeling: Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach https://arxiv.org/pdf/2502.05171 Puzzle solving: A Simple Loss Function for Convergent Algorithm Synthesis using RNNs https://openreview.net/pdf?id=WaAJ883AqiY End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinking https://arxiv.org/abs/2202.05826 Can You Learn an Algorithm?…

Paper and discussion history. A mention is not an endorsement.

Universal Transformers — mentioned by shawntan

Tue, 07 Oct 2025 20:08:31 GMT

Right. There should really be a vanilla Transformer baseline. With recurrence: The idea has been around: https://arxiv.org/abs/1807.03819 There are reasons why it hasn't really been picked up at scale, and the method tends to do well on synthetic tasks.

Paper and discussion history. A mention is not an endorsement.

Universal Transformers — mentioned by ed

Sun, 17 Aug 2025 00:10:15 GMT

TL;DR the “high” “low” level modules don’t matter much, and the rest of the architecture is similar to Universal Transformers https://arxiv.org/abs/1807.03819

Paper and discussion history. A mention is not an endorsement.

Universal Transformers — mentioned by lucidrains

Sun, 19 Jul 2020 19:00:26 GMT

https://arxiv.org/abs/1807.03819

Paper and discussion history. A mention is not an endorsement.

Universal Transformers — mentioned by kuu

Mon, 16 Sep 2019 12:55:14 GMT

In case you find Attention (and specially transformers) interesting I have some saved links introducing the topic: http://www.peterbloem.nl/blog/transformers https://nostalgebraist.tumblr.com/post/185326092369/the-tran... https://papers.nips.cc/paper/7181-attention-is-all-you-need.... http://jalammar.github.io/illustrated-transformer/ https://arxiv.org/pdf/1807.03819.pdf

Paper and discussion history. A mention is not an endorsement.

Universal Transformers — mentioned by solomatov

Sat, 10 Nov 2018 19:37:37 GMT

- Neural turing machines: https://arxiv.org/abs/1410.5401 - Universal transformers: https://arxiv.org/abs/1807.03819 These are just 2 of most well known.

Paper and discussion history. A mention is not an endorsement.