The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization — mentioned by shawntan

Wed, 08 Oct 2025 04:23:47 GMT

I should probably also add: It's long been known that Universal / Recursive Transformers are able to solve _simple_ synthetic tasks that vanilla transformers cannot. Just check out the original UT paper, or some of it's follow ups: Neural Data Router, https://arxiv.org/abs/2110.07732; Sparse Universal Transformers (SUT), https://arxiv.org/abs/2310.07096. There is even theoretical justification for why: https://arxiv.org/abs/2503.03961 The challenge is actually scaling them up to be useful as LLMs as well (I describe why it's a challenge in the SUT paper). It's hard to say with the way ARC-AGI…

Paper and discussion history. A mention is not an endorsement.

The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization · HackerPapers discussions

The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization — mentioned by shawntan