LAuReL: Learned Augmented Residual Layer — mentioned by cpldcpu

Mon, 12 Jan 2026 15:49:22 GMT

May be worth pointing out, that this is not the first residual connection innovation to be in production. Gemma 3n is also using a low-rank projection of the residual stream called LAuReL. Google did not publicize this too much, I noted it when poking around in the model file. https://arxiv.org/pdf/2411.07501v3 https://old.reddit.com/r/LocalLLaMA/comments/1kuy45r/gemma_3... Seems to be what they call LAuReL-LR in the paper, with D=2048 and R=64

Paper and discussion history. A mention is not an endorsement.

LAuReL: Learned Augmented Residual Layer · HackerPapers discussions

LAuReL: Learned Augmented Residual Layer — mentioned by cpldcpu