Attention Residuals · HackerPapers discussions

Attention Residuals — HN submission

Tue, 09 Jun 2026 17:18:43 GMT

Attention Residuals

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Attention Residuals — HN submission

Tue, 07 Apr 2026 03:34:58 GMT

Attention Residuals

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Attention Residuals — HN submission

Thu, 02 Apr 2026 08:53:54 GMT

Attention Residuals

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Attention Residuals — mentioned by westurner

Fri, 20 Mar 2026 19:54:37 GMT

ScholarlyArticle: "Attention Residuals" (2026) https://arxiv.org/abs/2603.15031 : > Abstract: Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer's contribution. We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights. To address…

Paper and discussion history. A mention is not an endorsement.

Attention Residuals — HN submission

Tue, 17 Mar 2026 10:51:39 GMT

Kimi introduces Attention Residuals: 1.25x compute performance at <2% overhead

9 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.