Rethinking Attention: Polynomial Alternatives to Softmax in Transformers · HackerPapers discussions

Rethinking Attention: Polynomial Alternatives to Softmax in Transformers — mentioned by D-Machine

Sat, 17 Oct 2026 15:01:10 GMT

Yes, the main problem with this whole way of thinking about transformers and self-attention is that when you look at how they are actually implemented (i.e. what Q, K, and V are applied to at each layer, and consider the loss function during training), then the analogy to memory quickly starts to look tenuous and far-removed from what is actually happening. Mainly, since self-attention layers operate on deeply-stacked representations of the outputs of previous layers, and since training has a loss function that doesn't make the updates work in the way the article says, the analogy to memory…

Paper and discussion history. A mention is not an endorsement.

Rethinking Attention: Polynomial Alternatives to Softmax in Transformers — mentioned by D-Machine

Thu, 08 Jan 2026 06:54:17 GMT

1. I do not think it is orthogonal, but, regardless, there is plenty of research trying to get explainability out of all aspects of scaled dot-product attention layers (weights, QKV projections, activations, other aspects), and trying to explain deep models generally via sort of bottom-up mechanistic approaches. I think it can be clearly argued this does not give us much and is probably a waste of time (see e.g. https://ai-frontiers.org/articles/the-misguided-quest-for-me...). I think this is especially clear when you have evidence (in research, at least) that other mechanisms and layers can…

Paper and discussion history. A mention is not an endorsement.

Rethinking Attention: Polynomial Alternatives to Softmax in Transformers — HN submission

Sun, 27 Oct 2024 03:42:04 GMT

Rethinking Softmax: Self-Attention with Polynomial Activations

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.