Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel · HackerPapers discussions

Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel — mentioned by antics

Wed, 14 Oct 2026 06:08:38 GMT

For those who are not math people, this comment is referring to the fact that "attention" is formally equivalent[1] to Nadaraya-Watson kernel smoothing, which is a kind of weighted average. Yes! Incredibly, attention is just a weighted average. Imagine you have a series of (x,y) coordinates. You receive a new x coordinate, and you'd like to predict the corresponding y. So, the absolute dumbest thing you can do is find the nearest x, and then report its corresponding y as your prediction. The second dumbest thing you can do is find the n closest x coordinates, and then average their y values…

Paper and discussion history. A mention is not an endorsement.

Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel — mentioned by antics

Tue, 08 Apr 2025 22:46:30 GMT

I just meant that there are lots of problems where I think TCS could have contributed but didn't, because the relevant analysis was not acceptable at FOCS or SODA (or whatever). A good example is the original transformers paper, which is vastly over-complicated relative to (say) a later treatment[1] by Ruslan Salakhutdinov's lab, which shows that attention is "just" plain-old Nadaraya-Watson kernel smoothing—which if you don't know is a weighted average of points covered in elementary grad stats. I'm not saying it was TCS people would be "better" at discovering this or anything like that, I'm…

Paper and discussion history. A mention is not an endorsement.

Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel — mentioned by jackblemming

Mon, 05 Feb 2024 02:11:57 GMT

> the topics in the parent post should not be a major surprise to anyone who has read https://people.math.harvard.edu/~ctm/home/text/others/shanno... ! > which clearly explains it (and said emergent phenomena) Very smart information theory people have looked at neural networks through the lens of information theory and published famous papers about it years ago. It couldn't explain many things about neural networks, but it was interesting nonetheless. FWIW it's not uncommon for smart people to say "this mathematical structure looks like this other idea with [+/- some structure]!!" and that it…

Paper and discussion history. A mention is not an endorsement.