Attention Mechanisms in Computer Vision: A Survey — mentioned by D-Machine

Thu, 08 Jan 2026 02:43:32 GMT

The main takeaway is that "attention" is a much broader concept generally, so worrying too much about the "scaled dot-product attention" of transformers deeply limits your understanding of what kinds of things really matter in general. A paper I found particularly useful on this was generalizing even farther to note the importance of multiplicative interactions more generally in deep learning (https://openreview.net/pdf?id=rylnK6VtDH). EDIT: Also, this paper I was looking for dramatically generalizes the notion of attention in a way I found to be quite helpful: https://arxiv.org/pdf/2111.07624

Paper and discussion history. A mention is not an endorsement.

Attention Mechanisms in Computer Vision: A Survey — mentioned by D-Machine

Thu, 08 Jan 2026 02:34:10 GMT

Don't get caught up in interpreting QKV, it is a waste of time, since completely different attention formulations (e.g. merged attention [1]) still give you the similarities / multiplicative interactions, but may even work better [2]. EDIT: Oh and attention is much more broad than scaled dot-product attention [3]. [1] https://www.emergentmind.com/topics/merged-attention [2] https://blog.google/innovation-and-ai/technology/developers-... [3] https://arxiv.org/abs/2111.07624

Paper and discussion history. A mention is not an endorsement.

Attention Mechanisms in Computer Vision: A Survey · HackerPapers discussions

Attention Mechanisms in Computer Vision: A Survey — mentioned by D-Machine

Attention Mechanisms in Computer Vision: A Survey — mentioned by D-Machine