Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws — mentioned by hackpert

Thu, 08 Jan 2026 12:30:29 GMT

These metaphorical database analogies bug me, and from what it seems like, a lot of other people in comments! So far some of the most reasonable explanations I have found that take training dynamics into account are from Lenka Zdeborova's lab (albeit in toy, linear attention settings but it's easy to see why they generalize to practical ones). For instance, this is a lovely paper: https://arxiv.org/abs/2509.24914

Paper and discussion history. A mention is not an endorsement.

Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws · HackerPapers discussions

Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws — mentioned by hackpert