Query-Key Normalization for Transformers · HackerPapers discussions

Query-Key Normalization for Transformers — mentioned by kouteiheika

Fri, 23 May 2025 20:03:30 GMT

> Let's look at some of the most important ones that have been developed over the years and try to implement the basic ideas as succinctly as possible. One big architectural tweak that comes to mind and isn't in the article is QK norm: https://arxiv.org/pdf/2010.04245 > Cosine Schedule A lot (most?) of new training runs actually don't use cosine schedule anymore; instead they keep the learning rate constant and only decay it at the very end, which gives equivalent or better results. See: https://arxiv.org/pdf/2405.18392 https://arxiv.org/pdf/2404.06395 > There is a highly optimized…

Paper and discussion history. A mention is not an endorsement.

Query-Key Normalization for Transformers — mentioned by mota7

Tue, 08 Oct 2024 20:04:15 GMT

I had the same thought: Just eye-balling the graphs, the result of the subtraction looks very close to just reducing the temperature. They're effectively doing softmax with a fixed temperature, but it's unclear that this work is going to do better than just learning a per-head temperature parameter. c.f. https://arxiv.org/abs/2010.04245 which shows an improvement by learning per-head temperature. The other way to think about this is that it looks like a hacked-up kinda-sorta gated attention. If that's the case, then doing softmax(alphaq_1k_1^T - log_sigmoid(betaq_2k_2^T)) might be better?…

Paper and discussion history. A mention is not an endorsement.