Fast Transformer Decoding: One Write-Head is All You Need · HackerPapers discussions

Fast Transformer Decoding: One Write-Head is All You Need — mentioned by why_only_15

Tue, 11 Jul 2023 08:21:00 GMT

This is the original paper: https://arxiv.org/abs/1911.02150 . The idea is that with a transformer you have many heads, say 64 for LLaMa, and for each head you have 1 "query" vector one "key" vector and one "value" vector per token. Most of the cost of inferencing models is loading the key and value vectors from GPU memory to the GPU itself. the idea behind MQA is that instead of having 64 queries, 64 keys, and 64 values, you have 64 queries, 1 key, and 1 value ("Multi-Query" as opposed to "Multi-Head", the original name). This means that there is much less data to load from GPU memory to the…

Paper and discussion history. A mention is not an endorsement.

Fast Transformer Decoding: One Write-Head is All You Need — mentioned by RC_ITR

Tue, 11 Jul 2023 07:45:20 GMT

>Autoregressive decoder inference is a severe bottleneck for Transformer models due to the memory bandwidth overhead from loading decoder weights and all attention keys and values at every decoding step (Shazeer, 2019; Pope et al., 2022; de Jong et al., 2022). The memory bandwidth from loading keys and values can be sharply reduced through multi-query attention (Shazeer, 2019), which uses multiple query heads but single key and value heads. Emphasis mine, source here [0] [0] https://arxiv.org/pdf/2305.13245.pdf FWIW the original MQA paper is called One Write head is all you need. Here's the…

Paper and discussion history. A mention is not an endorsement.

Fast Transformer Decoding: One Write-Head is All You Need — HN submission

Tue, 30 May 2023 16:17:37 GMT

Fast Transformer Decoding: One Write-Head Is All You Need

4 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.