GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — mentioned by woadwarrior01

Tue, 18 Jul 2023 17:27:37 GMT

Not quite a drop in replacement, but close enough. From the paper[1]: > Llama 2, an updated version of Llama 1, trained on a new mix of publicly available data. We also increased the size of the pretraining corpus by 40%, doubled the context length of the model, and adopted grouped-query attention (Ainslie et al., 2023)[2]. [1]: https://ai.meta.com/research/publications/llama-2-open-found... [2]: https://arxiv.org/abs/2305.13245

Paper and discussion history. A mention is not an endorsement.

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — mentioned by RC_ITR

Tue, 11 Jul 2023 07:45:20 GMT

>Autoregressive decoder inference is a severe bottleneck for Transformer models due to the memory bandwidth overhead from loading decoder weights and all attention keys and values at every decoding step (Shazeer, 2019; Pope et al., 2022; de Jong et al., 2022). The memory bandwidth from loading keys and values can be sharply reduced through multi-query attention (Shazeer, 2019), which uses multiple query heads but single key and value heads. Emphasis mine, source here [0] [0] https://arxiv.org/pdf/2305.13245.pdf FWIW the original MQA paper is called One Write head is all you need. Here's the…

Paper and discussion history. A mention is not an endorsement.

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints · HackerPapers discussions

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — mentioned by woadwarrior01

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — mentioned by RC_ITR