TransMLA: Multi-Head Latent Attention Is All You Need — HN submission

Tue, 13 May 2025 03:29:47 GMT

TransMLA: Multi-head latent attention is all you need

123 points; 32 discussion replies on HN. Paper and source history. A submission is not an endorsement.

TransMLA: Multi-Head Latent Attention Is All You Need — mentioned by refibrillator

Mon, 24 Feb 2025 07:40:17 GMT

vLLM supports MLA for Deepseek models as of 3 weeks ago. 3x higher generation throughput and 10x token memory capacity. https://github.com/vllm-project/vllm/releases/tag/v0.7.1 MHA is still faster in low QPS regime apparently. https://neuralmagic.com/blog/enhancing-deepseek-models-with-... Also published this month was theoretical proof showing that for the same KV Cache overhead, MLA consistently offers greater expressive power than GQA. Furthermore, widely used GQA-based pre-trained models (e.g. LLaMA, Qwen, Mixtral) can be converted into MLA-based models. https://arxiv.org/pdf/2502.07864

Paper and discussion history. A mention is not an endorsement.

TransMLA: Multi-Head Latent Attention Is All You Need · HackerPapers discussions

TransMLA: Multi-Head Latent Attention Is All You Need — HN submission

TransMLA: Multi-Head Latent Attention Is All You Need — mentioned by refibrillator