Element-wise Attention Is All You Need · HackerPapers discussions

Element-wise Attention Is All You Need — mentioned by verdverm

Wed, 22 Jan 2025 07:29:12 GMT

The more meaningful contribution may be (section 3.4) > These variants illustrate TPA’s versatility in balancing memory cost, computational overhead, and representation power. By choosing which dimensions (heads or tokens) remain contextual and adjusting ranks (RQ, RK, RV ), TPA unifies multiple existing attention mechanisms— such as MHA, MQA, and GQA—under one framework, while potentially reducing the KV cache size by an order of magnitude during autoregressive inference. re: the title, it might be the true one if their proofs hold up --- I'm now curious if the Element-wise Attention is All…

Paper and discussion history. A mention is not an endorsement.

Element-wise Attention Is All You Need — mentioned by verdverm

Wed, 22 Jan 2025 04:59:45 GMT

My math is rusty, but it looks to have a higher complexity than the original attention. I cannot say if it is an issue. Generally it seems we are willing to spend more computation at training time if it produces better results at inference time. In this case they are reducing the resources needed at inference time (an order of magnitude for the KV cache) or enabling longer sequences given the same resources. There's another paper I saw yesterday, "Element-wise Attention is All You Need" which looks like an early preprint, written by a solo author with a solo A800, and tested on some smaller…

Paper and discussion history. A mention is not an endorsement.