Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention · HackerPapers discussions

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention — mentioned by westurner

Thu, 30 May 2024 03:35:25 GMT

Because self-attention can be replaced with FFT for a loss in accuracy and a reduction in kWh [1], I suspect that the Quantum Fourier Transform can also be substituted for attention in LLMs. [1] https://syncedreview.com/2021/05/14/deepmind-podracer-tpu-ba... "You Need to Pay Better Attention" (2024) https://arxiv.org/abs/2403.01643 : > Our first contribution is Optimised Attention, which performs similarly to standard attention, but has 3/4 as many parameters and one matrix multiplication fewer per head. Next, we introduce Efficient Attention, which performs on par with standard attention…

Paper and discussion history. A mention is not an endorsement.

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention — mentioned by underlines

Wed, 17 Apr 2024 21:01:48 GMT

tons and tons of papers, most of them had some disadvantages. Can't have the cake and eat it too: https://arxiv.org/html/2404.08801v1 Meta Megalodon https://arxiv.org/html/2404.07143v1 Google Infini-Attention https://arxiv.org/html/2402.13753v1 LongRoPE and a ton more

Paper and discussion history. A mention is not an endorsement.

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention — mentioned by ttul

Sun, 14 Apr 2024 23:16:48 GMT

https://arxiv.org/html/2404.07143v1 “This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation”

Paper and discussion history. A mention is not an endorsement.

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention — mentioned by westurner

Sat, 13 Apr 2024 05:33:53 GMT

"Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" (2024) https://arxiv.org/abs/2404.07143 - https://news.ycombinator.com/item?id=40001626#40020560

Paper and discussion history. A mention is not an endorsement.

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention — mentioned by westurner

Sat, 13 Apr 2024 04:44:34 GMT

"Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" (2024) https://arxiv.org/abs/2404.07143 : > This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We…

Paper and discussion history. A mention is not an endorsement.

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention — HN submission

Fri, 12 Apr 2024 02:09:04 GMT

Leave No Context Behind: Efficient Infinite Context Transformers

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention — mentioned by HarHarVeryFunny

Thu, 11 Apr 2024 13:32:55 GMT

Scaling alone will certainly not get us to AGI, but that's trivially obvious. The "scale is all you need" argument, from anyone intelligent, assumes that other obvious deficiencies such as lack of short-term memory (to enable more capable planning/reasoning) will also be taken care of along the way, and I'd not be surprised to see that particular one addressed in upcoming next-gen models. There's a recent paper here from Google suggesting one way to do it, although may other ways too. https://arxiv.org/pdf/2404.07143.pdf The more interesting question is what other components/capabilities…

Paper and discussion history. A mention is not an endorsement.

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention — HN submission

Thu, 11 Apr 2024 13:01:02 GMT

Leave No Context Behind: Efficient Infinite Context Transformers

39 points; 4 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention — HN submission

Thu, 11 Apr 2024 03:53:24 GMT

Google: Efficient Infinite Context Transformers with Infini-Attention

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.