FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness · HackerPapers discussions

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — mentioned by elegantly

Fri, 25 Apr 2025 16:57:10 GMT

Next Monday, 28 Apr 2025, we are launching the public permanently free unlimited, no payment, no catch of Purem – a revolutionary CPU-accelerated AI kernel that matches the performance of leading in-house infrastructures like OpenAI, Tesla, and Meta. A major leap for macOS users – explore Purem here: https://worktif.com/ After strict recalculation accounting for GPU-to-CPU architecture and memory parallelism, our CPU implementation of Softmax (6500 ops/sec) outperforms the GPU-optimized FlashAttention recalculated for standard CPUs (estimated 800–5000 ops/sec) Benchmark Highlights: • Purem…

Paper and discussion history. A mention is not an endorsement.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — mentioned by Kelteseth

Tue, 30 Apr 2024 10:13:55 GMT

Looks like this is based on https://arxiv.org/abs/2205.14135

Paper and discussion history. A mention is not an endorsement.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — mentioned by araes

Sat, 16 Mar 2024 17:00:41 GMT

For those who have no idea what's being discussed, quick background. Discussing: Transformer [1] memory issues and approximate attention [2] in machine learning training. Specifically: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. [3] As a side comment, this entire industry is sorely in need of at least intros. The entire space has moved so fast in the last year I need an entire new dictionary and thesaurus for all the terms they've created. Notably, because of this, found out Google has a glossary of machine learning terms. Actually somewhat handy. [1] Google…

Paper and discussion history. A mention is not an endorsement.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — mentioned by kristjansson

Thu, 22 Feb 2024 17:20:08 GMT

FlashAttention(2)[0] reduces context-length space complexity to linear. Compute is still O(n^2) in length though, AFAIK, so we'd expect these long sequence lengths to take some time to compute. I'm a bit out of my depth, but I think ultra-long exact-attention work like this also probably has to answer some questions about where to put the KV-cache before it can be used in practice? [0]: https://arxiv.org/abs/2205.14135

Paper and discussion history. A mention is not an endorsement.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — HN submission

Tue, 16 Jan 2024 17:12:12 GMT

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

1 point; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — mentioned by scribu

Wed, 20 Dec 2023 17:42:03 GMT

I understand that it's a different approach, but I would still have expected this paper to at least mention FlashAttention [1] since they both leverage flash memory. [1] https://arxiv.org/abs/2205.14135

Paper and discussion history. A mention is not an endorsement.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — mentioned by varunshenoy

Tue, 21 Nov 2023 04:13:18 GMT

Good question. Yes, the 10GB available for batching is in the HBM. In a single forward pass, you move the entire model from HBM -> SRAM exactly once. In a batched forward pass, this is still the case, so you end up doing more compute for the same amount of memory movement. You can calculate the SRAM as follows: an A100 has 108 SMs, and each SM has 192 KB in SRAM (shared memory, aka its L1 cache) [1]. Multiplied out, this is ~20 MB of total SRAM. This happens to match up with the diagram in the Flash Attention paper [2]. [1] https://developer.nvidia.com/blog/cuda-refresher-cuda-progra... [2]…

Paper and discussion history. A mention is not an endorsement.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — mentioned by why_only_15

Tue, 11 Jul 2023 23:39:52 GMT

presumably you mean a dot product of Q and K, and no you do not have to store this: https://arxiv.org/abs/2205.14135

Paper and discussion history. A mention is not an endorsement.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — mentioned by EnricoShippole

Tue, 09 May 2023 14:09:30 GMT

Introducing three new open-source PaLM models trained at a context length of 8k on C4. Open-sourcing LLMs is a necessity for the fair and equitable democratization of AI. The models of sizes 150m, 410m, and 1b are available to download and use here: https://github.com/conceptofmind/PaLM If you have any questions about the models or training be sure to reach out and ask! I will try to respond promptly. https://twitter.com/EnricoShippole/status/165559930145459404... The models are also compatible with many of Lucidrain's popular repositories such as Toolformer-pytorch, PaLM-rlhf-pytorch, and…

Paper and discussion history. A mention is not an endorsement.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — HN submission

Fri, 05 May 2023 23:21:37 GMT

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

3 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — HN submission

Thu, 20 Apr 2023 04:02:36 GMT

Flash Attention

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — mentioned by lhl

Wed, 19 Apr 2023 21:36:23 GMT

https://arxiv.org/abs/2205.14135 - Section 5 suggests that the biggest limitation is that custom CUDA kernels need to be coded on a per-GPU architecture basis.

Paper and discussion history. A mention is not an endorsement.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — mentioned by dpflan

Tue, 11 Apr 2023 16:00:03 GMT

Sharing a comment from a similar line of discussion: """ Complexity is quadratic in sequence length. For 512 tokens it is 262K, but for 4000 tokens it becomes 16M and goes OOM on a single GPU. We need about 100K-1M tokens to load whole books at once. Since 2017 there have been hundreds of attempts to bring O(N^2) to O(N), but none of them replaced the vanilla attention yet in large models. They lose on accuracy. Maybe Flash attention has a shot (https://arxiv.org/abs/2205.14135). """ - source: https://news.ycombinator.com/item?id=34171503

Paper and discussion history. A mention is not an endorsement.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — mentioned by visarga

Thu, 29 Dec 2022 17:55:19 GMT

Complexity is quadratic in sequence length. For 512 tokens it is 262K, but for 4000 tokens it becomes 16M and goes OOM on a single GPU. We need about 100K-1M tokens to load whole books at once. Since 2017 there have been hundreds of attempts to bring O(N^2) to O(N), but none of them replaced the vanilla attention yet in large models. They lose on accuracy. Maybe Flash attention has a shot (https://arxiv.org/abs/2205.14135).

Paper and discussion history. A mention is not an endorsement.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — HN submission

Wed, 01 Jun 2022 21:07:08 GMT

Flash Attention accelerates GPT2 3x

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.