Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention · HackerPapers discussions

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — mentioned by Palmik

Sun, 07 Dec 2025 20:33:09 GMT

DeepSeek and other Chinese companies. Not only do they publish research, they also put their resources where their mouth (research) is. They actually use it and prove it through their open models. Most research coming out of big US labs is counter indicative of practical performance. If it worked (too) well in practice, it wouldn't have been published. Some examples from DeepSeek: https://arxiv.org/abs/2405.04434 https://arxiv.org/abs/2502.11089

Paper and discussion history. A mention is not an endorsement.

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — mentioned by Palmik

Sun, 07 Dec 2025 10:26:12 GMT

DeepSeek's MLA paper was published in 2024: https://arxiv.org/abs/2405.04434 DeepSeek's Sparse Attention paper was published in February: https://arxiv.org/abs/2502.11089 DeepSeek 3.2 Exp (combining MLA and DSA) was released in September. You also had several other Chinese hybrid models, like Qwen3 Next and Minimax M1.

Paper and discussion history. A mention is not an endorsement.

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — mentioned by boroboro4

Wed, 30 Apr 2025 04:03:15 GMT

I think in a way it makes transformers superior to humans, their short term memory is much more powerful =) Supporting extra long contexts also make transformers super human. Because, again, human's short term memory is exactly this - short term. And much shorter than millions of tokens we expect from models nowadays. As for SSMs - I think they compress model memory state way too much. Mixed global/local attention layers do just as well. And sparse/block attention seems like a way forward much more (https://arxiv.org/abs/2502.11089).

Paper and discussion history. A mention is not an endorsement.

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — HN submission

Sun, 02 Mar 2025 14:44:33 GMT

HW-Aligned Sparse Attention Architecture for Efficient Long-Context Modeling

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — HN submission

Wed, 19 Feb 2025 13:15:50 GMT

Native Sparse Attention: Hardware-Aligned and Natively Trainable

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — HN submission

Wed, 19 Feb 2025 03:12:01 GMT

NSA: Hardware-Aligned and Natively Trainable Sparse Attention

4 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — HN submission

Tue, 18 Feb 2025 16:17:40 GMT

DeepSeek Native Sparse Attention

16 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention — HN submission

Tue, 18 Feb 2025 07:04:47 GMT

Native Sparse Attention: Hardware-Aligned, Natively Trainable Sparse Attention

15 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.