The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers — mentioned by m_ke

Sat, 22 Nov 2025 20:05:49 GMT

Before we had proper GPUs everyone said the same thing about Neural Networks. Current model architectures are optimized to get the most out of GPUs, which is why we have transformers dominating as they're mostly large dense matrix multiplies. There's plenty of work showing transformers improve with inner dimension size but it's not feasible to scale them up further because it blows up parameter and activation sizes (including KV caches) so people to turn to low rank ("sparse") decompositions like MLA. Lottery ticket hypothesis shows that most of the weights in current models are redundant and…

Paper and discussion history. A mention is not an endorsement.

The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers · HackerPapers discussions

The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers — mentioned by m_ke