The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks · HackerPapers discussions

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by DARSHANFOFADIYA

Fri, 24 Apr 2026 01:15:40 GMT

Most "sparse training" in PyTorch today isn't actually sparse. A binary mask gets multiplied into a dense weight matrix, which means the zeros still consume memory, still move through the cache, and still get multiplied. That's pruning simulation, not sparse computation. SparseLab does the other thing: real sparse storage (a custom Padded-CSR layout), custom NEON kernels, in an nn.Linear-compatible layer. The premise. It's been known since the Lottery Ticket Hypothesis (https://arxiv.org/abs/1803.03635) and RigL (https://arxiv.org/abs/1911.11134) that most models train competitively with ~10%…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — HN submission

Fri, 02 Jan 2026 22:53:53 GMT

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018)

121 points; 27 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by srean

Wed, 26 Nov 2025 14:00:58 GMT

What you are looking for is the lottery ticket hypothesis for neural networks. Hit a search engine with those words you will find examples. https://arxiv.org/abs/1803.03635 ( you can follow up on semantic scholar for more) Selecting which weights to discard seems as hard as the original problem. But random decimation, sometimes barely informed decimation have been observed to be effective. On the theory side now it's understood that in the thicket of weights, lurk a much much smaller subset that can have nearly the same output. These observations are for DNNs in general. For time series…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by m_ke

Sat, 22 Nov 2025 18:08:27 GMT

https://transformer-circuits.pub/2022/toy_model/index.html https://arxiv.org/abs/1803.03635 EDIT: don't have time to write it up, but here's gemini 3 with a short explanation: To simulate the brain's efficiency using Transformer-like architectures, we would need to fundamentally alter three layers of the stack: the *mathematical representation* (moving to high dimensions), the *computational model* (moving to sparsity), and the *physical hardware* (moving to neuromorphic chips). Here is how we could simulate a "Brain-Like Transformer" by combining High-Dimensional Computing (HDC) with Spiking…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by gotoeleven

Mon, 18 Aug 2025 21:08:33 GMT

This article gives a really bad/wrong explanation of the lottery ticket hypothesis. Here's the original paper https://arxiv.org/abs/1803.03635

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by ZeljkoS

Wed, 05 Feb 2025 23:42:01 GMT

We have a partial understanding of why distillation works—it is explained by The Lottery Ticket Hypothesis (https://arxiv.org/abs/1803.03635). But if I am understanding correctly, that doesn't mean you can train a smaller network from scratch. You need a lot of randomness in the initial large network, for some neurons to have "winning" states. Then you can distill those winning subsystems to a smaller network. Note that similar process happens with human brain, it is called Synaptic pruning (https://en.wikipedia.org/wiki/Synaptic_pruning). Relevant quote from Wikipedia…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by patcon

Thu, 30 Jan 2025 08:24:24 GMT

Are we it rediscovering the evolutionary benefit of progeny (from an information theoretic lens)? And is this related to the lottery ticket hypothesis? https://arxiv.org/pdf/1803.03635.pdf

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by whoistraitor

Tue, 15 Oct 2024 00:23:54 GMT

Cool! Seems we’re moving closer and closer to realizing the Lottery Ticket Hypothesis https://arxiv.org/abs/1803.03635

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by tedivm

Thu, 30 Nov 2023 17:12:33 GMT

This was something we thought about a bit at Rad AI. I think one of the problems this paper ignores isn't whether you can get a general purpose model to beat a special purpose one, but whether it's really worth it (outside of the academic sense). Accuracy is obviously important, and anything that sacrifices accuracy in a medical areas is potentially dangerous. So everything I'm about to say assumes that accuracy remains the same or improves for a special purpose model (and in general that is the case- papers such as this talk about shrinking models as a way of improvement without sacrificing…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by godelski

Sun, 04 Jun 2023 20:11:27 GMT

There's been an existential issue I've seen with publishing, concerning this (same/related issues to the article). There's models and datasets that are closed and being published in works. Here's an example[0]. ICLR states > Submissions will be double blind: reviewers cannot see author names when conducting reviews, and authors cannot see reviewer names. This means that the submission must not contain acknowledgements or any link (e.g., github) that would reveal authors’ identity. While Chinchila certainly is not a github link it is an acknowledgement that reveals the lab identity. Which…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by godelski

Wed, 31 May 2023 20:36:49 GMT

This is not always true. ML has reproducibility issues despite the status quo being open models. Even if we don't include proprietary datasets there are still issues. Compute is one issue, but we can even ignore that[0]. The Lottery Ticket[1,2] plays a big role, in that you can just get lucky. This really should have resulted in treating benchmarks as weaker indicators but see other comments about reviewing incentives. Another issue is that it is status quo to optimize hyperparameters on test data and this results in information leakage. While this won't affect results of running a checkpoint…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by codethief

Wed, 03 May 2023 21:34:03 GMT

> lottery ticket hypothesis For those that, like me, didn't know the reference: https://arxiv.org/abs/1803.03635

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by kelseyfrog

Sat, 11 Mar 2023 22:39:06 GMT

Idk, I think it's like looking at a winning lottery ticket[1] to a game you haven't played yet. 1. https://arxiv.org/abs/1803.03635

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by polygamous_bat

Sun, 29 Jan 2023 21:09:40 GMT

What you are describing is known as the "lottery ticket hypothesis" [0] in the ML world -- it is a well-studied phenomenon! [0] https://arxiv.org/abs/1803.03635

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by kristianp

Sun, 29 Jan 2023 21:06:53 GMT

This is interesting. Is it possible to run training for 10 minutes and check the loss, then again for another set of random weights? A bit like bitcoin mining. Then continue training for the best set of starting weights. The study referenced is (2018): The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks https://arxiv.org/abs/1803.03635

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by kristianp

Sat, 28 Jan 2023 20:58:11 GMT

https://arxiv.org/abs/1803.03635

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by m3047

Wed, 26 Oct 2022 19:08:11 GMT

Counter argument: For every trainable NN, there exists an equivalent, sparser [edit] trained NN. https://numenta.com/blog/2019/08/30/case-for-sparsity-in-neu... https://web.archive.org/web/20211130211241/https://www.bhaut... https://arxiv.org/abs/1803.03635

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by stared

Thu, 10 Feb 2022 17:40:28 GMT

I am surprised that the paper does not even cite the Lottery Ticket Hypothesis (https://arxiv.org/abs/1803.03635, https://eng.uber.com/deconstructing-lottery-tickets/). In the LTH paper (IMHO the most fundamental deep learning publication in the last few years), the number of tickets goes as layer_size^n_layers.

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by kn8a

Tue, 25 Jan 2022 22:05:24 GMT

Sorry I meant the 'lottery ticket' hypothesis. https://arxiv.org/abs/1803.03635v1 The lottery tickets are random but they are basically - as I understand - smaller graphs that have the same strength as a larger graph.

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by kylevedder

Sat, 25 Sep 2021 00:26:43 GMT

You should look at Frankle's work on The Lottery Ticket Hypothesis [1]; it turns out that in most cases you can remove 80+% of the network's weights and still get very similar output quality. The hypothesized reason is that regions of the network which are randomly initialized end up getting a "winning lottery ticket" which already has structure reasonably well optimized for your end task to be further finetuned, and everything else mostly ends up just being set dressing, which is why you can delete those neurons without a major impact to performance. That said, I don't think we are going to…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by talolard

Tue, 21 Sep 2021 18:44:58 GMT

There are two answers to this. First, empirically we see that the more parameters we add the better the model performs ==> Weights continue to contribute (and aren't dead) . Second, there is a very popular paper called "The lottery ticket hypothesis" [1] that in any network you can find subnetworks that work just as well. e.g. The parameters are redundant. This was written in 2018, which is a long time ago in big NN world, so I'm not sure how it holds up to current insanity sized models. [1]https://arxiv.org/abs/1803.03635

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by gojomo

Tue, 27 Jul 2021 00:20:47 GMT

Something about this reminds me of the 'Winning Ticket Hypothesis' in artifical neural networks: that some 'random' initializations prime a network far better for later faster learning. From https://arxiv.org/abs/1803.03635 - the abstract: Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by pretty_dumm_guy

Sun, 20 Dec 2020 20:34:01 GMT

Thank you for your answer. It appears to me that we are trying to achieve an algorithm that has better time complexity than the one that we have right now(reverse mode differentiation with gradient descent). Is it possible to combine these methods in a straight forward manner with methods that try to reduce the space complexity? For example, Lottery ticket hypothesis(https://arxiv.org/abs/1803.03635) seems to reduce spacial complexity(Please do correct me if I am wrong). Also, based on my rather poor and limited knowledge, it appears to me that set of proposed methods that reduced space…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by nl

Wed, 09 Dec 2020 05:32:11 GMT

I only properly read the Lottery Ticket Hypothesis paper[1] properly at the start of 2020. I think it's going to be years before we understand this properly, but in 2020 we are beginning to see practical uses. At the moment I think it's a toss up: it's either going to be a curiosity that people read, have their mind exploded but can't do anything with or else it's a good chance to be the most influential deep learning paper of the decade. [1] https://arxiv.org/abs/1803.03635

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — HN submission

Fri, 23 Oct 2020 22:45:36 GMT

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2019)

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by ricklamers

Fri, 04 Sep 2020 08:19:53 GMT

> Actually that's not the point. Pruning typically results in networks that still perform well but are harder to train. The idea is to explicitly search for a subnetwork (via pruning) that is easy to train. They state "smaller network, same test accuracy, with similar number of iterations". So it seems the original network size wasn't necessary for best test accuracy, and compute requirement is reduced only because it's a smaller network. Sample efficiency isn't increased according to https://arxiv.org/abs/1803.03635. Good performance with random weights seems to indicate good 'priors'…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by Reelin

Fri, 04 Sep 2020 06:11:19 GMT

I agree that there are clearly algorithmic improvements remaining to be made. However, a counterpoint to your specific example would be the lottery ticket hypothesis and related weight agnostic neural networks. https://arxiv.org/abs/1803.03635 https://ai.facebook.com/blog/understanding-the-generalizatio... https://ai.googleblog.com/2019/08/exploring-weight-agnostic-...

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by pshc

Tue, 30 Jun 2020 00:52:23 GMT

Would you say you found a result similar to the lottery ticket hypothesis? https://arxiv.org/abs/1803.03635

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by orange3xchicken

Fri, 08 May 2020 21:20:19 GMT

Yeah, the original work this paper follows up (by the same group: https://arxiv.org/abs/1803.03635) received a lot of attention in 2018 when it was uploaded to Arxiv & spawned a lot of followup work. Even though it's been demonstrated that these lottery tickets - sparse trainable subnetworks exist and can approximate the complete nn arbitrarily well, their properties are still not really understood. What is understood is that these subnetworks depend heavily on the initialization of the network, but that training the entire network together is necessary for generalization. These findings…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by stfwn

Sat, 21 Mar 2020 22:46:30 GMT

You might be interested in pruning neural networks. Some networks can be reduced in size by over 90% without performance loss. The lottery ticket hypothesis paper is good place to start, if you don’t already know about it of course. https://arxiv.org/abs/1803.03635 Some pruning methods have recently been added in the latest feature release of PyTorch.

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by stared

Sat, 07 Mar 2020 17:30:41 GMT

The lottery ticket hypothesis is IMHO the single most interesting finding for deep learning. It explains why does deep learning works (vs shallow neural nets), why is initial over-parametrization is often useful, why deeper is often better than shallow, etc. I recommend for an overview: - the original paper "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks", https://arxiv.org/abs/1803.03635 - "Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask" https://eng.uber.com/deconstructing-lottery-tickets/ showing that if we remove "non-winning tickets" before the…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by choppaface

Thu, 29 Aug 2019 12:06:56 GMT

The authors draw from traditional genetic algorithms (NEAT http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf ) but this appears to be one of the first papers (at least recently) where proposed architectures are not trained but rather given random weights. The authors here try to qualitatively distill the role of architecture (vs optimization) in neural net research. The work here is in a similar vein as the Lottery Ticket Hypothesis ( https://arxiv.org/pdf/1803.03635.pdf ), which found that deep nets (for vision) contain discriminative sub-networks at initialization time (due to…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by jsharf

Sun, 02 Jun 2019 15:48:27 GMT

Does this remind anyone else of the lottery ticket hypothesis deep learning paper that was published recently? [0] I think the gist was that neural networks are usually much larger and more connected than necessary. The paper provided an algorithm to iteratively prune connections by seeing which links in the neural network had the lowest weights. Here's a quote from the paper: "The Lottery Ticket Hypothesis: A randomly-initialized, dense neural network contains a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by nestorD

Tue, 28 May 2019 14:33:33 GMT

Better coverage of the space of weights (less clumps and holes). If I understand the lottery ticket hypothesis [0] correctly, this would lead to a better exploration of the space and thus better results. [0] https://arxiv.org/abs/1803.03635

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by fromthestart

Mon, 20 May 2019 23:56:48 GMT

Actually, there's already a better way to perform initialization, based on the so called lottery ticket hypothesis [1]. I haven't gotten to the article, so I'll just regurgitate the abstract, but basically there frequently are subnetworks which may be exposed by pruning trained networks which perform on par with full sizes neural nets with ≈20% of the parameters and substantially quicker training time. It turns out that with some magic algorithm described in the paper, one can initialize weights to quickly find these "winning tickets" to drastically reduce neural network size and training…

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by sanxiyn

Fri, 17 May 2019 02:19:52 GMT

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, https://arxiv.org/abs/1803.03635 This is in fact the most interesting hypothesis on why neural networks work I have ever read.

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — mentioned by dplarson

Tue, 07 May 2019 06:26:30 GMT

Since the article didn't link to the paper: - "The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks" - https://arxiv.org/abs/1803.03635

Paper and discussion history. A mention is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — HN submission

Thu, 05 Jul 2018 21:25:43 GMT

The Lottery Ticket Hypothesis: Finding Small, Trainable Neural Networks

137 points; 14 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks — HN submission

Mon, 19 Mar 2018 16:44:25 GMT

The Lottery Ticket Hypothesis: Training Pruned Neural Networks

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.