HackerPapers

arXiv:1905.01067cs.LG

Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask

Hattie Zhou, Janice Lan, Rosanne Liu, Jason Yosinski

Abstract

The recent "Lottery Ticket Hypothesis" paper by Frankle & Carbin showed that a simple approach to creating sparse networks (keeping the large weights) results in models that are trainable from scratch, but only when starting from the same initial weights. The performance of these networks often exceeds the performance of the non-sparse base model, but for reasons that were not well understood. In this paper we study the three critical components of the Lottery Ticket (LT) algorithm, showing that each may be varied significantly without impacting the overall results. Ablating these factors leads to new insights for why LT networks perform as well as they do. We show why setting weights to zero is important, how signs are all you need to make the reinitialized network train, and why masking behaves like training. Finally, we discover the existence of Supermasks, masks that can be applied to an untrained, randomly initialized network to produce a model with performance far better than chance (86% on MNIST, 41% on CIFAR-10).

arXiv comments
NeurIPS 2019 camera ready version
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.CVComputer Vision and Pattern Recognition stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 2 identifiable commenters in 2 comments across 2 threads, from May 2019 to Mar 2020.

Comments linking this paper, per year.

Discussed in comments2 threads

  1. Proving the Lottery Ticket Hypothesis: Pruning is All You Need 1 comment

    • What you are stating is the original Lottery Ticket Hypothesis. What they prove in this paper is the stronger version, empirically noticed here https://arxiv.org/abs/1905.01067 and referred to as "supermasks". To quote from the paper posted here: "within a sufficiently…

      nil-secView on HNParent context

  2. Neural nets typically contain smaller “subnetworks” that can often learn faster 1 comment

    • Follow up from ICLR really elucidates some of the questions raised by this: https://eng.uber.com/deconstructing-lottery-tickets/ https://arxiv.org/abs/1905.01067

      bitforgerView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.