HackerPapers

arXiv:1911.13299cs.CV

What's Hidden in a Randomly Weighted Neural Network?

Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, Mohammad Rastegari

Abstract

Training a neural network is synonymous with learning the values of the weights. By contrast, we demonstrate that randomly weighted neural networks contain subnetworks which achieve impressive performance without ever training the weight values. Hidden in a randomly weighted Wide ResNet-50 we show that there is a subnetwork (with random weights) that is smaller than, but matches the performance of a ResNet-34 trained on ImageNet. Not only do these "untrained subnetworks" exist, but we provide an algorithm to effectively find them. We empirically show that as randomly weighted neural networks with fixed weights grow wider and deeper, an "untrained subnetwork" approaches a network with learned weights in accuracy. Our code and pretrained models are available at https://github.com/allenai/hidden-networks.

arXiv comments
Accepted to CVPR 2020
Journal reference
DOI
Subjects
cs.CVComputer Vision and Pattern Recognition cs.LGMachine Learning

On Hacker News

Linked by 7 identifiable commenters in 8 comments across 6 threads, from Feb 2020 to Apr 2023. Found in 3 HN submissions from Dec 2019 to Apr 2020. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Gradients are not all you need 1 comment

    • Are you familiar with the edge popup algorithm introduced in "What's Hidden in a Randomly Weighted Neural Network?" https://arxiv.org/abs/1911.13299v2 Seems relevant to what you're working on. It starts with a randomly initialized, overparameterized neural net, but instead of…

      kylloView on HNParent context

  2. I should have loved biology 1 comment

    • There was a paper a few years ago about a similar effect in artificial neural networks [0]. The gist was that a large network can contain many subnetworks, and the number of subnetworks grows much faster than the size of the network they are contained in. They were able to find…

      MereInterestView on HNParent context

  3. Mammals dream about the world they are about to experience before they are born 1 comment
  4. On moving from statistics to machine learning, the final stage of grief (2019) 2 comments

    • Yes, I have reduced both statistics and ML to the subsets that are usually used when working in the field, because the blog post was about employment options. I would wager that people doing non-parametric statistics are both very rare and most likely advertise themselves as…

      fxtentacleView on HNParent context

    • >As for the random network, I was referring to https://arxiv.org/abs/1911.13299 and I have seen similar effects in my own work where a new architecture was performing significantly better before training than the old one was after training. The fact that a dense neural network…

      contravariantView on HNParent context

  5. Proving the Lottery Ticket Hypothesis: Pruning is All You Need 2 comments

    • Neural networks are basically graphs with weighted edges. As you increase number of nodes N in a graph, the number of edges grows at most quadratically, but the number of possible subgraphs grows exponentially. So even if you use totally random edge weights, with high…

      csenseView on HNParent context

    • What you are stating is the original Lottery Ticket Hypothesis. What they prove in this paper is the stronger version, empirically noticed here https://arxiv.org/abs/1905.01067 and referred to as "supermasks". To quote from the paper posted here: "within a sufficiently…

      nil-secView on HNParent context

  6. Contrastive Self-Supervised Learning 1 comment

    • The problem that I see with supervised training of a linear classifier after unsupervised training is that if the unsupervised network is large enough, it allows the supervised trainer to choose the working components. As shown in [1] that can lead to randomly initialized…

      fxtentacleView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.