What's Hidden in a Randomly Weighted Neural Network? · HackerPapers discussions

What's Hidden in a Randomly Weighted Neural Network? — mentioned by kyllo

Mon, 24 Apr 2023 00:32:15 GMT

Are you familiar with the edge popup algorithm introduced in "What's Hidden in a Randomly Weighted Neural Network?" https://arxiv.org/abs/1911.13299v2 Seems relevant to what you're working on. It starts with a randomly initialized, overparameterized neural net, but instead of gradient descent backpropagation, it learns by deleting connection edges.

Paper and discussion history. A mention is not an endorsement.

What's Hidden in a Randomly Weighted Neural Network? — mentioned by MereInterest

Sat, 09 Jul 2022 18:51:31 GMT

There was a paper a few years ago about a similar effect in artificial neural networks [0]. The gist was that a large network can contain many subnetworks, and the number of subnetworks grows much faster than the size of the network they are contained in. They were able to find a subnetwork in a randomly weighted network with equivalent performance to a trained network of a much smaller size. [0] https://arxiv.org/abs/1911.13299

Paper and discussion history. A mention is not an endorsement.

What's Hidden in a Randomly Weighted Neural Network? — mentioned by belgian_guy

Tue, 27 Jul 2021 08:54:30 GMT

Surprisingly, it's quite doable in practice. See e.g. https://arxiv.org/abs/1911.13299.

Paper and discussion history. A mention is not an endorsement.

What's Hidden in a Randomly Weighted Neural Network? — mentioned by contravariant

Fri, 03 Jul 2020 11:54:07 GMT

>As for the random network, I was referring to https://arxiv.org/abs/1911.13299 and I have seen similar effects in my own work where a new architecture was performing significantly better before training than the old one was after training. The fact that a dense neural network with 20M parameters performs equally well as a model with 20M random values and 20M _bits_ worth of parameters means nothing more than that the parameter space is ridiculously large. The only models that perform well given random parameters are those that are sufficiently restrictive. Like weather forecasts, where…

Paper and discussion history. A mention is not an endorsement.

What's Hidden in a Randomly Weighted Neural Network? — mentioned by fxtentacle

Fri, 03 Jul 2020 10:51:30 GMT

Yes, I have reduced both statistics and ML to the subsets that are usually used when working in the field, because the blog post was about employment options. I would wager that people doing non-parametric statistics are both very rare and most likely advertise themselves as machine learning experts, not as statisticians. As for the random network, I was referring to https://arxiv.org/abs/1911.13299 and I have seen similar effects in my own work where a new architecture was performing significantly better before training than the old one was after training. If you want a generally agreed upon…

Paper and discussion history. A mention is not an endorsement.

What's Hidden in a Randomly Weighted Neural Network? — HN submission

Fri, 24 Apr 2020 19:52:09 GMT

What's Hidden in a Randomly Weighted Neural Network?

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

What's Hidden in a Randomly Weighted Neural Network? — mentioned by nil-sec

Sun, 08 Mar 2020 20:03:53 GMT

What you are stating is the original Lottery Ticket Hypothesis. What they prove in this paper is the stronger version, empirically noticed here https://arxiv.org/abs/1905.01067 and referred to as "supermasks". To quote from the paper posted here: "within a sufficiently overparameterized neural network with random weights (e.g. at initialization), there exists a subnetwork that achieves competitive accuracy". Edit: See also https://arxiv.org/abs/1911.13299

Paper and discussion history. A mention is not an endorsement.

What's Hidden in a Randomly Weighted Neural Network? — mentioned by csense

Sun, 08 Mar 2020 03:46:23 GMT

Neural networks are basically graphs with weighted edges. As you increase number of nodes N in a graph, the number of edges grows at most quadratically, but the number of possible subgraphs grows exponentially. So even if you use totally random edge weights, with high probability some of those O(2^N) subgraphs will be pretty good. OP paper is basically providing formal mathematical definitions for these loose notions, then chasing down a bunch of epsilons and deltas to prove that, in fact, good subgraphs actually do always exist (with high probability for large enough N). From a practical…

Paper and discussion history. A mention is not an endorsement.

What's Hidden in a Randomly Weighted Neural Network? — mentioned by fxtentacle

Mon, 03 Feb 2020 10:45:41 GMT

The problem that I see with supervised training of a linear classifier after unsupervised training is that if the unsupervised network is large enough, it allows the supervised trainer to choose the working components. As shown in [1] that can lead to randomly initialized networks working well, too, meaning that this does not necessarily show that the unsupervised training produced useful features. I would instead suggest to train a categorization classifier unsupervised, too, for example using mutual information loss with the correct number of categories, as suggested in [2]. Afterwards, one…

Paper and discussion history. A mention is not an endorsement.

What's Hidden in a Randomly Weighted Neural Network? — HN submission

Wed, 04 Dec 2019 19:32:53 GMT

What's Hidden in a Randomly Weighted Neural Network?

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

What's Hidden in a Randomly Weighted Neural Network? — HN submission

Tue, 03 Dec 2019 11:10:10 GMT

What's Hidden in a Randomly Weighted Neural Network?

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.