Feature Learning in Infinite-Width Neural Networks · HackerPapers discussions

Feature Learning in Infinite-Width Neural Networks — mentioned by medlyyy

Fri, 05 Mar 2021 11:55:54 GMT

Why exactly is it a problem that powerful models can memorise training data with random labels, if they generalise well on non-memorisation problems? I also think that a lot of progress is being made seemingly in parallel on this "understanding NN [generalisation]" topic, and the different threads don't necessarily seem to be aware of each other. This article seems to have failed to mention most of this recent progress. Possibly this is an artifact of the time it takes to produce and publish a completed article, but it presents a picture that we are more clueless than we actually are. The…

Paper and discussion history. A mention is not an endorsement.

Feature Learning in Infinite-Width Neural Networks — HN submission

Tue, 09 Feb 2021 17:13:50 GMT

Wide Neural Networks Are Feature Learners, Not Kernel Machines

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Feature Learning in Infinite-Width Neural Networks — HN submission

Mon, 07 Dec 2020 13:44:04 GMT

Wide Neural Networks Are Feature Learners, Not Kernel Machines

5 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Feature Learning in Infinite-Width Neural Networks — mentioned by edwardjhu

Sun, 06 Dec 2020 22:53:55 GMT

The claim here is a bit misleading, as already pointed out by other comments, since the kernel is an evolving one that is essentially learned after seeing the data. Contrary to many related works that compare wide neural networks to kernel methods, our recent work shows that one can study a feature learning infinite-width limit with realistic learning rate. https://arxiv.org/abs/2011.14522 We identified what separates the kernel regime (e.g., NTK) and the feature learning regime. In the infinite-width limit, OP's work could belong to either regime depending on the parametrization, i.e. the…

Paper and discussion history. A mention is not an endorsement.