Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent · HackerPapers discussions

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent — mentioned by gyang

Thu, 28 Nov 2019 22:59:31 GMT

@fgabriel mentioned this below: if the network is parametrized in a certain way, then the GP evolves according to a linear equation (if trained with square loss). In this linear equation, a different kernel shows up, known as the Neural Tangent Kernel. An intuitive way to think about this is to Taylor expand the parameters-to-function map around the initial set of parameters: f = f_0 + J d\theta, where J is the Jacobian of the neural network function against the parameters. Following this logic, the change in parameters affects the neural network function roughly linearly, as long as the…

Paper and discussion history. A mention is not an endorsement.

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent — HN submission

Wed, 20 Feb 2019 14:55:45 GMT

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent — HN submission

Wed, 20 Feb 2019 05:34:01 GMT

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.