The Loss Surfaces of Multilayer Networks · HackerPapers discussions

The Loss Surfaces of Multilayer Networks — mentioned by ChrisRackauckas

Thu, 27 Aug 2020 13:35:02 GMT

There is a saying in mathematics that the fastest way to a solution is through the complex plane. This was discovered because a lot of proofs are nicer by doing analytical continuation and analyzing the properties of the continued function. Complex-step differentiation is another example of this. In some sense, something similar applies to neural networks in this context. Have you done a lot of fitting of classical basis methods inside of differential equations? They are very prone to local minima, so direct training of polynomials inside of a differential equation is rather hard. But through…

Paper and discussion history. A mention is not an endorsement.

The Loss Surfaces of Multilayer Networks — HN submission

Mon, 28 Jan 2019 10:57:50 GMT

The Loss Surfaces of Multilayer Networks

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Loss Surfaces of Multilayer Networks — mentioned by nabla9

Sat, 16 Dec 2017 19:40:00 GMT

https://arxiv.org/abs/1412.0233

Paper and discussion history. A mention is not an endorsement.

The Loss Surfaces of Multilayer Networks — mentioned by aoki

Tue, 12 Dec 2017 05:26:26 GMT

i suspect that what he's referring to is that he's heuristically minimizing a somewhat arbitrary (loss) function in a million-ish dimensions using the simple variants of gradient descent that work under these conditions. it sounds far too WIBNI to produce good results reliably (in practice, let alone in theory). the landscape has so many stationary points at which to get stuck; why would you ever get good results? there's a small cottage industry of papers (like [0]) that try to explain this. [0] https://arxiv.org/pdf/1412.0233.pdf

Paper and discussion history. A mention is not an endorsement.

The Loss Surfaces of Multilayer Networks — mentioned by deepnet

Wed, 31 May 2017 19:55:08 GMT

Analogously Neural Network learning, i.e. learning modeled as a phase transition between disorder and order: “... studies the connection between the highly non-convex loss function of a simple model of the fully-connected feed-forward neural network and the Hamiltonian of the spherical spin-glass model” from "The Loss Surfaces of Multilayer Networks" Choromanska, Henaff, Mathieu, Arous, & LeCun [1] [1] https://arxiv.org/abs/1412.0233 Also the physics of collapsing sand piles.

Paper and discussion history. A mention is not an endorsement.

The Loss Surfaces of Multilayer Networks — mentioned by reader5000

Sun, 07 May 2017 23:18:44 GMT

Are you referencing https://arxiv.org/abs/1412.0233?

Paper and discussion history. A mention is not an endorsement.

The Loss Surfaces of Multilayer Networks — mentioned by rm999

Fri, 19 Aug 2016 22:12:39 GMT

If you mean the lack of theoretical guarantees due to a highly non-convex optimization, you could argue the same thing about 30 year old 2-layer networks. And we actually do have a decent idea why deep learning works well: http://arxiv.org/pdf/1412.6544v4.pdf http://arxiv.org/pdf/1412.0233.pdf

Paper and discussion history. A mention is not an endorsement.

The Loss Surfaces of Multilayer Networks — mentioned by Houshalter

Sat, 06 Feb 2016 20:49:55 GMT

I believe that has to do with the vanishing gradient issue which has been studied. As for why nets converge at all, there's this paper whichI believe tries to establish some theory why bigger nets don't have such a big problem with local optima: http://arxiv.org/abs/1412.0233

Paper and discussion history. A mention is not an endorsement.

The Loss Surfaces of Multilayer Networks — mentioned by Houshalter

Sat, 06 Feb 2016 20:45:41 GMT

>an infinitely deep net (whatever that means) would not generalize on little data, and would get even harder to train the deeper it gets. Not with proper priors/regularization. >You've argued some ideas from intuition. There is little theoretical rigor around this, however. There's this paper which goes more into theoretical depth on the idea: http://arxiv.org/abs/1412.0233

Paper and discussion history. A mention is not an endorsement.

The Loss Surfaces of Multilayer Networks — mentioned by Houshalter

Sat, 06 Feb 2016 20:44:31 GMT

http://arxiv.org/abs/1412.0233 >The number of local minima outside that band diminishes exponentially with the size of the network. We empirically verify that the mathematical model exhibits similar behavior as the computer simulations, despite the presence of high dependencies in real networks. We conjecture that both simulated annealing and SGD converge to the band of low critical points, and that all critical points found there are local minima of high quality measured by the test error. This emphasizes a major difference between large- and small-size networks where for the latter poor…

Paper and discussion history. A mention is not an endorsement.

The Loss Surfaces of Multilayer Networks — mentioned by deepnet

Wed, 05 Aug 2015 18:01:39 GMT

As a counterpoint, the Blessing of Dimensionality is that local minima are mostly as good as the global minima. In David Silver's RLDM 2015 talk @20 mins in http://videolectures.net/rldm2015_silver_reinforcement_learn... he recounts this as his experience. the paper The Loss Surfaces of Multilayer Networks goes in depth http://arxiv.org/abs/1412.0233 [edit] and the paper Qualitative Characterisation of Neural Net Optimisation http://arxiv.org/abs/1412.6544v4

Paper and discussion history. A mention is not an endorsement.

The Loss Surfaces of Multilayer Networks — mentioned by selimthegrim

Sun, 14 Jun 2015 17:16:08 GMT

http://arxiv.org/abs/1412.0233 (and the references within he cites by Gerard Ben Arous)

Paper and discussion history. A mention is not an endorsement.

The Loss Surfaces of Multilayer Networks — mentioned by selimthegrim

Mon, 23 Mar 2015 00:59:21 GMT

This is as good a post as any about the theory of why deep learning works so well - esp. if you know a little about spin models from statistical mechanics https://charlesmartin14.wordpress.com/2015/02/26/why-does-de... additional flavor: http://arxiv.org/pdf/1412.0233.pdf and slides 16-22 of https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsd...

Paper and discussion history. A mention is not an endorsement.