HackerPapers

arXiv:1412.0233cs.LG

The Loss Surfaces of Multilayer Networks

Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, Yann LeCun

Abstract

We study the connection between the highly non-convex loss function of a simple model of the fully-connected feed-forward neural network and the Hamiltonian of the spherical spin-glass model under the assumptions of: i) variable independence, ii) redundancy in network parametrization, and iii) uniformity. These assumptions enable us to explain the complexity of the fully decoupled neural network through the prism of the results from random matrix theory. We show that for large-size decoupled networks the lowest critical values of the random loss function form a layered structure and they are located in a well-defined band lower-bounded by the global minimum. The number of local minima outside that band diminishes exponentially with the size of the network. We empirically verify that the mathematical model exhibits similar behavior as the computer simulations, despite the presence of high dependencies in real networks. We conjecture that both simulated annealing and SGD converge to the band of low critical points, and that all critical points found there are local minima of high quality measured by the test error. This emphasizes a major difference between large- and small-size networks where for the latter poor quality local minima have non-zero probability of being recovered. Finally, we prove that recovering the global minimum becomes harder as the network size increases and that it is in practice irrelevant as global minimum often leads to overfitting.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 8 identifiable commenters in 12 comments across 10 threads, from Mar 2015 to Aug 2020. Found in 1 HN submission from Jan 2019 to Jan 2019. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments10 threads

  1. Differentiable Control Problems 1 comment

    • There is a saying in mathematics that the fastest way to a solution is through the complex plane. This was discovered because a lot of proofs are nicer by doing analytical continuation and analyzing the properties of the continued function. Complex-step differentiation is…

      ChrisRackauckasView on HNParent context

  2. Mathematics of Deep Learning [pdf] 1 comment
  3. An Interview with an Anonymous Data Scientist (2016) 1 comment

    • i suspect that what he's referring to is that he's heuristically minimizing a somewhat arbitrary (loss) function in a million-ish dimensions using the simple variants of gradient descent that work under these conditions. it sounds far too WIBNI to produce good results reliably…

      aokiView on HNParent context

  4. Breaking Glass in Infinite Dimensions 1 comment

    • Analogously Neural Network learning, i.e. learning modeled as a phase transition between disorder and order: “... studies the connection between the highly non-convex loss function of a simple model of the fully-connected feed-forward neural network and the Hamiltonian of the…

      deepnetView on HNParent context

  5. Deep Learning’s Impact on Image Processing, Mathematics, and Humanity 1 comment
  6. What’s Next for Artificial Intelligence 1 comment

    • If you mean the lack of theoretical guarantees due to a highly non-convex optimization, you could argue the same thing about 30 year old 2-layer networks. And we actually do have a decent idea why deep learning works well: http://arxiv.org/pdf/1412.6544v4.pdf…

      rm999View on HNParent context

  7. AI is transforming Google search – the rest of the web is next 3 comments

    • http://arxiv.org/abs/1412.0233 >The number of local minima outside that band diminishes exponentially with the size of the network. We empirically verify that the mathematical model exhibits similar behavior as the computer simulations, despite the presence of high dependencies…

      HoushalterView on HNParent context

    • >an infinitely deep net (whatever that means) would not generalize on little data, and would get even harder to train the deeper it gets. Not with proper priors/regularization. >You've argued some ideas from intuition. There is little theoretical rigor around this, however.…

      HoushalterView on HNParent context

    • I believe that has to do with the vanishing gradient issue which has been studied. As for why nets converge at all, there's this paper whichI believe tries to establish some theory why bigger nets don't have such a big problem with local optima: http://arxiv.org/abs/1412.0233

      HoushalterView on HNParent context

  8. Machine Learning: Curse of Dimensionality 1 comment

    • As a counterpoint, the Blessing of Dimensionality is that local minima are mostly as good as the global minima. In David Silver's RLDM 2015 talk @20 mins in http://videolectures.net/rldm2015_silver_reinforcement_learn... he recounts this as his experience. the paper The Loss…

      deepnetView on HNParent context

  9. What's Wrong with Deep Learning? 1 comment
  10. Deep Learning vs. Machine Learning vs. Pattern Recognition 1 comment

    • This is as good a post as any about the theory of why deep learning works so well - esp. if you know a little about spin models from statistical mechanics https://charlesmartin14.wordpress.com/2015/02/26/why-does-de... additional flavor: http://arxiv.org/pdf/1412.0233.pdf and…

      selimthegrimView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.