Identifying and attacking the saddle point problem in high-dimensional non-convex optimization · HackerPapers discussions

Identifying and attacking the saddle point problem in high-dimensional non-convex optimization — mentioned by cherryteastain

Thu, 25 Jun 2026 14:52:20 GMT

A related viewpoint is that overparametrization is good because the model is stranded when the Hessian has all positive/zero eigenvalues. If we treat the probability that a particular Hessian eigenvalue turns positive as a Bernoulli process, the chance of all eigenvalues going positive/zero exponentially decreases as the parameter count increases [1] https://arxiv.org/abs/1406.2572

Paper and discussion history. A mention is not an endorsement.

Identifying and attacking the saddle point problem in high-dimensional non-convex optimization — mentioned by muppet_frog

Tue, 20 Oct 2020 21:29:53 GMT

This paper makes the points that it's the saddles and not local minima that are the problem: https://arxiv.org/abs/1406.2572 It was the basis for adding 'momentum' to optimizers - so that you could skate across the saddles.

Paper and discussion history. A mention is not an endorsement.

Identifying and attacking the saddle point problem in high-dimensional non-convex optimization — mentioned by _0ffh

Wed, 20 Mar 2019 10:29:25 GMT

Maybe GP is talking about the hypothesis that in high-dimensional space you will probably almost always find /some/ direction which decreases the error function. IIRC this hypothesis is well supported, but not yet proven. So the real problem may not be local minima but saddle points that just /appear/ to be minima due to slow optimisation progress. For more details, check out https://arxiv.org/abs/1406.2572

Paper and discussion history. A mention is not an endorsement.

Identifying and attacking the saddle point problem in high-dimensional non-convex optimization — mentioned by andreyk

Wed, 20 Mar 2019 04:26:11 GMT

See "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization" https://arxiv.org/abs/1406.2572 The 'bowl' you described are these saddle points, and this 2014 paper was pretty big for tackling it. But, as others pointed out Adam or such optimizers are used more often than steepest descent.

Paper and discussion history. A mention is not an endorsement.

Identifying and attacking the saddle point problem in high-dimensional non-convex optimization — mentioned by chillee

Wed, 27 Dec 2017 00:57:43 GMT

That's why I said high dimensional neural networks. There's been a lot of literature explaining why local minima aren't a problem in very high dimension loss surfaces. Check any of the literature on this subject: https://arxiv.org/abs/1611.06310v2 https://arxiv.org/abs/1406.2572 Local minima are something that people thought was gonna be a problem, especially back in the 2000s. They played around with small neural nets on toy examples such as yours, and thought it was intractable. It's the entire reason why neural nets fell out of the fashion in the early 2000s, and people moved towards…

Paper and discussion history. A mention is not an endorsement.

Identifying and attacking the saddle point problem in high-dimensional non-convex optimization — mentioned by jhj

Mon, 30 Oct 2017 05:03:35 GMT

Minima/maxima in high-dimensional spaces are not much of an issue; the bigger issue are saddle points. cf https://arxiv.org/pdf/1406.2572.pdf This is presuming that the problem has a continuous formulation anyways.

Paper and discussion history. A mention is not an endorsement.

Identifying and attacking the saddle point problem in high-dimensional non-convex optimization — mentioned by hiddencost

Tue, 12 Jan 2016 00:07:06 GMT

Your statement is true-ish, but not true enough. It is true there are more local minima, but they're generally pretty equivalent. In fact, in high dimensions, saddle points are a much bigger problem than local minima. [0] The fact that optimization works on these large, non-convex problems is kind of a marvel, and raises really interesting questions. (mostly "this shouldn't work this well unless there's some kind of regularity to these problems that makes them nicely behaved"). What this says about evolution I neither know nor care to speculate (above my pay grade). [0]…

Paper and discussion history. A mention is not an endorsement.

Identifying and attacking the saddle point problem in high-dimensional non-convex optimization — mentioned by modeless

Fri, 04 Dec 2015 18:35:18 GMT

Because our intuition about local minima is wrong in extremely high dimensional spaces. In two and three dimensions, local minima are common. In a million dimensions, local minima are rare. The intuitive explanation is that for a local minimum to exist, the function must be curving up (first derivative = 0, second derivative >= 0) simultaneously in every dimension. It makes sense that as you add more dimensions this becomes less and less likely, and for a million dimensions it's vanishingly unlikely. What you get instead of local minima are saddle points, where some dimensions are curving up…

Paper and discussion history. A mention is not an endorsement.

Identifying and attacking the saddle point problem in high-dimensional non-convex optimization — HN submission

Tue, 24 Feb 2015 15:39:15 GMT

Ask HN: How do you find important/interesting new papers

I'm having a problem with information overload and I'm betting that some HN readers have some knowledge that will be helpful. I'm an undergrad studying computer science and math and I recently came across this paper: http://arxiv.org/abs/1406.2572, which prompted me to begin reading more about optimization, random matrix theory, and other related topics. The problem I'm facing is that I don't know what the good papers to read are and there are so many new ones made available each day that taking the time to figure out which are relevant to me and which aren't would take more time than I have…

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.