Visualizing the Loss Landscape of Neural Nets · HackerPapers discussions

Visualizing the Loss Landscape of Neural Nets — mentioned by blackbear_

Sun, 02 Nov 2025 07:51:40 GMT

Two thoughts: > how important is computing the exact gradient using calculus Normally the gradient is computed with a small "minibatch" of examples, meaning that on average over many steps the true gradient is followed, but each step individually never moves exacty along the true gradient. This noisy walk is actually quite beneficial for the final performance of the network https://arxiv.org/abs/2006.15081 , https://arxiv.org/abs/1609.04836 so much so that people started wondering what is the best way to "corrupt" this approximate gradient even more to improve performance…

Paper and discussion history. A mention is not an endorsement.

Visualizing the Loss Landscape of Neural Nets — mentioned by dontwearitout

Tue, 04 Jun 2024 00:25:27 GMT

I haven't heard the term "shallow basin hypothesis" but I know what it refers to, these two papers spring to mind for me: 1) Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs https://arxiv.org/abs/1802.10026 2) Visualizing the Loss Landscape of Neural Nets https://arxiv.org/abs/1712.09913 There's also a very interesting body of work on merging trained models, such as by interpolating between points in weight space, which relates to the concept of "basins" of similar solutions. Skim the intro of this if you're interested in learning more: https://arxiv.org/abs/2211.08403

Paper and discussion history. A mention is not an endorsement.

Visualizing the Loss Landscape of Neural Nets — mentioned by Imnimo

Wed, 08 May 2024 03:38:12 GMT

Yes. To be more specific, it's that nearly all points where the derivative is zero are saddle points rather than minima. Note that some portion of this nice behavior seems to be due to design choices in modern architectures, like residual connections, rather than being a general fact about all high dimensional problems. https://arxiv.org/pdf/1712.09913 This paper has some nice visualizations.

Paper and discussion history. A mention is not an endorsement.

Visualizing the Loss Landscape of Neural Nets — mentioned by nerdponx

Sun, 18 Feb 2024 03:11:36 GMT

This is the one example I had in mind because of all the pretty pictures: https://arxiv.org/abs/1712.09913 Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein. 2018. "Visualizing the Loss Landscape of Neural Nets".

Paper and discussion history. A mention is not an endorsement.

Visualizing the Loss Landscape of Neural Nets — mentioned by muppet_frog

Tue, 20 Oct 2020 21:26:08 GMT

...hmm, that was counter to my understanding (limited though it may be...) which was partially formed by this paper: https://arxiv.org/abs/1712.09913 TLDR - loss landscapes are nasty, but you can tame them with skip connections.

Paper and discussion history. A mention is not an endorsement.

Visualizing the Loss Landscape of Neural Nets — mentioned by currymj

Tue, 01 Jan 2019 06:28:24 GMT

As others have said, you don't actually want the global optimum of a neural network because that would be terrible overfitting. There is some evidence that architectural tricks (like ResNet) that empirically help performance are making the loss landscape "more convex", though. https://arxiv.org/pdf/1712.09913.pdf

Paper and discussion history. A mention is not an endorsement.

Visualizing the Loss Landscape of Neural Nets — mentioned by hadsed

Sun, 11 Feb 2018 11:07:45 GMT

I find your comment a little strange. First you say something interesting about Deep Dream being an early attempt at better understanding of vision networks but that it's not enough. Then you say the end is in sight? If you just take the visualization part we have a massive amount of work to do. Understanding how these networks solve problem in sophisticated ways will motivate new advances to mitigate shortcomings. It is a software problem that can have huge benefits. For example, see the paper Visualizing Loss Functions For Neural Networks as an example of the incredible insight to be gained…

Paper and discussion history. A mention is not an endorsement.

Visualizing the Loss Landscape of Neural Nets — HN submission

Fri, 29 Dec 2017 20:11:12 GMT

Visualizing the Loss Landscape of Neural Nets [pdf]

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.