HackerPapers

arXiv:1712.09913cs.LG

Visualizing the Loss Landscape of Neural Nets

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein

Abstract

Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effects on the underlying loss landscape, are not well understood. In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple "filter normalization" method that helps us visualize loss function curvature and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture affects the loss landscape, and how training parameters affect the shape of minimizers.

arXiv comments
NIPS 2018 (extended version, 10.5 pages), code is available at https://github.com/tomgoldstein/loss-landscape
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.CVComputer Vision and Pattern Recognition stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 7 identifiable commenters in 7 comments across 7 threads, from Feb 2018 to Nov 2025. Found in 1 HN submission from Dec 2017 to Dec 2017. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. Backpropagation is a leaky abstraction (2016) 1 comment

    • Two thoughts: > how important is computing the exact gradient using calculus Normally the gradient is computed with a small "minibatch" of examples, meaning that on average over many steps the true gradient is followed, but each step individually never moves exacty along the…

      blackbear_View on HNParent context

  2. Grokfast: Accelerated Grokking by Amplifying Slow Gradients 1 comment

    • I haven't heard the term "shallow basin hypothesis" but I know what it refers to, these two papers spring to mind for me: 1) Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs https://arxiv.org/abs/1802.10026 2) Visualizing the Loss Landscape of Neural Nets…

      dontwearitoutView on HNParent context

  3. Gradient descent visualization 1 comment

    • Yes. To be more specific, it's that nearly all points where the derivative is zero are saddle points rather than minima. Note that some portion of this nice behavior seems to be due to design choices in modern architectures, like residual connections, rather than being a general…

      ImnimoView on HNParent context

  4. Training LLMs to generate text with citations via fine-grained rewards 1 comment

    • This is the one example I had in mind because of all the pretty pictures: https://arxiv.org/abs/1712.09913 Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein. 2018. "Visualizing the Loss Landscape of Neural Nets".

      nerdponxView on HNParent context

  5. Why deep learning works even though it shouldn’t 1 comment

    • ...hmm, that was counter to my understanding (limited though it may be...) which was partially formed by this paper: https://arxiv.org/abs/1712.09913 TLDR - loss landscapes are nasty, but you can tame them with skip connections.

      muppet_frogView on HNParent context

  6. An Idiot’s guide to Support vector machines (2003) [pdf] 1 comment

    • As others have said, you don't actually want the global optimum of a neural network because that would be terrible overfitting. There is some evidence that architectural tricks (like ResNet) that empirically help performance are making the loss landscape "more convex", though.…

      currymjView on HNParent context

  7. Greedy, Brittle, Opaque, and Shallow: The Downsides to Deep Learning 1 comment

    • I find your comment a little strange. First you say something interesting about Deep Dream being an early attempt at better understanding of vision networks but that it's not enough. Then you say the end is in sight? If you just take the visualization part we have a massive…

      hadsedView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.