Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs · HackerPapers discussions

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs — mentioned by oofbey

Sun, 14 Jun 2026 20:06:23 GMT

Correct. We used to think that because NN optimization is non-convex there are all these local minima. Now we know that once you get past the very early parts of training from random init, the loss surface is fairly smooth, and not really convex, but close enough in a bunch of ways - linear combinations of trained models are pretty much always valid combinations. You can think of fine tunings as deltas on the original model which can be summed together successfully. I think this paper first showed that to me: https://arxiv.org/pdf/1802.10026 which was 8 years ago now.

Paper and discussion history. A mention is not an endorsement.

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs — mentioned by dontwearitout

Tue, 04 Jun 2024 00:25:27 GMT

I haven't heard the term "shallow basin hypothesis" but I know what it refers to, these two papers spring to mind for me: 1) Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs https://arxiv.org/abs/1802.10026 2) Visualizing the Loss Landscape of Neural Nets https://arxiv.org/abs/1712.09913 There's also a very interesting body of work on merging trained models, such as by interpolating between points in weight space, which relates to the concept of "basins" of similar solutions. Skim the intro of this if you're interested in learning more: https://arxiv.org/abs/2211.08403

Paper and discussion history. A mention is not an endorsement.

Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs — mentioned by blackbear_

Thu, 23 Feb 2023 14:13:05 GMT

Cool I guess but not really surprising given previous work on mode connectivity ((1) and related) (1) https://arxiv.org/abs/1802.10026

Paper and discussion history. A mention is not an endorsement.