REPAIR: REnormalizing Permuted Activations for Interpolation Repair — mentioned by dontwearitout

Tue, 04 Jun 2024 00:25:27 GMT

I haven't heard the term "shallow basin hypothesis" but I know what it refers to, these two papers spring to mind for me: 1) Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs https://arxiv.org/abs/1802.10026 2) Visualizing the Loss Landscape of Neural Nets https://arxiv.org/abs/1712.09913 There's also a very interesting body of work on merging trained models, such as by interpolating between points in weight space, which relates to the concept of "basins" of similar solutions. Skim the intro of this if you're interested in learning more: https://arxiv.org/abs/2211.08403

Paper and discussion history. A mention is not an endorsement.

REPAIR: REnormalizing Permuted Activations for Interpolation Repair · HackerPapers discussions

REPAIR: REnormalizing Permuted Activations for Interpolation Repair — mentioned by dontwearitout