The jamming transition as a paradigm to understand the loss landscape of deep neural networks — mentioned by cs702

Tue, 20 Oct 2020 21:11:34 GMT

TL;DR: For high-dimensional models (say, with millions to billions of parameters), there's always a good set parameters nearby, and when we start descending towards it, we are highly unlikely to get stuck, because almost always there's at least one path down along at least one among of all those dimensions -- i.e., there are no local optima. Once we've stumbled upon a good set of parameters, as measured by validation, we can stop. These intuitions are consistent with my experience... but I think there's more to deep learning. For instance, these intuitions fail to explain "weird" phenomena…

Paper and discussion history. A mention is not an endorsement.

The jamming transition as a paradigm to understand the loss landscape of deep neural networks · HackerPapers discussions

The jamming transition as a paradigm to understand the loss landscape of deep neural networks — mentioned by cs702