Deep Ensembles: A Loss Landscape Perspective · HackerPapers discussions

Deep Ensembles: A Loss Landscape Perspective — mentioned by andy99

Thu, 08 May 2025 22:47:14 GMT

Different initialization converge to different places, e.g https://arxiv.org/abs/1912.02757 For LLMs (as with other models), many local optima appear to support roughly the same behavior. This is the idea of the problem being under-specified ie many more equations than unknowns so there are many ways to get the same result.

Paper and discussion history. A mention is not an endorsement.

Deep Ensembles: A Loss Landscape Perspective — mentioned by version_five

Thu, 23 Feb 2023 16:24:21 GMT

Related is this one https://arxiv.org/abs/1912.02757 which as I understand finds that there's little to be gained moving around locally in weight space, vs finding another mode. So it's not surprising that a model positioned between two modes would function sort of like an ensemble model

Paper and discussion history. A mention is not an endorsement.