Averaging Weights Leads to Wider Optima and Better Generalization · HackerPapers discussions

Averaging Weights Leads to Wider Optima and Better Generalization — mentioned by JieJie

Sat, 06 Apr 2024 19:22:29 GMT

From the Related Work section (best guess): Stochastic Weight Averaging (Izmailov et al 2018) https://arxiv.org/abs/1803.05407 Latest Weight Averaging (Kaddour 2022) https://arxiv.org/abs/2209.14981 Latest Weight Averaging? (Sanyal et al 2023) https://arxiv.org/abs/2311.16294 Cyclic Learning Rates (Portes et al 2022) https://arxiv.org/abs/2206.00832 Exponential Moving Average? (Zhanghan? et al 2019) https://arxiv.org/abs/1909.01804

Paper and discussion history. A mention is not an endorsement.

Averaging Weights Leads to Wider Optima and Better Generalization — mentioned by chillee

Mon, 27 Jul 2020 02:56:53 GMT

This just sounds like Stochastic Weight Averaging, which works quite well: https://arxiv.org/abs/1803.05407

Paper and discussion history. A mention is not an endorsement.

Averaging Weights Leads to Wider Optima and Better Generalization — mentioned by sillysaurusx

Thu, 09 Jan 2020 19:13:12 GMT

https://arxiv.org/abs/1803.05407 "Averaging Weights Leads to Wider Optima and Better Generalization" Weirdly, the averaging doesn't have to be synchronous.

Paper and discussion history. A mention is not an endorsement.