Is SGD a Bayesian sampler? Well, almost — mentioned by sinenomine

Tue, 10 Jan 2023 01:22:27 GMT

Ironic, since the relatively recently discovered double descent makes it clear that bias-variance tradeoff as we know it from statistical learning theory simply doesn't apply to "overparameterized" deep models. Much of old theory is barely applicable and people are, understandably, bewildered and in denial. If someone were to be inclined to theory, I'd just recommend reading papers that don't try oversimplify the domain: https://arxiv.org/abs/2006.15191 https://arxiv.org/abs/2210.10749 https://arxiv.org/abs/2205.10343 https://arxiv.org/abs/2105.04026

Paper and discussion history. A mention is not an endorsement.

Is SGD a Bayesian sampler? Well, almost — mentioned by sinenomine

Fri, 30 Dec 2022 18:44:32 GMT

If gradient-descent in DNNs indeed has an intrinsic bias for short and simple solutions, like this paper https://arxiv.org/abs/2006.15191 shows, and if the "reasoning" is among the shortest solutions to match the textual data with the language modeling loss, then at some limit of model+data scaling the model has to recover this solution. Or maybe even something better, "reasoning++"? It's all too easy to dismiss the success of infinitely plastic, scalable systems.

Paper and discussion history. A mention is not an endorsement.

Is SGD a Bayesian sampler? Well, almost · HackerPapers discussions

Is SGD a Bayesian sampler? Well, almost — mentioned by sinenomine

Is SGD a Bayesian sampler? Well, almost — mentioned by sinenomine