Train faster, generalize better: Stability of stochastic gradient descent · HackerPapers discussions

Train faster, generalize better: Stability of stochastic gradient descent — mentioned by tdullien

Thu, 23 Jun 2016 13:01:12 GMT

I am a mathematician by trade, and was doing development along with other stuff (reverse engineering and security work, first in my own company, then at Google). So ... 1) I think working knowledge of ML is extremely useful to many developers, and generally under-taught in universities. See the old Joel article which mentions "Google uses Bayesian filtering like MS uses the IF statement" http://www.joelonsoftware.com/items/2005/10/17.html). A well-rounded developer should know the basics (Logistic regression, SVMs, know some things about CDNNs etc.), it will make him much more adept at…

Paper and discussion history. A mention is not an endorsement.

Train faster, generalize better: Stability of stochastic gradient descent — mentioned by jey

Tue, 21 Jun 2016 19:26:19 GMT

Stochastic gradient actually performs better in terms of generalization performance! https://arxiv.org/abs/1509.01240 The intuition for it is that when optimizing a machine learning objective all the way to machine precision, you at some point cross an (unknown) threshold where you are over-optimizing the parameters to the particular model class you're using, but that's probably too much faith in your model specification. So stochastic optimization and early stopping (before gradient is zero) provides a form of regularization.

Paper and discussion history. A mention is not an endorsement.