Empirical Analysis of the Hessian of Over-Parametrized Neural Networks — mentioned by ericjang

Tue, 13 Nov 2018 18:40:32 GMT

Yes, a recent exciting phenomena of interest to researchers is how and why the spectrum of the Hessian appears to separate into 2 parts - a "bulk" part that changes very slowly and "outliers" that change quickly. This suggests that only a few weights in the model actually change during training. If one could determine which weights these are, it might lend to faster and more efficient learning algorithms that don't have to backprop to all the parameters in a large neural network. https://arxiv.org/pdf/1706.04454.pdf https://openreview.net/forum?id=ByeTHsAqtX

Paper and discussion history. A mention is not an endorsement.

Empirical Analysis of the Hessian of Over-Parametrized Neural Networks · HackerPapers discussions

Empirical Analysis of the Hessian of Over-Parametrized Neural Networks — mentioned by ericjang