Scalable Second Order Optimization for Deep Learning — mentioned by nil-sec

Wed, 29 Apr 2020 14:37:10 GMT

Actually there is no need for gradient decent at all if you have a linear regression problem, because you can find the min/max exactly by inverting your weight-matrix (pseudo-inverse if degenerate). Additionally, second order optimisation, i.e. using the Hessian in addition to the gradient is a well studied problem in the literature. My understanding is that it is in general not worth it because calculating the Hessian is very expensive (see e.g. https://arxiv.org/abs/2002.09018).

Paper and discussion history. A mention is not an endorsement.

Scalable Second Order Optimization for Deep Learning · HackerPapers discussions

Scalable Second Order Optimization for Deep Learning — mentioned by nil-sec