Krylov Subspace Descent for Deep Learning — mentioned by jph00

Sun, 23 Jul 2017 15:39:17 GMT

Krylov spaces often come up in derivations for optimization methods in DL, and have been studied as an explicit technique on their own. Eg see https://arxiv.org/abs/1111.4259 However more recent theoretical and empirical results show that these kinds of approaches don't deal well with the huge saddle point problem in DL loss functions. But momentum based techniques work great.

Paper and discussion history. A mention is not an endorsement.

Krylov Subspace Descent for Deep Learning · HackerPapers discussions

Krylov Subspace Descent for Deep Learning — mentioned by jph00