Hessian Averaging in Stochastic Newton Methods Achieves Superlinear Convergence — mentioned by hodgehog11

Thu, 15 Jan 2026 22:14:20 GMT

Just a heads up in case you didn't know, taking the Hessian over batches is indeed referred to as Stochastic Newton, and methods of this kind have been studied for quite some time. Inverting the Hessian is often done with CG, which tends to work pretty well. The only problem is that the Hessian is often not invertible so you need a regularizer (same as here I believe). Newton methods work at scale, but no-one with the resources to try them at scale seems to be aware of them. It's an interesting trick though, so I'd be curious to see how it compares to CG. [1] https://arxiv.org/abs/2204.09266…

Paper and discussion history. A mention is not an endorsement.

Hessian Averaging in Stochastic Newton Methods Achieves Superlinear Convergence · HackerPapers discussions

Hessian Averaging in Stochastic Newton Methods Achieves Superlinear Convergence — mentioned by hodgehog11