HackerPapers

arXiv:1910.02875cs.LG

The asymptotic spectrum of the Hessian of DNN throughout training

Arthur Jacot, Franck Gabriel, Clément Hongler

Early discovery. Linked by one identifiable commenter. This paper is outside the main ranking; a mention is not an endorsement.

Abstract

The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs. When the NTK is fixed during training, we obtain a full characterization of the asymptotics of the spectrum of the Hessian, at initialization and during training. In the so-called mean-field limit, where the NTK is not fixed during training, we describe the first two moments of the Hessian at initialization.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.NENeural and Evolutionary Computing stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 1 identifiable commenter in 1 comment across 1 thread, from Oct 2020 to Oct 2020.

Comments linking this paper, per year.

Discussed in comments1 thread

  1. Why deep learning works even though it shouldn’t 1 comment

    • True, but it has been confirmed experimentally several times. Unfortunately theoretical understanding in deep learning is quite lagging behind practical progress. This paper [1] seems to be getting closer to what you are looking for. [1] https://arxiv.org/abs/1910.02875

      blackbear_View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.