HackerPapers

arXiv:1611.07476cs.LG

Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond

Levent Sagun, Leon Bottou, Yann LeCun

Early discovery. Linked by one identifiable commenter. This paper is outside the main ranking; a mention is not an endorsement.

Abstract

We look at the eigenvalues of the Hessian of a loss function before and after training. The eigenvalue distribution is seen to be composed of two parts, the bulk which is concentrated around zero, and the edges which are scattered away from zero. We present empirical evidence for the bulk indicating how over-parametrized the system is, and for the edges that depend on the input data.

arXiv comments
ICLR submission, 2016 - updated to match the openreview.net version
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 1 identifiable commenter in 1 comment across 1 thread, from Oct 2020 to Oct 2020.

Comments linking this paper, per year.

Discussed in comments1 thread

  1. Why deep learning works even though it shouldn’t 1 comment

    • There is another reason why training deep neural networks is not as difficult as it sounds: the landscape of the loss function seems to be made of broad "U"-shaped valleys that gently descend towards a small loss region. At initialization, the network is likely close to such…

      blackbear_View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.