Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations — mentioned by jiayaozhang

Fri, 15 Oct 2021 01:03:42 GMT

Hello HackerNews! Author here :) TL;DR: We devise a linear SDE/ODE model to imitate per-class feature (thinking logits) dynamics of neural nets training based on local elasticity (LE) [1]. We found the emergence of LE implies linear separation of features from different classes as training progresses. The drift matrix of our model has a relatively simple structure; with that estimated, we can simulate the SDE using the forward Euler method, whose results align reasonably well with genuine dynamics. Local elasticity models the phenomenon observed in DNN training: the effect due to training on…

Paper and discussion history. A mention is not an endorsement.

Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations — HN submission

Thu, 14 Oct 2021 13:51:55 GMT

Imitating Deep Learning Dynamics via Stochastic Differential Equations

122 points; 21 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations · HackerPapers discussions

Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations — mentioned by jiayaozhang

Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations — HN submission