Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning — mentioned by charleshmartin

Tue, 12 Oct 2021 14:59:58 GMT

Here's an alternative approach, that actually provides real world results https://calculatedcontent.com/2019/12/03/towards-a-new-theor... Using techniques from statistical mechanics and strongly correlated systems, we can compute the average-case-behavior of a real world DNN We believe we can reproduce some of the results of the NTK by using a Gaussian Random Matrix. But if we use a more realistic, heavy tailed matrix, we get more practical results A early fork of the theory has been published in JMLR https://arxiv.org/abs/1810.01075 and the empirical results in Nature Communications…

Paper and discussion history. A mention is not an endorsement.

Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning — mentioned by charleshmartin

Sat, 10 Jul 2021 05:02:34 GMT

Great question. 4 is at the high edge of the fat tailed universality class. Most high performing models have alpha approaching 2, or at least below 3. See Figure 8(a) in the Nature paper, and our upcoming JMLR paper https://arxiv.org/abs/1810.01075

Paper and discussion history. A mention is not an endorsement.

Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning · HackerPapers discussions

Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning — mentioned by charleshmartin

Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning — mentioned by charleshmartin