Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks — mentioned by criticaltinker

Sat, 16 Apr 2022 17:25:02 GMT

> we develop the notion of representation group flow (RG flow) to characterize the propagation of signals through the network. By tuning networks to criticality, we give a practical solution to the exploding and vanishing gradient problem I am surprised and a bit disappointed this paper does not mention mean field theory or dynamical isometry at all. Mean field theory applies methods from physics - namely random matrix and free probability theory - to derive an exact analytical solution for information flow through a neural network. It turns out that simply initializing the weights of a plain…

Paper and discussion history. A mention is not an endorsement.

Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks — HN submission

Wed, 20 Jun 2018 08:01:38 GMT

Dynamical Isometry and a Mean Field Theory of CNNs

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks · HackerPapers discussions

Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks — mentioned by criticaltinker

Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks — HN submission