Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping — mentioned by bcaine

Tue, 12 Oct 2021 14:14:14 GMT

While I sort of agree that machine learning will end up as an experimental science, it's way, way too early to say whether the theory relating deep learning to kernel methods (e.g. Neural Tangent Kernels) will be useful or not. As an example, just last week a (huge) paper [1] was put on arXiv that used these theoretical methods to analyze a bunch of common architecture building blocks (skip connections, normalization, etc), and then applied their theoretical findings to figure out how to train Resnet like models in similar training time without these seemingly "required" building blocks. Deep…

Paper and discussion history. A mention is not an endorsement.

Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping · HackerPapers discussions

Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping — mentioned by bcaine