Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers — mentioned by p1esk

Tue, 02 Jan 2024 16:57:49 GMT

You're doing it again - presenting guesses as facts. Why would a resnet - a batch normalized network using ReLU activations suffer from vanishing gradient problem? Does it? Have you actually done the experiment you've described? I have, and I didn't see gradients vanish. Sometimes gradients exploded - likely from a bad weights initialization (to be clear - that's a guess), and sometimes they didn't, but even when they didn't the networks never converged. The best we can do is to say: "skip connections seem to help training deep networks, and we have a few guesses as why, none of which is very…

Paper and discussion history. A mention is not an endorsement.

Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers · HackerPapers discussions

Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers — mentioned by p1esk