Practical recommendations for gradient-based training of deep architectures · HackerPapers discussions

Practical recommendations for gradient-based training of deep architectures — HN submission

Wed, 25 May 2016 02:16:23 GMT

Ask HN: What Is the Best Way to Tune Hyper Parameters for a Deep Neural Network?

I found this quite tricky. Paper Practical Recommendations for Gradient-Based Training of Deep Architectures http://arxiv.org/pdf/1206.5533v2.pdf is a good source. What else?

3 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Practical recommendations for gradient-based training of deep architectures — mentioned by sabalaba

Tue, 30 Dec 2014 09:15:26 GMT

I would add "Practical recommendations for gradient-based training of deep architectures" to the list for those who already have a feel for training multi-layer neural nets. It provides a good overview for those that want to learn more about gradient descent, hyperparameter tuning, and other practical considerations involved with training deep architectures. http://arxiv.org/abs/1206.5533

Paper and discussion history. A mention is not an endorsement.

Practical recommendations for gradient-based training of deep architectures — mentioned by agibsonccc

Tue, 27 May 2014 05:49:35 GMT

To address some of the comments being presented here, neural nets despite being harder to train can be debugged visually. A few tips for those of you who use neural nets: Debug the weights with histograms. Track the gradient and make sure the magnitude is not too large and its normally distributed. Keep track of your gradient changes when using either gradient descent or conjugate gradient. Plot your filters, visualize what each neuron is learning. Watch the rate of change of your cost function. If it seems like its changing too fast and stops early lower your learning rate. Plot your…

Paper and discussion history. A mention is not an endorsement.

Practical recommendations for gradient-based training of deep architectures — mentioned by kkjkok

Fri, 15 Nov 2013 01:57:53 GMT

Check out this paper: Practical recommendations for gradient-based training of deep architectures, Y. Bengio http://arxiv.org/abs/1206.5533 There is a section on weight initialization on page 15. In general, this paper has a lot of good information in one place.

Paper and discussion history. A mention is not an endorsement.