HackerPapers

arXiv:1206.5533cs.LG

Practical recommendations for gradient-based training of deep architectures

Yoshua Bengio

Abstract

Learning algorithms related to artificial neural networks and in particular for Deep Learning may seem to involve many bells and whistles, called hyper-parameters. This chapter is meant as a practical guide with recommendations for some of the most commonly used hyper-parameters, in particular in the context of learning algorithms based on back-propagated gradient and gradient-based optimization. It also discusses how to deal with the fact that more interesting results can be obtained when allowing one to adjust many hyper-parameters. Overall, it describes elements of the practice used to successfully and efficiently train and debug large-scale and often deep multi-layer neural networks. It closes with open questions about the training difficulties observed with deeper architectures.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 3 identifiable commenters in 3 comments across 3 threads, from Nov 2013 to Dec 2014. Found in 1 HN submission from May 2016 to May 2016. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments3 threads

  1. Deep Learning Reading List 1 comment

    • I would add "Practical recommendations for gradient-based training of deep architectures" to the list for those who already have a feel for training multi-layer neural nets. It provides a good overview for those that want to learn more about gradient descent, hyperparameter…

      sabalabaView on HNParent context

  2. Deep Learning From The Bottom Up 1 comment

    • To address some of the comments being presented here, neural nets despite being harder to train can be debugged visually. A few tips for those of you who use neural nets: Debug the weights with histograms. Track the gradient and make sure the magnitude is not too large and its…

      agibsoncccView on HNParent context

  3. Deep Learning 101 1 comment

    • Check out this paper: Practical recommendations for gradient-based training of deep architectures, Y. Bengio http://arxiv.org/abs/1206.5533 There is a section on weight initialization on page 15. In general, this paper has a lot of good information in one place.

      kkjkokView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.