HackerPapers

arXiv:1502.03492stat.ML

Gradient-based Hyperparameter Optimization through Reversible Learning

Dougal Maclaurin, David Duvenaud, Ryan P. Adams

Abstract

Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyperparameters, including step-size and momentum schedules, weight initialization distributions, richly parameterized regularization schemes, and neural network architectures. We compute hyperparameter gradients by exactly reversing the dynamics of stochastic gradient descent with momentum.

arXiv comments
10 figures. Submitted to ICML
Journal reference
DOI
Subjects
stat.MLMachine Learning (Statistics) cs.LGMachine Learning

On Hacker News

Linked by 3 identifiable commenters in 3 comments across 3 threads, from May 2015 to Jun 2025. Found in 1 HN submission from Feb 2016 to Feb 2016. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments3 threads

  1. How can AI researchers save energy? By going backward 1 comment

    • Another machine learning paper ("ancient", 2015) where being able to exactly reverse a computation was useful: https://arxiv.org/abs/1502.03492

      imurrayView on HNParent context

  2. Gradient Descent: The Ultimate Optimizer 1 comment

    • No, it's been done. For example, "Gradient-based Hyperparameter Optimization through Reversible Learning" https://arxiv.org/abs/1502.03492 , Maclaurin et al 2015 (one of their cites). The idea is pretty obvious: of course you'd like to learn the hyperparameters like they were…

      gwernView on HNParent context

  3. Keras: Fast Deep Learning Prototyping for Python 1 comment

    • Have you taken a look at this ? https://github.com/HIPS/autograd and paper http://arxiv.org/pdf/1502.03492.pdf They reverse the computation of the gradient to calculate the derivative wrt the hyper parameters.

      Fede_VView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.