Gradient-based Hyperparameter Optimization through Reversible Learning · HackerPapers discussions

Gradient-based Hyperparameter Optimization through Reversible Learning — mentioned by imurray

Mon, 02 Jun 2025 12:20:21 GMT

Another machine learning paper ("ancient", 2015) where being able to exactly reverse a computation was useful: https://arxiv.org/abs/1502.03492

Paper and discussion history. A mention is not an endorsement.

Gradient-based Hyperparameter Optimization through Reversible Learning — mentioned by gwern

Thu, 03 Oct 2019 00:35:24 GMT

No, it's been done. For example, "Gradient-based Hyperparameter Optimization through Reversible Learning" https://arxiv.org/abs/1502.03492 , Maclaurin et al 2015 (one of their cites). The idea is pretty obvious: of course you'd like to learn the hyperparameters like they were any other parameter. But that's easier said than done. The problem is that, well, to backpropagate through a hyperparameter, you would need to, say, track how it affects every iteration throughout the entire training run, rather than simply tracking one parameter through a single iteration on a single datapoint. And it's…

Paper and discussion history. A mention is not an endorsement.

Gradient-based Hyperparameter Optimization through Reversible Learning — HN submission

Tue, 02 Feb 2016 20:50:57 GMT

Gradient-Based Hyperparameter Optimization Through Reversible Learning

58 points; 16 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Gradient-based Hyperparameter Optimization through Reversible Learning — mentioned by Fede_V

Tue, 12 May 2015 08:42:52 GMT

Have you taken a look at this ? https://github.com/HIPS/autograd and paper http://arxiv.org/pdf/1502.03492.pdf They reverse the computation of the gradient to calculate the derivative wrt the hyper parameters.

Paper and discussion history. A mention is not an endorsement.