Gradient Descent: The Ultimate Optimizer · HackerPapers discussions

Gradient Descent: The Ultimate Optimizer — mentioned by blackbear_

Fri, 20 Jan 2023 13:45:27 GMT

In the general case no because the final metric we care about (eg, accuracy or AUC) is not differentiable (ie, we cannot compute its gradient) with respect to the hyperparameters, especially the discrete ones. However a recent work at this year's NeurIPS [1] did indeed use an outer gradient descent to tune an inner gradient descent so in some cases yours is indeed a good idea that works. [1] https://arxiv.org/abs/1909.13371

Paper and discussion history. A mention is not an endorsement.

Gradient Descent: The Ultimate Optimizer — HN submission

Thu, 20 Oct 2022 14:18:04 GMT

Gradient Descent: The Ultimate Optimizer

6 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Gradient Descent: The Ultimate Optimizer — HN submission

Wed, 02 Oct 2019 23:09:39 GMT

Gradient Descent: The Ultimate Optimizer

191 points; 41 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Gradient Descent: The Ultimate Optimizer — HN submission

Tue, 01 Oct 2019 07:02:23 GMT

Gradient Descent: The Ultimate Optimizer

17 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.