VeLO: Training Versatile Learned Optimizers by Scaling Up — mentioned by Legend2440

Tue, 18 Jul 2023 21:33:53 GMT

No one has the scale to make that happen. It's about information. Gradient-free methods integrate little or no information about the problem; they're a blind watchmaker. This works, but it's slow and gets slower the bigger your problem is. (curse of dimensionality) Gradients integrate some limited information about the problem. This lets you find solutions much faster, and neural networks are structured specifically to be easy to optimize with gradients. Local minima don't seem to be a problem. The future is probably even smarter optimizers that integrate more information about the problem…

Paper and discussion history. A mention is not an endorsement.

VeLO: Training Versatile Learned Optimizers by Scaling Up · HackerPapers discussions

VeLO: Training Versatile Learned Optimizers by Scaling Up — mentioned by Legend2440