Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers · HackerPapers discussions

Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers — mentioned by kajecounterhack

Thu, 03 Mar 2022 23:44:19 GMT

OP doesn't weight the size of each technology by adoption -- if they did that then yeah there are some centroids. But I think their point is correct that parts of the ecosystem remain quite fragmented. > For example, when I first got into ML the ADAM optimizer was the new big thing, since then hundreds of 'better' optimizers have been published. This paper from August '21 shows that most of that is overblown and no optimizer consistently outperforms ADAM: https://arxiv.org/pdf/2007.01547.pdf FWIW this paper acknowledges that optimizer is application-dependent, yet draws its conclusions from <…

Paper and discussion history. A mention is not an endorsement.

Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers — mentioned by a_bonobo

Thu, 03 Mar 2022 08:47:19 GMT

As a researcher applying ML in my work the OP reads like me looking from the outside at web-frameworks: there's a LOT of noise but in practice, people run just the same 3 things. For example, when I first got into ML the ADAM optimizer was the new big thing, since then hundreds of 'better' optimizers have been published. This paper from August '21 shows that most of that is overblown and no optimizer consistently outperforms ADAM: https://arxiv.org/pdf/2007.01547.pdf I'd even go so far and say that OP's big image of MLOps is misleading. 'Data Science Notebooks' shows Jupyter, binder, colab, I…

Paper and discussion history. A mention is not an endorsement.