HackerPapers

arXiv:1412.6980cs.LG

Adam: A Method for Stochastic Optimization

Diederik P. Kingma, Jimmy Ba

Abstract

We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.

arXiv comments
Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 8 identifiable commenters in 8 comments across 8 threads, from Jan 2017 to Oct 2025. Found in 1 HN submission from Oct 2015 to Oct 2015. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments8 threads

  1. Researchers Discover the Optimal Way to Optimize 1 comment

    • One of OpenAI's founding team members developed Adam [0] well before it was flashy and profitable. It's not like nobody is out there trying to develop new algorithms. The reality is that there are some great, mature solvers out there that work well enough for most cases. And…

      broscoView on HNParent context

  2. The PhD Metagame: Don't try to reform science – not yet 1 comment

    • As a (former) reviewer at 5 journals, I disagree first and foremost with the notion that > We need some process to gate-keep. Journals, when print was the medium through which academic research was disseminated, had to gatekeep because there were practical considerations…

      cherryteastainView on HNParent context

  3. Anthropic hires OpenAI co-founder Durk Kingma 1 comment

    • Kingma is most notable for writing one of the most cited papers in AI. Actually one of the most cited scientific papers ever published, right up there with the transformers paper if not higher. "Adam: A Method for Stochastic Optimization" https://arxiv.org/abs/1412.6980 I remain…

      modelessView on HNParent context

  4. xLSTM: Extended Long Short-Term Memory 1 comment

    • Acquisitions, like for DeepMind is usually a way to hire talent. It can make sense when the technology is new and getting a few year of lead time on what is going to be a growing market may make some financial sense. In this specific xLSTM case, the industry has matured, they…

      GistNoesisView on HNParent context

  5. A Theory on Adam Instability in Large-Scale Machine Learning 1 comment
  6. John Carmack’s ‘Different Path’ to Artificial General Intelligence 1 comment

    • Frankly, one does not need this paper to get towards the AI. Adam the optimization algo you might need (and even there I am not sure). And it is very readable. The fact that this particular proof of Adam's convergence is complicated is largely irrelevant.…

      lostmsuView on HNParent context

  7. Variational Inference and Deep Learning: A New Synthesis [pdf] 1 comment

    • This is the PhD thesis of Durk Kingma [1]. Reading this right now; appears to be a well-written introduction to stuff variational autoencoders [2] and ADAM [3]. [1] http://dpkingma.com/ [2] http://arxiv.org/abs/1312.6114 [3] http://arxiv.org/abs/1412.6980

      aqsaloseView on HNParent context

  8. A New Batch of Deep Learning Startups to Watch for in 2017 1 comment

    • This seems like a sincere comment, so in case it's helpful, here's my interpretation of why others may be downvoting it. Deep learning practitioners are aware that they're using non-convex optimization, stochastic gradient descent, and so on. In fact, those topics are core to…

      brandonbView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.