HackerPapers

arXiv:1503.03585cs.LG

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, Surya Ganguli

Abstract

A central problem in machine learning involves modeling complex data-sets using highly flexible families of probability distributions in which learning, sampling, inference, and evaluation are still analytically or computationally tractable. Here, we develop an approach that simultaneously achieves both flexibility and tractability. The essential idea, inspired by non-equilibrium statistical physics, is to systematically and slowly destroy structure in a data distribution through an iterative forward diffusion process. We then learn a reverse diffusion process that restores structure in data, yielding a highly flexible and tractable generative model of the data. This approach allows us to rapidly learn, sample from, and evaluate probabilities in deep generative models with thousands of layers or time steps, as well as to compute conditional and posterior probabilities under the learned model. We additionally release an open source reference implementation of the algorithm.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cond-mat.dis-nnDisordered Systems and Neural Networks q-bio.NCNeurons and Cognition stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 7 identifiable commenters in 9 comments across 9 threads, from Apr 2022 to Sep 2024. Found in 1 HN submission from May 2015 to May 2015. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments9 threads

  1. Too much efficiency makes everything worse (2022) 1 comment

    • I was trying to remember where I remember where I heard of this author's name before. Invented the first generative diffusion model in 2015. https://arxiv.org/abs/1503.03585

      LarsDu88View on HNParent context

  2. Physics is unreasonably good at creating new math 1 comment

    • One that many people may be familiar with is Stable Diffusion, which is used in many AI image generators today. There is an analogy between random noise -> image and a random distribution of gas particles -> concentrated volume of particles. https://arxiv.org/abs/1503.03585

      janalsncmView on HNParent context

  3. The Path to StyleGan2 – Implementing the Progressive Growing GAN 1 comment

    • > The rule in ML There's definitely attempts to revive things (in the general sense, not just GANs), but most successes appear to come from large labs dedicating equal computational resources to the older models and often by changing names. This can make things more confusing…

      godelskiView on HNParent context

  4. Diffusion Models 1 comment

    • The original Dickstein 2015 paper [1] formulated diffusion as maximizing (a lower bound of) log-likelihood of generating the distribution, so there was some theory. But my understanding is that the breakthrough was empirical results from Ho [2] and Nichol [3] showing diffusion…

      reasonablekloutView on HNParent context

  5. Mamba Explained: The State Space Model Taking On Transformers 1 comment

    • Yes and no. The thing is that Mamba is not perfect. There's no neural architecture to rule them all, if you will. I think the bigger issue is that we more act like there is and get on bandwagons. Let me give a clearer example from the past so we can see. The predecessor to DDPM…

      godelskiView on HNParent context

  6. Stanford A.I. Courses 1 comment

    • Yeah no problem, this is even closer to my area of focus! What do you know about physics and thermodynamics? I'd say a good intro for low background is from Tomczak[0]. He has a book, but the blog posts are nearly identical. He did a post doc with Max Welling (someone you should…

      godelskiView on HNParent context

  7. Denoising Diffusion models from first principle in Julia 1 comment

    • > The explanation in the original paper turns out not to be true; you can get rid of most of their assumptions and it still works I’ll admit it is amusing that some assumptions on why it works were incorrect. The core idea of a Markov chain[0] where each state change leads to…

      espadrineView on HNParent context

  8. The Illustrated Stable Diffusion 1 comment

    • > But, as we take a bunch of small steps and gradually move back through the diffusion process... ...but, the question is, why can't we take a big step and be at the end in one step. Obviously a series of small steps gets you there, but the question was why you need to take…

      wokwokwokView on HNParent context

  9. Dall-E 2 1 comment

    • I recommend this blog post: https://lilianweng.github.io/posts/2021-07-11-diffusion-mode... Personally, I find the core diffusion papers pretty dense and difficult to follow, so the blog post is where I'd begin. https://arxiv.org/pdf/1503.03585.pdf This paper is a decent…

      ImnimoView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.