Deep Unsupervised Learning using Nonequilibrium Thermodynamics · HackerPapers discussions

Deep Unsupervised Learning using Nonequilibrium Thermodynamics — mentioned by LarsDu88

Sun, 29 Sep 2024 04:30:35 GMT

I was trying to remember where I remember where I heard of this author's name before. Invented the first generative diffusion model in 2015. https://arxiv.org/abs/1503.03585

Paper and discussion history. A mention is not an endorsement.

Deep Unsupervised Learning using Nonequilibrium Thermodynamics — mentioned by janalsncm

Wed, 04 Sep 2024 19:07:37 GMT

One that many people may be familiar with is Stable Diffusion, which is used in many AI image generators today. There is an analogy between random noise -> image and a random distribution of gas particles -> concentrated volume of particles. https://arxiv.org/abs/1503.03585

Paper and discussion history. A mention is not an endorsement.

Deep Unsupervised Learning using Nonequilibrium Thermodynamics — mentioned by godelski

Tue, 06 Aug 2024 08:26:48 GMT

> The rule in ML There's definitely attempts to revive things (in the general sense, not just GANs), but most successes appear to come from large labs dedicating equal computational resources to the older models and often by changing names. This can make things more confusing and make things appear to be changing faster, but once you can see this, you'll have an easier time keeping up (so being new, watch out for this). I'll give some examples that are easier to read[0-2] (i.e. don't need expert knowledge to understand the nuances). As an insider (ML researcher), my complaint isn't so much…

Paper and discussion history. A mention is not an endorsement.

Deep Unsupervised Learning using Nonequilibrium Thermodynamics — mentioned by reasonableklout

Sun, 26 May 2024 21:33:14 GMT

The original Dickstein 2015 paper [1] formulated diffusion as maximizing (a lower bound of) log-likelihood of generating the distribution, so there was some theory. But my understanding is that the breakthrough was empirical results from Ho [2] and Nichol [3] showing diffusion could produce not only high-quality samples but better than GANs in some cases. [1] https://arxiv.org/abs/1503.03585 [2] https://arxiv.org/abs/2006.11239 [3] https://arxiv.org/abs/2105.05233

Paper and discussion history. A mention is not an endorsement.

Deep Unsupervised Learning using Nonequilibrium Thermodynamics — mentioned by godelski

Sun, 25 Feb 2024 18:59:25 GMT

Yes and no. The thing is that Mamba is not perfect. There's no neural architecture to rule them all, if you will. I think the bigger issue is that we more act like there is and get on bandwagons. Let me give a clearer example from the past so we can see. The predecessor to DDPM (the work that kicked off the diffusion model era) was published in 2015[0], only a year after GANs[1]. Diffusion then showed promise but why did DDPM come out in 2020[2]? Because everyone was working on GANs. GANs produced far better images and diffusion (still is) was a lot more computationally intensive. Plus, all…

Paper and discussion history. A mention is not an endorsement.

Deep Unsupervised Learning using Nonequilibrium Thermodynamics — mentioned by godelski

Sun, 02 Jul 2023 23:10:11 GMT

Yeah no problem, this is even closer to my area of focus! What do you know about physics and thermodynamics? I'd say a good intro for low background is from Tomczak[0]. He has a book, but the blog posts are nearly identical. He did a post doc with Max Welling (someone you should learn about if you want to get deep, like I was suggesting before). So I'd switch things up slightly. I'd go Intro -> Autoregressive -> Flow -> VAE -> Hierarchical VAE -> Energy Based Models -> Diffusion. It is worth learning about GANs btw, but this progression should be natural and build up. Continuing from there…

Paper and discussion history. A mention is not an endorsement.

Deep Unsupervised Learning using Nonequilibrium Thermodynamics — mentioned by espadrine

Sat, 07 Jan 2023 15:06:22 GMT

> The explanation in the original paper turns out not to be true; you can get rid of most of their assumptions and it still works I’ll admit it is amusing that some assumptions on why it works were incorrect. The core idea of a Markov chain[0] where each state change leads to higher likelihood, is bound to work, even if the rest doesn’t. In my mind, the Muse paper[1] gets closer to why it works: ultimately, the denoiser tries to match the latent space for an implicit encoder. The Muse system does this more directly and more effectively, by using cross-entropy loss on latent tokens instead. In…

Paper and discussion history. A mention is not an endorsement.

Deep Unsupervised Learning using Nonequilibrium Thermodynamics — mentioned by wokwokwok

Wed, 05 Oct 2022 07:36:51 GMT

> But, as we take a bunch of small steps and gradually move back through the diffusion process... ...but, the question is, why can't we take a big step and be at the end in one step. Obviously a series of small steps gets you there, but the question was why you need to take small steps. I feel like this is just a 'intuitive explanation' that doesn't actually do anything other than rephrase the question; "You take a series of small steps to reduce the noise in each step and end up with a picture with no noise". The real reason is that big steps result in worse results (1); the model was…

Paper and discussion history. A mention is not an endorsement.

Deep Unsupervised Learning using Nonequilibrium Thermodynamics — mentioned by Imnimo

Wed, 06 Apr 2022 20:47:41 GMT

I recommend this blog post: https://lilianweng.github.io/posts/2021-07-11-diffusion-mode... Personally, I find the core diffusion papers pretty dense and difficult to follow, so the blog post is where I'd begin. https://arxiv.org/pdf/1503.03585.pdf This paper is a decent starting point on the literature side, but it's a doozy. Both the paper and blog post are pretty math heavy. I have not yet found a really clear intuitive explanation that doesn't get down in the weeds of the math, and it took me a long time to understand what the hell the math is trying to say (and there are some parts I…

Paper and discussion history. A mention is not an endorsement.

Deep Unsupervised Learning using Nonequilibrium Thermodynamics — HN submission

Sat, 23 May 2015 15:47:38 GMT

Deep Unsupervised Learning Using Nonequilibrium Thermodynamics [pdf]

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.