Structured Denoising Diffusion Models in Discrete State-Spaces · HackerPapers discussions

Structured Denoising Diffusion Models in Discrete State-Spaces — mentioned by jaaustin

Mon, 20 Oct 2025 15:19:18 GMT

To my knowledge this connection was first noted in 2021 in https://arxiv.org/abs/2107.03006 (page 5). We wanted to do text diffusion where you’d corrupt words to semantically similar words (like “quick brown fox” -> “speedy black dog”) but kept finding that masking was easier for the model to uncover. Historically this goes back even further to https://arxiv.org/abs/1904.09324, which made a generative MLM without framing it in diffusion math.

Paper and discussion history. A mention is not an endorsement.

Structured Denoising Diffusion Models in Discrete State-Spaces — mentioned by thomasahle

Thu, 22 May 2025 19:30:29 GMT

> Now, diffusion LMs take this idea further. BERT can recover 15% of masked tokens ("noise"), but why stop here. Let's train a model to recover texts with 30%, 50%, 90%, 100% of masked tokens. > Once you've trained that, in order to generate something from scratch, you start by feeding the model all [MASK]s. It will generate you mostly gibberish, but you can take some tokens (let's say, 10%) at random positions and assume that these tokens are generated ("final") This is clearly wrong. If you actually froze 10% of gibberish tokens, your output would be terrible! What you actually do in…

Paper and discussion history. A mention is not an endorsement.

Structured Denoising Diffusion Models in Discrete State-Spaces — mentioned by jdeaton

Sun, 11 Dec 2022 19:22:44 GMT

Because of criteria 0, 1, 2 these entirely depend on the individual. However, some papers which fit the criteria for me at the time were the following: - Score-Based Generative Modeling through Stochastic Differential Equations https://arxiv.org/abs/2011.13456 - Structured Denoising Diffusion Models in Discrete State-Spaces https://arxiv.org/abs/2107.03006 - Efficient and Modular Implicit Differentiation https://arxiv.org/abs/2105.15183 - Scalable Gradients for Stochastic Differential Equations https://arxiv.org/abs/2001.01328 - Bayesian Optimization with Unknown Constraints…

Paper and discussion history. A mention is not an endorsement.