Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution · HackerPapers discussions

Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution — mentioned by mxs_

Mon, 07 Jul 2025 20:54:16 GMT

In their tech report, they say this is based on: > "Our methods extend [28] through careful modifications to the data and computation to scale up learning." [28] is Lou et al. (2023), the "Score Entropy Discrete Diffusion" (SEDD) model (https://arxiv.org/abs/2310.16834). I wrote the first (as far as I can tell) independent from-scratch reimplementation of SEDD: https://github.com/mstarodub/dllm My goal was making it as clean and readable as possible. I also implemented the more complex denoising strategy they described (but didn't implement). It runs on a single GPU in a few hours on a toy…

Paper and discussion history. A mention is not an endorsement.

Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution — mentioned by AlexCoventry

Thu, 01 May 2025 00:27:33 GMT

Thanks, yes, I was thinking specifically of "Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution". They actually consider two noise distributions: one with uniform sampling for each noised token position, and one with a terminal masking (the Q^{uniform} and Q^{absorb}.) However, the terminal-masking system is clearly superior in their benchmarks. https://arxiv.org/pdf/2310.16834#page=6

Paper and discussion history. A mention is not an endorsement.

Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution — mentioned by freeqaz

Thu, 01 May 2025 00:00:57 GMT

Here are two papers linked from Inception's site: 1. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution - https://arxiv.org/abs/2310.16834 2. Simple and Effective Masked Diffusion Language Models - https://arxiv.org/abs/2406.07524

Paper and discussion history. A mention is not an endorsement.

Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution — mentioned by AlexCoventry

Sat, 15 Mar 2025 01:47:33 GMT

The SEDD architecture [1] probably allows for parallel sampling of all tokens in a block at once, which may be faster but not necessarily less computationally demanding in terms of runtime times computational resources used. [1] Which Inception Labs's new models may be based on; one of the cofounders is a co-author. See equations 18-20 in https://arxiv.org/abs/2310.16834

Paper and discussion history. A mention is not an endorsement.

Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution — mentioned by cs702

Thu, 06 Mar 2025 22:35:26 GMT

Few details about the model have been made public, as far as I know. However, Steano Ermon, one of the founders of Inception Labs, the company that released the model, co-authored a paper in late 2023 (revised in 2024), titled "Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution." The authors tested their proposed method on small autoregressive language modeling tasks, and claimed promising results: Paper: https://arxiv.org/abs/2310.16834 Code: https://github.com/louaaron/Score-Entropy-Discrete-Diffusion Blog post: https://aaronlou.com/blog/2024/discrete-diffusion/…

Paper and discussion history. A mention is not an endorsement.

Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution — mentioned by volodia

Wed, 26 Feb 2025 22:38:52 GMT

Not today, but we will be following up with a technical report over the next week or so. In the meantime, you can take a look at some of the research papers that inspired our work: - https://arxiv.org/abs/2310.16834 - https://arxiv.org/abs/2406.07524

Paper and discussion history. A mention is not an endorsement.

Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution — mentioned by jxmorris12

Fri, 21 Feb 2025 19:20:45 GMT

Thanks for bringing this up! As far as I understand it current text diffusion models are limited to fairly short context windows. The idea of a text diffusion model continuously updating and revising a million-token-long chain-of-thought is pretty mind-boggling. I agree that these non-autoregressive models could potentially behave in completely different ways. That said, I'm pretty sure we're a long way from building equally-competent diffusion-based base models, let alone reasoning models. If anyone's interested in this topic, here are some more foundational papers to take a look at…

Paper and discussion history. A mention is not an endorsement.