DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models — mentioned by namibj

Tue, 15 Jul 2025 18:52:40 GMT

> One thing we can do which current LLMs can't, at least directly as far as I'm aware, is to go back and re-read a section. Like on-demand RAG, or something. Encoders can do that. And we can use them with diffusion to generate text [0]. This works because you don't impose a masked self attention for autoregressive decoding in the encoder, so subsequent layers can re-focus their key/query vector space to steer "backwards" information flow. Happy reading. Feel free to get back! [0]: https://arxiv.org/abs/2211.15029

Paper and discussion history. A mention is not an endorsement.

DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models · HackerPapers discussions

DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models — mentioned by namibj