Step-unrolled Denoising Autoencoders for Text Generation · HackerPapers discussions

Step-unrolled Denoising Autoencoders for Text Generation — mentioned by thatguysaguy

Fri, 22 Dec 2023 21:10:17 GMT

Self-attention and positional embeddings are ancient at this point (i.e., more than 5 years old). Here are a couple examples of methods: https://arxiv.org/abs/2112.06749 https://arxiv.org/abs/2205.12558

Paper and discussion history. A mention is not an endorsement.

Step-unrolled Denoising Autoencoders for Text Generation — mentioned by kastnerkyle

Sat, 25 Mar 2023 14:22:18 GMT

Discrete diffusion with rewriting can work well. It feels loosely similar to backtracking, if you assume n_steps large enough - need to be able to rewrite any non-provided position though I think (not all setups do this). Downside is the noise in discrete diffusion (in simplest case randomizing over all vocabulary space) is pretty harsh and makes things very difficult practically. Don't have an exact reference on the relationship, but feels similar to backtracking type mechanics in my experience. I found things tend to "lock in" quickly once a good path is found, which feels a lot like…

Paper and discussion history. A mention is not an endorsement.

Step-unrolled Denoising Autoencoders for Text Generation — mentioned by kastnerkyle

Thu, 12 May 2022 18:36:52 GMT

I really like the descriptions from SUNDAE (https://arxiv.org/abs/2112.06749) if you have some background about general neural net style modeling, and generally find the multinomial or binomial diffusion settings a bit simpler to think about conceptually (if a bit more difficult in practice due to the harshness of the noise). There are other papers focused on these settings too (even the origin diffusion work in the NN sphere http://proceedings.mlr.press/v37/sohl-dickstein15.html) but again - the math is at the forefront (https://arxiv.org/abs/2111.14822 , https://arxiv.org/abs/2102.05379)…

Paper and discussion history. A mention is not an endorsement.

Step-unrolled Denoising Autoencoders for Text Generation — mentioned by kastnerkyle

Thu, 12 May 2022 03:18:30 GMT

I've had pretty good luck recently with a mix of SUNDAE (https://arxiv.org/abs/2112.06749) and coconet (https://arxiv.org/abs/1903.07227) and/or Music Transformer based internal models recently for modeling very small datasets of polyphonic "midified" music. Research paper hopefully soon to come... Not sure what your pipeline looks like, but those papers might be worth putting on your radar. And as you mention, symbolic music datasets are both surprisingly small, surprisingly low quality, and generally a huge pain to work with. Cool stuff - I like the sax! For anyone unfamiliar with diffusion…

Paper and discussion history. A mention is not an endorsement.