Pixel Recurrent Neural Networks · HackerPapers discussions

Pixel Recurrent Neural Networks — mentioned by vvolhejn

Tue, 21 Oct 2025 19:40:01 GMT

I had a part about this but I took it out: for compression, you could keep the embeddings unquantized and it would still compress quite well, depending on the embedding dimension and the number of quantization levels. But categorical distributions are better for modelling. It's a little difficult to explain here without using diagrams. The intuition is that if you try to have a model predict the next embedding and not the next token, you can't model multimodal distributions - you'll end up predicting the mean of the possible continuations and not the mode, which is not what you want. Check…

Paper and discussion history. A mention is not an endorsement.

Pixel Recurrent Neural Networks — mentioned by WithinReason

Mon, 14 Oct 2024 08:17:11 GMT

Diffusion works significantly better for images than sequential pixel generation, there is a good chance it would work better for language as well. Sequential generation used to be state of the art in 2016 and it's basically how current LLMs work: https://arxiv.org/abs/1601.06759

Paper and discussion history. A mention is not an endorsement.

Pixel Recurrent Neural Networks — mentioned by blueblimp

Fri, 03 Jul 2020 07:59:55 GMT

Are the results actually good? Table 2 reports 3.40 bits/dim on CIFAR-10, but PixelRNN in 2016 got 3.06 bits/dim (Table 3 in https://arxiv.org/abs/1601.06759). I would like to compare the MNIST results also but I'm having trouble converting between bits/dim and nats in a way that gives a sensible result. It's a bit annoying that the paper does not compare to previously-reported numbers on these benchmarks.

Paper and discussion history. A mention is not an endorsement.

Pixel Recurrent Neural Networks — mentioned by etaioinshrdlu

Wed, 17 Jun 2020 19:37:40 GMT

This looks like a new iteration on the Pixel-RNN idea: https://arxiv.org/pdf/1601.06759.pdf GPT2 definitely looks better.

Paper and discussion history. A mention is not an endorsement.

Pixel Recurrent Neural Networks — mentioned by 0-_-0

Thu, 20 Feb 2020 13:22:23 GMT

As an example from 29 Feb 2016, for lossless compression you can go from 8 to 3 bits per color according to Table 5 here: https://arxiv.org/abs/1601.06759

Paper and discussion history. A mention is not an endorsement.

Pixel Recurrent Neural Networks — mentioned by cs702

Fri, 12 Oct 2018 01:45:45 GMT

These guys have done what looks like an impressive amount of work... but I was disappointed to see there's no mention of generative language modeling at all and only a very brief mention of key advances in image generation over the past few years. Examples of generative language modeling ignored by this paper (not even mentioned): * Unsupervised Transformer - https://blog.openai.com/language-unsupervised/ * ELMo - https://arxiv.org/abs/1802.05365 * ULMFit - https://arxiv.org/abs/1801.06146 Examples of generative image modeling ignored by this paper (some are mentioned only in passing): * Glow…

Paper and discussion history. A mention is not an endorsement.

Pixel Recurrent Neural Networks — mentioned by kastnerkyle

Fri, 03 Jun 2016 08:43:53 GMT

Yes, although the particular technique used here (variational autoencoder, or VAE) doesn't benefit from increasing the code space due to a particular penalty (KL divergence against a fixed N(0, 1) Gaussian prior) during training. So even by making it go to 1000 digits in the code space, the model will probably still only choose to use 5 or 10 dimensions unless the KL penalty is relaxed, making it closer to a standard autoencoder. An autoencoder with a much larger code space could be an improvement, or some newer works such as Adversarially Learned Inference / Adversarial Feature Learning [0…

Paper and discussion history. A mention is not an endorsement.