HackerPapers

arXiv:1601.06759cs.CV

Pixel Recurrent Neural Networks

Aaron van den Oord, Nal Kalchbrenner, Koray Kavukcuoglu

Abstract

Modeling the distribution of natural images is a landmark problem in unsupervised learning. This task requires an image model that is at once expressive, tractable and scalable. We present a deep neural network that sequentially predicts the pixels in an image along the two spatial dimensions. Our method models the discrete probability of the raw pixel values and encodes the complete set of dependencies in the image. Architectural novelties include fast two-dimensional recurrent layers and an effective use of residual connections in deep recurrent networks. We achieve log-likelihood scores on natural images that are considerably better than the previous state of the art. Our main results also provide benchmarks on the diverse ImageNet dataset. Samples generated from the model appear crisp, varied and globally coherent.

arXiv comments
Journal reference
DOI
Subjects
cs.CVComputer Vision and Pattern Recognition cs.LGMachine Learning cs.NENeural and Evolutionary Computing

On Hacker News

Linked by 7 identifiable commenters in 7 comments across 7 threads, from Jun 2016 to Oct 2025.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. Neural audio codecs: how to get audio into LLMs 1 comment

    • I had a part about this but I took it out: for compression, you could keep the embeddings unquantized and it would still compress quite well, depending on the embedding dimension and the number of quantization levels. But categorical distributions are better for modelling. It's…

      vvolhejnView on HNParent context

  2. Faster convergence for diffusion models 1 comment

    • Diffusion works significantly better for images than sequential pixel generation, there is a good chance it would work better for language as well. Sequential generation used to be state of the art in 2016 and it's basically how current LLMs work: https://arxiv.org/abs/1601.06759

      WithinReasonView on HNParent context

  3. The Reformer – Pushing the limits of language modeling 1 comment

    • Are the results actually good? Table 2 reports 3.40 bits/dim on CIFAR-10, but PixelRNN in 2016 got 3.06 bits/dim (Table 3 in https://arxiv.org/abs/1601.06759). I would like to compare the MNIST results also but I'm having trouble converting between bits/dim and nats in a way…

      blueblimpView on HNParent context

  4. Image GPT 1 comment
  5. JPEG Committee releases a call for evidence for image compression based on AI 1 comment

    • As an example from 29 Feb 2016, for lossless compression you can go from 8 to 3 bits per color according to Table 5 here: https://arxiv.org/abs/1601.06759

      0-_-0View on HNParent context

  6. Tale of Three Probabilistic Families: Discriminative, Descriptive and Generative 1 comment

    • These guys have done what looks like an impressive amount of work... but I was disappointed to see there's no mention of generative language modeling at all and only a very brief mention of key advances in image generation over the past few years. Examples of generative language…

      cs702View on HNParent context

  7. Blade Runner re-encoded using neural networks 1 comment

    • Yes, although the particular technique used here (variational autoencoder, or VAE) doesn't benefit from increasing the code space due to a particular penalty (KL divergence against a fixed N(0, 1) Gaussian prior) during training. So even by making it go to 1000 digits in the…

      kastnerkyleView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.