HackerPapers

arXiv:1711.00165stat.ML

Deep Neural Networks as Gaussian Processes

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein

Abstract

It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of evaluating the corresponding GP. Recently, kernel functions which mimic multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network. In this work, we derive the exact equivalence between infinitely wide deep networks and GPs. We further develop a computationally efficient pipeline to compute the covariance function for these GPs. We then use the resulting GPs to perform Bayesian inference for wide deep neural networks on MNIST and CIFAR-10. We observe that trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that the GP uncertainty is strongly correlated with trained network prediction error. We further find that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite-width networks. Finally we connect the performance of these GPs to the recent theory of signal propagation in random neural networks.

arXiv comments
Published version in ICLR 2018. 10 pages + appendix
Journal reference
DOI
Subjects
stat.MLMachine Learning (Statistics) cs.LGMachine Learning

On Hacker News

Linked by 7 identifiable commenters in 7 comments across 6 threads, from Jan 2018 to Feb 2024.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Every model learned by gradient descent is approximately a kernel machine (2020) 2 comments

    • Thanks for the submission. I keep an eye out for connections to potentially cheaper and simpler methods. The new comparison reminds me of this old paper which compared them to Gaussian networks: https://arxiv.org/abs/1711.00165 For causality, I'm also keeping an eye out for ways…

      nickpsecurityView on HNParent context

    • You may enjoy some works on the connections between gaussian processes, neural networks, and linear(in rkhs) models. https://papers.nips.cc/paper_files/paper/2019/hash/39d929972... https://papers.nips.cc/paper_files/paper/1996/hash/ae5e3ce40... https://arxiv.org/abs/1711.00165

      mildmay_riderView on HNParent context

  2. Any Deep ReLU Network Is Shallow 1 comment

    • > How well does a classic deep net architecture like AlexNet or VGG19 classify on a standard dataset such as CIFAR-10 when its “width”— namely, number of channels in convolutional layers, and number of nodes in fully-connected internal layers — is allowed to increase to…

      PartiallyTypedView on HNParent context

  3. Understanding deep learning requires rethinking generalization 1 comment

    • I believe it's this: https://arxiv.org/abs/1711.00165 And then the NTK paper: https://arxiv.org/abs/1806.07572 There is also this which seems to have been produced in parallel, closely related to the first but not exactly the same: https://arxiv.org/abs/1804.11271

      medlyyyView on HNParent context

  4. U.S. government limits exports of artificial intelligence software 1 comment

    • You can do basic math to a multivariate normal distribution to approximate other distributions. In classic feedforward networks, that's achieved by using nonlinear activation functions. Very wide networks turn into Gaussian processes in the limit [0]. There are also approaches…

      yorwbaView on HNParent context

  5. Gaussian Processes for Machine Learning (2010) 1 comment

    • There are some interesting connections between GPs and neural networks. Deep neural networks with random weights concentrate about gaussian processes, so the two may not be so far apart in practice. https://arxiv.org/pdf/1711.00165.pdf

      vladislavView on HNParent context

  6. Using Self-Organizing Maps to Solve the Traveling Salesman Problem 1 comment

    • How exactly did theoretical equivalence result kill SOM's as field of study? There is similar connection between deep fully connected neural networks and Gaussian process. Deep Neural Networks as Gaussian Processes https://arxiv.org/abs/1711.00165

      nabla9View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.