HackerPapers

arXiv:1506.02142stat.ML

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Yarin Gal, Zoubin Ghahramani

Abstract

Deep learning tools have gained tremendous attention in applied machine learning. However such tools for regression and classification do not capture model uncertainty. In comparison, Bayesian models offer a mathematically grounded framework to reason about model uncertainty, but usually come with a prohibitive computational cost. In this paper we develop a new theoretical framework casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes. A direct result of this theory gives us tools to model uncertainty with dropout NNs -- extracting information from existing models that has been thrown away so far. This mitigates the problem of representing uncertainty in deep learning without sacrificing either computational complexity or test accuracy. We perform an extensive study of the properties of dropout's uncertainty. Various network architectures and non-linearities are assessed on tasks of regression and classification, using MNIST as an example. We show a considerable improvement in predictive log-likelihood and RMSE compared to existing state-of-the-art methods, and finish by using dropout's uncertainty in deep reinforcement learning.

arXiv comments
12 pages, 6 figures; fixed a mistake with standard error and added a new table with updated results (marked "Update [October 2016]"); Published in ICML 2016
Journal reference
DOI
Subjects
stat.MLMachine Learning (Statistics) cs.LGMachine Learning

On Hacker News

Linked by 6 identifiable commenters in 6 comments across 6 threads, from Sep 2015 to Jan 2023.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Bayesian statistics and machine learning: How do they differ? 1 comment

    • Indeed, there are Bayesian neural networks and there are non-Bayesian neural networks, and I shouldn't have implied that all neural networks are non-Bayesian. I'm just trying to point out that there is a dichotomy between the Bayesian and the non-Bayesian, and that the standard…

      datastoatView on HNParent context

  2. Gaussian Processes for Machine Learning (2010) 1 comment

    • I'm thinking about studying these for my MSc. A colleague has given me the idea of applying the Monte Carlo Dropout technique [1] (A interpretation of Dropout as a GP) to the CEAL framework [0]. On the other hand my mentor wanted me to work on a regression problem with a dataset…

      thiago_liraView on HNParent context

  3. Bayesian Generative Adversarial Networks 1 comment

    • Interesting blog post! From a 10,000 ft overview is this method utilizing the conditional posterior's to incorporate uncertainty information into the model GAN model? If so how does it differ from using drop-out's to approximate Bayesian behavior ala…

      elcritchView on HNParent context

  4. Braess’ paradox 1 comment

    • I'm not sure dropout has anything to do with local optima or removing greedily optimal paths, since it is random. The original dropout paper's handwavy justification for dropout is that it prevents co-adaptation. It prevents individual units (nodes/neurons) in the network from…

      brianchuView on HNParent context

  5. Machine learning works spectacularly well, but mathematicians aren’t sure why 1 comment

    • 1.) Dropout. Gal and Ghahramani "We show that a neural network with arbitrary depth and non-linearities, with dropout applied before every weight layer, is mathematically equivalent to an approximation to the probabilistic deep Gaussian process"…

      MrQuincleView on HNParent context

  6. A Brief Introduction to Graphical Models and Bayesian Networks (1998) 1 comment

    • Causality is a bit harder to integrate with current machine learning models as it's hard even with standard probabilistic graphical models. On the other hand, there has been a lot of work integrating deep neural networks with probabilistic models. For example, the variational…

      tabacofView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.