Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning · HackerPapers discussions

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning — mentioned by datastoat

Sun, 15 Jan 2023 21:05:28 GMT

Indeed, there are Bayesian neural networks and there are non-Bayesian neural networks, and I shouldn't have implied that all neural networks are non-Bayesian. I'm just trying to point out that there is a dichotomy between the Bayesian and the non-Bayesian, and that the standard neural network models are non-Bayesian, and that we need Bayesianism (or something like it) to talk about (epistemic) uncertainty. Standard neural networks are non-Bayesian, because they do not treat the neural network parameters as random variables. This includes most of the examples that have been mentioned in this…

Paper and discussion history. A mention is not an endorsement.

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning — mentioned by thiago_lira

Sun, 22 Jul 2018 17:31:10 GMT

I'm thinking about studying these for my MSc. A colleague has given me the idea of applying the Monte Carlo Dropout technique [1] (A interpretation of Dropout as a GP) to the CEAL framework [0]. On the other hand my mentor wanted me to work on a regression problem with a dataset from a cement factory. Have people been using GPs for problems like that? [0] https://arxiv.org/abs/1701.03551 (CEAL) [1] https://arxiv.org/pdf/1506.02142 (MC dropout)

Paper and discussion history. A mention is not an endorsement.

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning — mentioned by elcritch

Fri, 10 Nov 2017 19:51:12 GMT

Interesting blog post! From a 10,000 ft overview is this method utilizing the conditional posterior's to incorporate uncertainty information into the model GAN model? If so how does it differ from using drop-out's to approximate Bayesian behavior ala https://arxiv.org/pdf/1506.02142.pdf ?

Paper and discussion history. A mention is not an endorsement.

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning — mentioned by brianchu

Tue, 10 Jan 2017 00:12:26 GMT

I'm not sure dropout has anything to do with local optima or removing greedily optimal paths, since it is random. The original dropout paper's handwavy justification for dropout is that it prevents co-adaptation. It prevents individual units (nodes/neurons) in the network from relying on specific units in the previous layer firing as well. This is a bad thing because it's fragile (if one unit is off). I say handwavy because this is just intuition; there is not really any proof that this is actually what is happening. Another commonly cited motivation is that dropout is like learning an…

Paper and discussion history. A mention is not an endorsement.

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning — mentioned by MrQuincle

Fri, 04 Dec 2015 21:39:12 GMT

1.) Dropout. Gal and Ghahramani "We show that a neural network with arbitrary depth and non-linearities, with dropout applied before every weight layer, is mathematically equivalent to an approximation to the probabilistic deep Gaussian process" http://arxiv.org/pdf/1506.02142.pdf 2.) Deep. There have been many hierarchical networks in Bayesian statistics. Hierarchical Dirichlet Process, hierarchical beta process, Pitman-Yor process, Gaussian process, nested, dependent, translated, generalized versions of the Chinese Restaurant process, or the Indian Buffet one. The deep part isn't new. 3.)…

Paper and discussion history. A mention is not an endorsement.

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning — mentioned by tabacof

Wed, 02 Sep 2015 20:15:07 GMT

Causality is a bit harder to integrate with current machine learning models as it's hard even with standard probabilistic graphical models. On the other hand, there has been a lot of work integrating deep neural networks with probabilistic models. For example, the variational auto-encoders are a graphical model with Gaussian latent variables whose mean and variance are determined by (deep) neural networks [1]. There has been work exploring the neural network weights as latent variables themselves [2]. Finally, some new developments such as dropout can be interpreted as some form of deep…

Paper and discussion history. A mention is not an endorsement.