Improving neural networks by preventing co-adaptation of feature detectors · HackerPapers discussions

Improving neural networks by preventing co-adaptation of feature detectors — mentioned by esafak

Wed, 26 Jul 2023 00:22:05 GMT

That is one lens to view it through. Co-adaptation reduction is another, and it is an intuitive one: generalization ability is improved if a neuron has to support multiple contexts instead of relying on other neurons to lift the weight, if you pardon the pun. Improving neural networks by preventing co-adaptation of feature detectors https://arxiv.org/abs/1207.0580

Paper and discussion history. A mention is not an endorsement.

Improving neural networks by preventing co-adaptation of feature detectors — mentioned by spacehacker

Thu, 10 Nov 2016 17:09:29 GMT

The dropout regularization method for neural networks was sort of inspired by this observation, i.e. that recombination (combining genes of more than one individual) makes the resulting structures more robust [1]. The idea is basically that it avoids overfitting to the evolutionary context. This has two effects: the fitness cannot change very quickly in response to single mutations, and, conversely, it avoids that genes co-adapt such that the entire genome becomes inflexible at exploring alternatives. To achieve the same without sex, mutations would would have both break up the co-adapted…

Paper and discussion history. A mention is not an endorsement.

Improving neural networks by preventing co-adaptation of feature detectors — mentioned by sja

Wed, 24 Aug 2016 19:17:08 GMT

Good list! I think it's important to note that this article is (intentionally) focused on modern CNN architectures, and not "deep learning" in general. I'd also add in the following "technique" articles: Geoff Hinton et al.'s dropout paper[0] and Loffe and Szegedy's Batch Normalization paper[1]. I don't think there's been enough time for the dust to settle, but I'm excited about the possibilities Stochastic Depth[2] could offer, too. [0]: http://arxiv.org/abs/1207.0580 [1]: http://arxiv.org/abs/1502.03167 [2]: http://arxiv.org/abs/1603.09382

Paper and discussion history. A mention is not an endorsement.

Improving neural networks by preventing co-adaptation of feature detectors — mentioned by eivarv

Tue, 20 Oct 2015 18:35:15 GMT

1) The evidence is that when you for instance visualize the features learned in the layers of a deep convolutional neural net, you'll see that these correspond to layers of abstraction, with each layer's features building upon concepts from the previous layer(s). I found an image [0] (on a site [1]) that illustrates it nicely. 2) Deep learning is really a term that denotes machine learning using models that attempt to abstract the data via multiple layers (popularly in artificial neural networks). Not all deep neural nets are unsupervised, but unsupervised pre-training [2] was an approach…

Paper and discussion history. A mention is not an endorsement.

Improving neural networks by preventing co-adaptation of feature detectors — mentioned by billderose

Sun, 25 May 2014 22:54:49 GMT

Hinton's DropOut [1] and Wan's DropConnect [2] have ameliorated some of the overfitting issues present in traditional NN's. In fact, DropConnect in conjunction with deep learning are responsible for new records being set on classical datasets such as MNIST. [1] http://arxiv.org/pdf/1207.0580.pdf [2] http://cs.nyu.edu/~wanli/dropc/

Paper and discussion history. A mention is not an endorsement.

Improving neural networks by preventing co-adaptation of feature detectors — mentioned by thisisdave

Tue, 25 Feb 2014 00:09:25 GMT

>Neural networks? See The Vanishing Gradient Problem. For what it's worth, Geoff Hinton and others have had a lot of success in the last few years by intentionally injecting noise into their neural network computations. The best-known example is [0], but he's also had some success just adding noise to the calculation of the gradient in good old sigmoid MLPs--even reducing the information content of the gradient below a single bit in some cases. Going back further, stochastic gradient descent and other algorithms have shown for decades that trading accuracy for speed can be very desirable for…

Paper and discussion history. A mention is not an endorsement.

Improving neural networks by preventing co-adaptation of feature detectors — mentioned by psb217

Wed, 26 Dec 2012 18:18:15 GMT

In addition to what bravura said, Geoff Hinton's group at UT has recently introduced a new approach to training neural networks that they call "dropout". You can read about it in their paper (http://arxiv.org/pdf/1207.0580.pdf) or watch Hinton describe it in a recent talk he gave at Google (http://www.youtube.com/watch?v=DleXA5ADG78). Roughly speaking, dropout training provides a strong regularizing effect through a sort of model averaging that is conceptually related to the well-known bagging approach from which random forests derive their power and flexibility. Dropout training has already…

Paper and discussion history. A mention is not an endorsement.

Improving neural networks by preventing co-adaptation of feature detectors — HN submission

Sun, 04 Nov 2012 20:49:21 GMT

Improving neural networks by preventing co-adaptation of feature detectors [pdf]

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.