HackerPapers

arXiv:1207.0580cs.NE

Improving neural networks by preventing co-adaptation of feature detectors

Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov

Abstract

When a large feedforward neural network is trained on a small training set, it typically performs poorly on held-out test data. This "overfitting" is greatly reduced by randomly omitting half of the feature detectors on each training case. This prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors. Instead, each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate. Random "dropout" gives big improvements on many benchmark tasks and sets new records for speech and object recognition.

arXiv comments
Journal reference
DOI
Subjects
cs.NENeural and Evolutionary Computing cs.CVComputer Vision and Pattern Recognition cs.LGMachine Learning

On Hacker News

Linked by 7 identifiable commenters in 7 comments across 7 threads, from Dec 2012 to Jul 2023. Found in 1 HN submission from Nov 2012 to Nov 2012. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. What we know about LLMs 1 comment

    • That is one lens to view it through. Co-adaptation reduction is another, and it is an intuitive one: generalization ability is improved if a neuron has to support multiple contexts instead of relying on other neurons to lift the weight, if you pardon the pun. Improving neural…

      esafakView on HNParent context

  2. Learning about evolution from species who thrive without sexual reproduction 1 comment

    • The dropout regularization method for neural networks was sort of inspired by this observation, i.e. that recombination (combining genes of more than one individual) makes the resulting structures more robust [1]. The idea is basically that it avoids overfitting to the…

      spacehackerView on HNParent context

  3. Some of the most important deep learning papers 1 comment

    • Good list! I think it's important to note that this article is (intentionally) focused on modern CNN architectures, and not "deep learning" in general. I'd also add in the following "technique" articles: Geoff Hinton et al.'s dropout paper[0] and Loffe and Szegedy's Batch…

      sjaView on HNParent context

  4. Theoretical Motivations for Deep Learning 1 comment

    • 1) The evidence is that when you for instance visualize the features learned in the layers of a deep convolutional neural net, you'll see that these correspond to layers of abstraction, with each layer's features building upon concepts from the previous layer(s). I found an…

      eivarvView on HNParent context

  5. What does a neural network actually do? 1 comment

    • Hinton's DropOut [1] and Wan's DropConnect [2] have ameliorated some of the overfitting issues present in traditional NN's. In fact, DropConnect in conjunction with deep learning are responsible for new records being set on classical datasets such as MNIST. [1]…

      billderoseView on HNParent context

  6. Computing 10,000X more efficiently (2010) [pdf] 1 comment

    • >Neural networks? See The Vanishing Gradient Problem. For what it's worth, Geoff Hinton and others have had a lot of success in the last few years by intentionally injecting noise into their neural network computations. The best-known example is [0], but he's also had some…

      thisisdaveView on HNParent context

  7. Why use SVM 1 comment

    • In addition to what bravura said, Geoff Hinton's group at UT has recently introduced a new approach to training neural networks that they call "dropout". You can read about it in their paper (http://arxiv.org/pdf/1207.0580.pdf) or watch Hinton describe it in a recent talk he…

      psb217View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.