HackerPapers

arXiv:1312.6184cs.LG

Do Deep Nets Really Need to be Deep?

Lei Jimmy Ba, Rich Caruana

Abstract

Currently, deep neural networks are the state of the art on problems such as speech recognition and computer vision. In this extended abstract, we show that shallow feed-forward networks can learn the complex functions previously learned by deep nets and achieve accuracies previously only achievable with deep models. Moreover, in some cases the shallow neural nets can learn these deep functions using a total number of parameters similar to the original deep model. We evaluate our method on the TIMIT phoneme recognition task and are able to train shallow fully-connected nets that perform similarly to complex, well-engineered, deep convolutional architectures. Our success in training shallow neural nets to mimic deeper models suggests that there probably exist better algorithms for training shallow feed-forward nets than those currently available.

arXiv comments
final revision coming soon
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.NENeural and Evolutionary Computing

On Hacker News

Linked by 5 identifiable commenters in 5 comments across 5 threads, from Aug 2014 to Jun 2016. Found in 3 HN submissions from Jan 2014 to Oct 2014. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. The truth about deep learning 1 comment
  2. A Brief Overview of Deep Learning 1 comment

    • Deep neural networks are quite powerful because they don't require someone to design the network specific to the task. Large deep nets will generally be able to learn feature representations by themselves, which makes them extremely powerful for tasks where we aren't quite sure…

      nharadaView on HNParent context

  3. Do Deep Nets Really Need to be Deep? [pdf] 1 comment

    • A note -- if you're linking to arXiv, it's better to link to the abstract (http://arxiv.org/abs/1312.6184) rather than directly to the PDF. From the abstract, one can easily click through to the PDF; not so the reverse. And the abstract allows one to do things like see different…

      SniffnoyView on HNParent context

  4. A Revolutionary Technique That Changed Machine Vision 1 comment

    • Have you read Rich Cuarana's work on approximating the behavior of deep nets with single hidden layer networks? http://arxiv.org/pdf/1312.6184.pdf From that paper, it seems like it's just that deep nets are easier to train to find state of the art solutions than shallow nets.…

      robrenaudView on HNParent context

  5. IBM Chip Processes Data Similar to the Way Your Brain Does 1 comment

    • I find it interesting that no group (to my knowledge) has tried something similar to [Do Deep Networks Need to Be Deep?](http://arxiv.org/abs/1312.6184) for ImageNet scale networks. There have been several results which show that the knowledge learned in larger networks can be…

      kastnerkyleView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.