HackerPapers

arXiv:1512.03385cs.CV

Deep Residual Learning for Image Recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

Abstract

Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

arXiv comments
Tech report
Journal reference
DOI
Subjects
cs.CVComputer Vision and Pattern Recognition

On Hacker News

Linked by 22 identifiable commenters in 23 comments across 20 threads, from Dec 2015 to Oct 2026. Found in 3 HN submissions from Dec 2015 to Sep 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments20 threads

  1. We're releasing the weights of Mistral Large 4 1 comment

    • AFAICT there are several things that affect the size of the model: 1. the number of transformer layers -- e.g. each layer adds the ability to provide more information to the residual stream [1], [2]; each layer is `tokens * features * active context` 2. the number of tokens…

      rhdunnView on HNParent context

  2. 30papers.com – Ilya's 30 essential ML papers, in a beginner friendly format 2 comments

    • https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436... https://papers.nips.cc/paper_files/paper/2012/file/c399862d3... https://arxiv.org/pdf/1512.03385 https://arxiv.org/pdf/1511.07122…

      gooobView on HNParent context

    • No need stupid moving texts. CS231n: Convolutional Neural Networks for Visual Recognition - https://cs231n.github.io/ The Unreasonable Effectiveness of Recurrent Neural Networks - https://karpathy.github.io/2015/05/21/rnn-effectiveness/ Understanding LSTM Networks…

      cute_boiView on HNParent context

  3. Gemini Diffusion 1 comment

    • > What contributed more towards success in my opinion are "shortcut connections" through layers which enable more influence on early layers during learning. For those who don't know, that is the idea behind ResNet (He et al., Deep Residual Learning for Image Recognition…

      grumbelbart2View on HNParent context

  4. Ilya Sutskever: “If you learn all of these, you’ll know 90% of what matters” 1 comment

    • a backup just in case https://nlp.seas.harvard.edu/annotated-transformer/ https://scottaaronson.blog/?p=762 https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://arxiv.org/pdf/1409.2329.pdf…

      is_trueView on HNParent context

  5. Simplifying Transformer Blocks 1 comment

    • > The field of ML research is moving so fast that people don’t even take time anymore to explain the design decisions behind their architectures. Has it ever? I've read a lot of "old" papers and idk if there's ever been a strong theoretical framework. I'd say it is strongest…

      godelskiView on HNParent context

  6. Neural networks in the 1990s 1 comment

    • We were missing two architecture patterns that were needed to get deeper nets to converge: residual nets [1] which solved gradient propagation, and batch normalization [2] which solved initialization. [1] Residual nets (2015): https://arxiv.org/abs/1512.03385 [2] Batch…

      radqView on HNParent context

  7. Do wide and deep networks learn the same things? 2 comments

    • Haven't read thoroughly but it seems they are investigating ResNet models [0] trained for image classification. > We apply CKA to a family of ResNets of varying depths and widths, trained on common benchmark datasets (CIFAR-10, CIFAR-100 and ImageNet) [0]…

      andersourceView on HNParent context

    • For more context to people, we have the universal approximation theorem for neural nets that basically says if a network is wide enough it can approximate anything (with at least 2 layers). So a lot of stuff was really wide. Then VGG[0] came out and showed that deep networks…

      godelskiView on HNParent context

  8. Predictive coding has been unified with backpropagation 1 comment

    • 1) How similar is this to creating a resnet? [1]. What are some key differences and similarities? 2) Has a CNN version of this been implemented in PyTorch? [1] https://arxiv.org/pdf/1512.03385.pdf (Figure 2)

      laplacesdemon48View on HNParent context

  9. A Primer in BERTology: What We Know About How Bert Works 1 comment

    • Intelligent trial and error. 1. Transformers are an extension of the attention mechanism, a well known LSTM extension that worked well for machine translation (2014, https://arxiv.org/abs/1409.0473). A transformer model is essentially building a multi headed attention module…

      alquemistView on HNParent context

  10. China hires over 100 TSMC engineers in push for chip leadership 1 comment

    • ResNet https://arxiv.org/abs/1512.03385 came out of Microsoft Research Asia. I think that's a pretty good thing they produced. Evidently they managed to hire the top talent behind that work at some point. Looks like Kaiming He works at Facebook now, though. http://kaiminghe.com/

      yorwbaView on HNParent context

  11. Neural Ordinary Differential Equations 1 comment
  12. Ask HN: Can you recommend any ML papers or blogs for visual art detection? 1 comment

    • Conceptually, if you’re trying to classify artwork by a category like artist or period, the problem is just ordinary image classification and won’t require anything unique so I’d recommend just reading the current SOA papers for that task. The foundation for many of the current…

      Q6T46nT668w6i3mView on HNParent context

  13. China is catching up to the USA, while Japan is being left behind 1 comment

    • https://arxiv.org/abs/1512.03385 Deep Residual Learning for Image Recognition Citation 4000+. Milestone of image recognition. https://arxiv.org/abs/1502.01852 Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. Also known as PreLU…

      tanilamaView on HNParent context

  14. Neural Network Architectures 1 comment

    • In natural language processing tasks you see a lot of non-CNN architectures. These usually are designed to be able to deal with sequential data, so some kind of "memory" is needed. Sometimes you see this combined with a CNN. There has been a few question answering systems that…

      nlView on HNParent context

  15. What is the difference between deep learning and usual machine learning? 1 comment

    • Deep Residual Learning for Image Recognition https://arxiv.org/abs/1512.03385 Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We…

      Kip9000View on HNParent context

  16. Tinker with a Neural Network in Your Browser 1 comment

    • I'm not qualified to answer this, but I will anyway. To "operate" neural networks (as opposed to writing a framework for them), you need to know the building blocks. There are basic blocks like fully connected layers, convolutions, and nonlinear activations. Beyond those, there…

      walrusView on HNParent context

  17. Deep Networks with Stochastic Depth 1 comment

    • Having not read the paper, something I find unclear: is it only the feedback path that is skipped, or is the feedforward path also skipped? The abstract mentions replacing the layer with an identity function. I'm not sure how this would work, wouldn't it change the result (i.e…

      radarsat1View on HNParent context

  18. Leaf: Machine learning framework in Rust 1 comment

    • Yes. The general task of looking at arbitrary images and labeling objects (from a set of known categories) in those images is called "detection." In fact the problem you described is easier, because there's only one category (people), and the system only needs to provide a…

      argonautView on HNParent context

  19. Ask HN: Papers you read in 2015? 2 comments

    • Deep Residual Learning for Image Recognition http://arxiv.org/abs/1512.03385 Batch Normalization http://jmlr.org/proceedings/papers/v37/ioffe15.pdf Deep Neural Decision Forests http://research.microsoft.com/pubs/255952/ICCV15_DeepNDF_mai... Spatial Transformer Networks…

      racoonearView on HNParent context

    • I've been trying to read a paper a day since midsummer. These are a few of the interesting, for me personally, since then: Generating Sequences With Recurrent Neural Networks - http://arxiv.org/abs/1308.0850 Older one, but important to understand deeply since other recent ideas…

      nrmnView on HNParent context

  20. Microsoft Research wins image recognition competition 1 comment

    • It's mentioned in the article, but here's a direct link to the paper on arXiv: http://arxiv.org/abs/1512.03385 And direct link to the PDF: http://arxiv.org/pdf/1512.03385v1.pdf

      dplarsonView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.