HackerPapers

arXiv:1603.05027cs.CV

Identity Mappings in Deep Residual Networks

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

Abstract

Deep residual networks have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62% error) and CIFAR-100, and a 200-layer ResNet on ImageNet. Code is available at: https://github.com/KaimingHe/resnet-1k-layers

arXiv comments
ECCV 2016 camera-ready
Journal reference
DOI
Subjects
cs.CVComputer Vision and Pattern Recognition cs.LGMachine Learning

On Hacker News

Linked by 4 identifiable commenters in 4 comments across 3 threads, from Nov 2016 to Jul 2026. Found in 1 HN submission from Mar 2016 to Mar 2016. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments3 threads

  1. 30papers.com – Ilya's 30 essential ML papers, in a beginner friendly format 2 comments

    • https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436... https://papers.nips.cc/paper_files/paper/2012/file/c399862d3... https://arxiv.org/pdf/1512.03385 https://arxiv.org/pdf/1511.07122…

      gooobView on HNParent context

    • No need stupid moving texts. CS231n: Convolutional Neural Networks for Visual Recognition - https://cs231n.github.io/ The Unreasonable Effectiveness of Recurrent Neural Networks - https://karpathy.github.io/2015/05/21/rnn-effectiveness/ Understanding LSTM Networks…

      cute_boiView on HNParent context

  2. Ilya Sutskever: “If you learn all of these, you’ll know 90% of what matters” 1 comment

    • a backup just in case https://nlp.seas.harvard.edu/annotated-transformer/ https://scottaaronson.blog/?p=762 https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://arxiv.org/pdf/1409.2329.pdf…

      is_trueView on HNParent context

  3. MXNet – Deep Learning Framework of Choice at AWS 1 comment

    • For deep convnets the vanishing gradient problems can mostly be solved by using residual architectures. See: https://arxiv.org/abs/1603.05027 This is kind of related to solving the vanishing gradient issue in RNNs by using additive recurrent architectures like LSTMs and GRUs.…

      ogriselView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.