HackerPapers

arXiv:1404.7828cs.NE

Deep Learning in Neural Networks: An Overview

Juergen Schmidhuber

Abstract

In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarises relevant work, much of it from the previous millennium. Shallow and deep learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.

arXiv comments
88 pages, 888 references
Journal reference
Neural Networks, Vol 61, pp 85-117, Jan 2015
DOI
doi:10.1016/j.neunet.2014.09.003
Subjects
cs.NENeural and Evolutionary Computing cs.LGMachine Learning

On Hacker News

Linked by 10 identifiable commenters in 10 comments across 10 threads, from Dec 2014 to Sep 2024. Found in 3 HN submissions from May 2014 to Dec 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments10 threads

  1. OpenAI to become for-profit company 1 comment

    • > but it would be impractical to make it book-sized Not really: History: https://arxiv.org/abs/2212.11279 (75 pp.) Survey: https://arxiv.org/abs/1404.7828 (88 pp.) Conveniently skim-read over the course of the four weekends on one month.

      jll29View on HNParent context

  2. Show HN: Tiny LLMs – Browser-based private AI models for a wide array of tasks 1 comment

    • Ah good question. I think I have read that statement by some other people. But the limit is kind of arbitrary. And of course, this limit will be higher and higher over time, that's why I put the year. I think the limit should also not be much lower. We already have language…

      albertzeyerView on HNParent context

  3. Deep Learning: Our Miraculous Year 1990-1991 1 comment

    • I mean it's self-promotional yes, but I read this as more of a blog post about advances specifically in his own group. For the Schmidhuberian take on the broader history of deep learning, this other one's the go-to article (though it's much longer)…

      mjnView on HNParent context

  4. Ernst Dickmanns, a German scientist who developed self-driving cars in the 1980s 1 comment

    • It's not widely known, but Alan Turing tinkered with neural nets in 1948, see [1], with his boss at the time calling Turing's work a "schoolboy essay". For a history of the subject, see Section 5.1 of [2]. Note that some parts of [2] are rather controversial. For a historical…

      YorkshireSeasonView on HNParent context

  5. Coding the History of Deep Learning 1 comment

    • If you want a more nuanced research on the history on deep learning in neural networks, here is an excellent historical survey paper: https://arxiv.org/abs/1404.7828

      iluvmylifeView on HNParent context

  6. AI Pioneer Wants to Build the Renaissance Machine of the Future 1 comment

    • For more details, refer to Schmidhuber's own website: http://people.idsia.ch/~juergen/ and his excellent review paper, "Deep Learning in Neural Networks: An Overview": https://arxiv.org/pdf/1404.7828.pdf

      cosmoharriganView on HNParent context

  7. Artificial Intelligence Software Is Booming, But Why Now? 1 comment

    • https://arxiv.org/abs/1404.7828 (Disclaimer: I don't know that much about ML/Deep Learning, the above was recommended by deeplearning.net and is what I'm slogging through ATM. I do have some background in "classical" statistics & optimization theory, and the above article is not…

      jablView on HNParent context

  8. Critique of Paper by “Deep Learning Conspiracy” 1 comment

    • So, for anyone not aware: Schmidhuber is _obsessed_ with this. He wrote an enormous literature review of deep learning [0] basically because he felt that people weren't crediting ideas enough. This isn't a one-off essay, for him, he's been banging this drum for quite a while.…

      speechduhView on HNParent context

  9. Technical Papers Every Programmer Should Read at Least Twice (2011) 1 comment

    • Machine Learning is a fast moving field, and many papers are incremental improvements, so an always up-to-date list will be unlikely. None the less, there are quite a few lists out there if you look. Here are some I've seen posted recently…

      jcrView on HNParent context

  10. Unsupervised Feature Learning and Deep Learning Recommended Readings 1 comment

    • There are some survey papers like this: http://arxiv.org/abs/1404.7828 The point isn't to read every paper it references, but to just get a summary of what the current state of research is. If you are interested in an area then you can go read the papers there.

      HoushalterView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.