Why does deep and cheap learning work so well? · HackerPapers discussions

Why does deep and cheap learning work so well? — mentioned by cs702

Wed, 26 Apr 2023 12:34:48 GMT

Yes, I agree: In practice, all learning tasks have a common underlying structure because our physical reality can behave only in certain ways, determined by the symmetries in our universe. However, as far as I know, no one has formally specified or proved this commonality in learning tasks. The most persuasive argument I've seen for this view -- our view -- is laid out in this paper by Lin, Tegmark, and Rolnick, written between 2016 and 2017 (although it feels like it was written a century ago!): https://arxiv.org/abs/1608.08225

Paper and discussion history. A mention is not an endorsement.

Why does deep and cheap learning work so well? — mentioned by mikhailfranco

Fri, 23 Oct 2020 02:16:44 GMT

Another interesting explanation for deep learning's success in the physical world: Why does deep and cheap learning work so well? Max Tegmark et al., https://arxiv.org/abs/1608.08225

Paper and discussion history. A mention is not an endorsement.

Why does deep and cheap learning work so well? — HN submission

Thu, 24 Oct 2019 20:50:28 GMT

ML paper – Why does deep and cheap learning work so well?

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Why does deep and cheap learning work so well? — HN submission

Thu, 15 Aug 2019 17:44:17 GMT

Why does deep and cheap learning work so well? (2017)

16 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Why does deep and cheap learning work so well? — mentioned by laclar

Wed, 10 Oct 2018 05:08:23 GMT

As for over-fitting, I am not in my 50s, and I knew what over-fitting was from my undergrad (or highschool?). The idea is very commonplace in engineering. Numerous times, when encountering a "new term" in a different field, such as overfitting in machine learning, I have found myself thinking in overly complicated ways, because I never thought that what I considered so simple could be the central point of discussion. Perhaps the physicists who could not describe what over-fitting was, were in the same shoes. The other unfortunate scenario is that the interviewers expect certain responses that…

Paper and discussion history. A mention is not an endorsement.

Why does deep and cheap learning work so well? — mentioned by bfirsh

Sat, 12 May 2018 10:40:15 GMT

Here's an HTML version of the paper if you're on a phone: https://www.arxiv-vanity.com/papers/1608.08225/

Paper and discussion history. A mention is not an endorsement.

Why does deep and cheap learning work so well? — HN submission

Fri, 11 May 2018 20:05:36 GMT

Why does deep and cheap learning work so well?

434 points; 49 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Why does deep and cheap learning work so well? — mentioned by cs702

Mon, 06 Nov 2017 21:42:03 GMT

As stated by the author, this is necessarily a speculative piece ("assuming that..."), but still worth reading -- as long as you keep that in mind. It makes no sense to criticize it or dismiss it for being speculative, as others have done here! By the way, if you enjoy reading this piece, you might also enjoy really interesting paper coauthored by the same person: "Why does deep and cheap learning work so well?," available at https://arxiv.org/abs/1608.08225 -- it posits that deep learning works well because of the particular structure of the laws of physics in the universe in which we happen…

Paper and discussion history. A mention is not an endorsement.

Why does deep and cheap learning work so well? — mentioned by mikhailfranco

Wed, 23 Aug 2017 11:50:00 GMT

My random stream-of-consciousness reference reactions ... Tegmark & Lin's discussion of how deep learning maps onto the physical world: Why does deep and cheap learning work so well? [1][2] The work of Aerts et al. more than 10 years ago, including how vector space models of human categorization show QM structure [3]. Lucien Hardy's exquisite teasing out of the difference between classical and quantum probability: Quantum Theory From Five Reasonable Axioms [4] It turns out that the innovation, power and strangeness of QM is related to separating physical processes into: 1. Linear continuous…

Paper and discussion history. A mention is not an endorsement.

Why does deep and cheap learning work so well? — mentioned by cs702

Sat, 13 May 2017 12:42:46 GMT

Hmm... it could very well be true. Perhaps deep networks work well because they learn to memorize "the most common patterns of auto-correlation" they see in the training data at different levels of function composition. In fact, we do this explicitly in convolutional layers, which by design learn to represent every input sample as a combination of a finite number of fixed-size square filters. ...and the reason why deep networks might be "generalizing" so well is because not all distributions of natural data are equally likely! In practice, objects with the same or similar labels tend to lie…

Paper and discussion history. A mention is not an endorsement.

Why does deep and cheap learning work so well? — mentioned by colorincorrect

Sun, 09 Apr 2017 15:19:24 GMT

Perhaps this paper provides an explanation? https://arxiv.org/pdf/1608.08225.pdf "The exceptional simplicity of physics-based functions hinges on properties such as symmetry, locality, compositionality and polynomial log-probability, and we explore how these properties translate into exceptionally simple neural networks approximating both natural phenomena such as images and abstract representations thereof such as drawings."

Paper and discussion history. A mention is not an endorsement.

Why does deep and cheap learning work so well? — mentioned by epberry

Mon, 05 Dec 2016 19:28:57 GMT

Part of me says that you are correct on all of these points. However I think you went too far on 3. Experimentation does not always follow theory in science. I would argue that many of the great discoveries of physics in the 20th century followed directly from startling results of experiments conducted at the end of the 19th century (the photoelectric effect for example). I agree that there seems to be an awful lot of experimenting going on and that hardware and large datasets have helped tremendously but there are also many researchers poking and prodding at deep learning theory [1, 2]. So…

Paper and discussion history. A mention is not an endorsement.

Why does deep and cheap learning work so well? — mentioned by BickNowstrom

Mon, 05 Dec 2016 18:52:57 GMT

1. Games are great for studying intelligent agent behavior and intelligence itself. An intelligent agent will find hidden abstractions and patterns in a large dimensionality. Language understanding, vision, decision science, solving long-temporal credit assignment etc. all are present. Russell & Norvig (2003) considered goal-directed behavior as the essence of intelligence. Also see possibly the first papers on AI and their focus on parlor games: https://en.wikipedia.org/wiki/Theory_of_Games_and_Economic_B... & http://www.loebner.net/Prizef/TuringArticle.html 2. This platform will provide a…

Paper and discussion history. A mention is not an endorsement.

Why does deep and cheap learning work so well? — mentioned by cs702

Fri, 21 Oct 2016 13:12:36 GMT

Houshalter: the no-free-lunch theorem assumes that all possible "real world" problems are equally probable. That does not appear to be true in theory (due to the laws of physics, e.g., see https://arxiv.org/abs/1608.08225 ) nor in practice (e.g., in complicated AI-type problems, the objects of interest always seem to lie on/near lower-dimensional manifolds embedded in a high-dimensional input space).

Paper and discussion history. A mention is not an endorsement.

Why does deep and cheap learning work so well? — mentioned by throwaway000002

Tue, 11 Oct 2016 20:50:30 GMT

Thanks for sharing this lecture by Max Tegmark. He comments on the video linking to two papers on arXiv which relate to the material in the lecture: https://arxiv.org/abs/1608.08225 and https://arxiv.org/abs/1606.06737

Paper and discussion history. A mention is not an endorsement.

Why does deep and cheap learning work so well? — mentioned by sgt101

Thu, 01 Sep 2016 21:56:30 GMT

No one really knows, there was a paper by Max Tegmark on HN yesterday with some new ideas and results, but I haven't had time to read it yet. http://arxiv.org/abs/1608.08225 The other responses to your question in this thread are as good I as could give you, but I would feel like I am recounting ideas that may be true but for which there is little evidence.

Paper and discussion history. A mention is not an endorsement.

Why does deep and cheap learning work so well? — HN submission

Wed, 31 Aug 2016 22:28:37 GMT

Why does deep and cheap learning work so well?

38 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.