HackerPapers

arXiv:1410.3831stat.ML

An exact mapping between the Variational Renormalization Group and Deep Learning

Pankaj Mehta, David J. Schwab

Abstract

Deep learning is a broad set of techniques that uses multiple layers of representation to automatically learn relevant features directly from structured data. Recently, such techniques have yielded record-breaking results on a diverse set of difficult machine learning tasks in computer vision, speech recognition, and natural language processing. Despite the enormous success of deep learning, relatively little is understood theoretically about why these techniques are so successful at feature learning and compression. Here, we show that deep learning is intimately related to one of the most important and successful techniques in theoretical physics, the renormalization group (RG). RG is an iterative coarse-graining scheme that allows for the extraction of relevant features (i.e. operators) as a physical system is examined at different length scales. We construct an exact mapping from the variational renormalization group, first introduced by Kadanoff, and deep learning architectures based on Restricted Boltzmann Machines (RBMs). We illustrate these ideas using the nearest-neighbor Ising Model in one and two-dimensions. Our results suggests that deep learning algorithms may be employing a generalized RG-like scheme to learn relevant features from data.

arXiv comments
8 pages, 3 figures
Journal reference
DOI
Subjects
stat.MLMachine Learning (Statistics) cond-mat.stat-mechStatistical Mechanics cs.LGMachine Learning cs.NENeural and Evolutionary Computing

On Hacker News

Linked by 9 identifiable commenters in 10 comments across 9 threads, from Dec 2014 to May 2025. Found in 4 HN submissions from Dec 2014 to Nov 2016. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments9 threads

  1. Deep Learning Is Applied Topology 1 comment

    • Thanks for this additional link, which really underscores for me at least how you're right about patterns in circuits being a better abstraction layer for capturing interesting patterns than topological manifolds. I wasn't familiar with the term "equivariance" but I "woke up" to…

      riemannzetaView on HNParent context

  2. How Does GPT-4o Encode Images? 1 comment

    • Love this curious and open-minded exploration of how this stuff works. The pyramid strategy loosely tracks with renormalization group theory, which has been formally studied for years as a method of interpreting machine learning models: https://arxiv.org/abs/1410.3831 I love the…

      riemannzetaView on HNParent context

  3. Discrete Differential Geometry: An Applied Introduction [pdf] 1 comment

    • it's quite plausible that at the fringe/edge there is research being done that involves differential geometry - hell there's this paper https://arxiv.org/pdf/1410.3831v1.pdf that's occupies very rarefied air, but that's not what the person to whom i'm responding to claims (that…

      mleventalView on HNParent context

  4. Deep Learning and Quantum Entanglement [pdf] 1 comment

    • Thank you for posting this. Similar thoughts have been percolating in my stream-of-consciousness for a while, especially after coming across an earlier version of Tegmark & Lin's paper some months ago. My take is that deep neural nets work so well in practice because not all…

      cs702View on HNParent context

  5. OpenAI Universe 1 comment

    • Part of me says that you are correct on all of these points. However I think you went too far on 3. Experimentation does not always follow theory in science. I would argue that many of the great discoveries of physics in the 20th century followed directly from startling results…

      epberryView on HNParent context

  6. Ask HN: What projects are you currently working on? 1 comment

    • I am working on a project (using Julia) to "learn" an efficient representation of scale invariant many-body quantum states from a microscopic Hamiltonian. The particular structure being learned is supposed to be highly efficient both in representational complexity and…

      ssivarkView on HNParent context

  7. Link Between Deep Neural Networks and the Nature of the Universe 1 comment

    • You should watch Max Tegmark's talk "Connections between Physics and Deep Learning" if this interests you [1]. Additionally, a paper that has everyone excited about deep connections between the mathematical analysis of physical systems and the hierarchical feature learning…

      adityabView on HNParent context

  8. Strike for physics funding 1 comment

    • To establish in a semi-rigorous way the (btw obvious) relationship between coarsening in physics and in stacked neural networks, as was done long after the latter became a thing [1][2], is at the same level as observing the analogy between Black-Scholes and diffusion. It's cute…

      T-AView on HNParent context

  9. Renormalization: A Common Logic to Seeing Cats and Cosmos 2 comments

    • This is the paper in question: http://arxiv.org/abs/1410.3831

      digital55View on HNParent context

    • This could be an important result, since there's a lot of theory and expertise regarding renormalization group methods, but I think it's too early to tell. The write-up is a bit misleading, though: the model in their preprint[1] is about stacks of restricted Boltzmann machines…

      thisisdaveView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.