HackerPapers

arXiv:1503.02406cs.LG

Deep Learning and the Information Bottleneck Principle

Naftali Tishby, Noga Zaslavsky

Abstract

Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.

arXiv comments
5 pages, 2 figures, Invited paper to ITW 2015; 2015 IEEE Information Theory Workshop (ITW) (IEEE ITW 2015)
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 4 identifiable commenters in 5 comments across 4 threads, from Nov 2016 to Aug 2023.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Do Machine Learning Models Memorize or Generalize? 1 comment

    • Gotcha. I'm happy to do the trace as it likely would be fruitful for me. Do you have a link to a specific post you're thinking of? It's likely going to be a Tishby-like (the classic paper from 2015 {with much more work going back into the early aughts, just outside of the NN…

      tbalsamView on HNParent context

  2. Why does deep and cheap learning work so well? 1 comment

    • if you are interested in this type of analysis are you familiar with the tishby papers? you will at the very least find them enjoyable/mind opening... but some of his key claims are probably contradicted by empirical results. the original paper: https://arxiv.org/abs/1503.02406…

      mathinpensView on HNParent context

  3. Exact mapping between Variational Renormalization Group and Deep Learning (2014) 2 comments

    • Here's an interesting follow up from a noted researcher (Tali Tishby) with one foot in the stat mech camp, and one in the information theory camp: https://arxiv.org/pdf/1503.02406.pdf

      mturmonView on HNParent context

    • As a complement to all the other comments so far: It is plausible that any goal directed inference must "compress" the raw data into a small number of bits required to make a decision on the goal. For physicists "renormalization group flow" (RG) is nothing but a theory behind…

      ssivarkView on HNParent context

  4. Ask HN: What projects are you currently working on? 1 comment

    • I am working on a project (using Julia) to "learn" an efficient representation of scale invariant many-body quantum states from a microscopic Hamiltonian. The particular structure being learned is supposed to be highly efficient both in representational complexity and…

      ssivarkView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.