Deep Learning and the Information Bottleneck Principle · HackerPapers discussions

Deep Learning and the Information Bottleneck Principle — mentioned by tbalsam

Fri, 11 Aug 2023 11:01:28 GMT

Gotcha. I'm happy to do the trace as it likely would be fruitful for me. Do you have a link to a specific post you're thinking of? It's likely going to be a Tishby-like (the classic paper from 2015 {with much more work going back into the early aughts, just outside of the NN regime IIRC}: https://arxiv.org/abs/1503.02406) lineage, but I'm happy to look to see if it's novel.

Paper and discussion history. A mention is not an endorsement.

Deep Learning and the Information Bottleneck Principle — mentioned by mathinpens

Fri, 11 May 2018 22:27:14 GMT

if you are interested in this type of analysis are you familiar with the tishby papers? you will at the very least find them enjoyable/mind opening... but some of his key claims are probably contradicted by empirical results. the original paper: https://arxiv.org/abs/1503.02406 a refutation:(with very harsh/unprofessional words by tishby in counter-refutation) https://openreview.net/forum?id=ry_WPG-A-&noteId=ry_WPG-A-

Paper and discussion history. A mention is not an endorsement.

Deep Learning and the Information Bottleneck Principle — mentioned by ssivark

Wed, 30 Nov 2016 15:57:56 GMT

As a complement to all the other comments so far: It is plausible that any goal directed inference must "compress" the raw data into a small number of bits required to make a decision on the goal. For physicists "renormalization group flow" (RG) is nothing but a theory behind that process of "simplifying" a theory enough to make predictions (towards a specific goal). It is also natural that this compression process is iterative. There are many implementations of RG that fall within the framework, each with their own pros and cons, which makes each of them suitable for solving different…

Paper and discussion history. A mention is not an endorsement.

Deep Learning and the Information Bottleneck Principle — mentioned by mturmon

Wed, 30 Nov 2016 07:09:01 GMT

Here's an interesting follow up from a noted researcher (Tali Tishby) with one foot in the stat mech camp, and one in the information theory camp: https://arxiv.org/pdf/1503.02406.pdf

Paper and discussion history. A mention is not an endorsement.

Deep Learning and the Information Bottleneck Principle — mentioned by ssivark

Mon, 28 Nov 2016 07:36:03 GMT

I am working on a project (using Julia) to "learn" an efficient representation of scale invariant many-body quantum states from a microscopic Hamiltonian. The particular structure being learned is supposed to be highly efficient both in representational complexity and computational complexity for predicting physical observables. (Work in progress at https://github.com/sivark/jumera/ ) Also reading/thinking about ideas at the heart of theoretical physics and machine learning. In particular, I've spent the last several months in trying to understand the following two papers in increasing levels…

Paper and discussion history. A mention is not an endorsement.