A Hierarchical Decomposition of Kullback-Leibler Divergence: Disentangling Marginal Mismatches from Statistical Dependencies — mentioned by will-cook-25

Tue, 15 Apr 2025 21:09:03 GMT

Okay, so I've been stuck on this for about a month and it's driving me mad. Everyone keeps talking about KL divergence (comparing two probability distributions P and Q), but it's always just one number—no explanation, no breakdown, nothing. Yeah, they're different, great—but where? Marginals wrong? Interactions wrong? What? I tried to see if you could split KL exactly into two clean parts, no approximations or dodgy assumptions: KL(P || Q) = sum of marginal divergences + total correlation So one part is just how much each variable individually differs (the marginals), and the other part is…

Paper and discussion history. A mention is not an endorsement.

A Hierarchical Decomposition of Kullback-Leibler Divergence: Disentangling Marginal Mismatches from Statistical Dependencies — HN submission

Tue, 15 Apr 2025 21:09:03 GMT

Derived exact decomposition of KL divergence into marginal and interaction parts

2 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.

A Hierarchical Decomposition of Kullback-Leibler Divergence: Disentangling Marginal Mismatches from Statistical Dependencies · HackerPapers discussions

A Hierarchical Decomposition of Kullback-Leibler Divergence: Disentangling Marginal Mismatches from Statistical Dependencies — mentioned by will-cook-25

A Hierarchical Decomposition of Kullback-Leibler Divergence: Disentangling Marginal Mismatches from Statistical Dependencies — HN submission