White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is? · HackerPapers discussions

White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is? — HN submission

Sat, 25 Nov 2023 00:17:36 GMT

White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is? — mentioned by westurner

Fri, 24 Nov 2023 03:24:38 GMT

"White-Box Transformers via Sparse Rate Reduction" (2023) ; https://arxiv.org/abs/2311.13110 https://scholar.google.com/scholar?cites=1536453281127121652... : > Abstract: In this paper, we contend that the objective of representation learning is to compress and transform the distribution of the data, say sets of tokens, towards a mixture of low-dimensional Gaussian distributions supported on incoherent subspaces. The quality of the final representation can be measured by a unified objective function called sparse rate reduction. From this perspective, popular deep networks such as…

Paper and discussion history. A mention is not an endorsement.

White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is? — mentioned by WhitneyLand

Fri, 24 Nov 2023 03:21:32 GMT

I don’t think the statement is annoying to some because they see it as a damning indictment of AI. It’s because it’s not literally true. If it were literally true, a zip file would be ChatGPT, but it is not. The paper was named in an intentionally provocative way. What they really mean is, at the heart of these transformer models, compression is the fundamental principle or goal. This trend in naming papers has evolved over time. It didn’t always used to be this way. Some would say a provocative title is thought-provoking and drives a wider interest. Others would say it’s a research paper and…

Paper and discussion history. A mention is not an endorsement.

White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is? — mentioned by kklisura

Fri, 24 Nov 2023 01:17:32 GMT

Relevant paper: https://arxiv.org/abs/2311.13110

Paper and discussion history. A mention is not an endorsement.

White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is? — HN submission

Fri, 24 Nov 2023 01:16:28 GMT

White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.