On Linear Identifiability of Learned Representations — mentioned by tsurba

Tue, 09 Dec 2025 05:36:00 GMT

Many discriminative models converge to same representation space up to a linear transformation. Makes sense that a linear transformation (like PCA) would be able to undo that transformation. https://arxiv.org/abs/2007.00810 Without properly reading the linked article, if thats all this is, not a particularly new result. Nevertheless this direction of proofs is imo at the core of understanding neural nets.

Paper and discussion history. A mention is not an endorsement.

On Linear Identifiability of Learned Representations — mentioned by tsurba

Tue, 09 Dec 2025 05:25:37 GMT

Edit: actually this paper is the canonical reference (?): https://arxiv.org/abs/2007.00810 models converge to same space up to a linear transformation. Makes sense that a linear transformation (like PCA) would be able to undo that transformation. You can show for example that siamese encoders for time-series, with MSE loss on similarity, without a decoder, will converge to the the same latent space up to orthogonal transformations (as MSE is kinda like gaussian prior which doesn’t distinguish between different rotations). Similarly I would expect that transformers trained on the same loss…

Paper and discussion history. A mention is not an endorsement.

On Linear Identifiability of Learned Representations · HackerPapers discussions

On Linear Identifiability of Learned Representations — mentioned by tsurba

On Linear Identifiability of Learned Representations — mentioned by tsurba