Disentangling Dense Embeddings with Sparse Autoencoders — mentioned by jwuphysics

Fri, 22 Nov 2024 00:13:12 GMT

Incredible, well-documented work -- this is an amazing effort! Two things that caught my eye were (i) your loss curves and (ii) the assessment of dead latents. Our team also studied SAEs -- trained to reconstruct dense embeddings of paper abstracts rather than individual tokens [1]. We observed a power-law scaling of the lower bound of loss curves, even when we varied the sparsity level and the dimensionality of the SAE latent space. We also were able to totally mitigate dead latents with an auxiliary loss, and we saw smooth sinusoidal patterns throughout training iterations. Not sure if…

Paper and discussion history. A mention is not an endorsement.

Disentangling Dense Embeddings with Sparse Autoencoders · HackerPapers discussions

Disentangling Dense Embeddings with Sparse Autoencoders — mentioned by jwuphysics