Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders · HackerPapers discussionshttps://hackerpaper.kaipfister.de/p/2407.14435/Collected HN comments and submissions linking this paper.enMon, 02 Nov 2026 11:16:22 GMTJumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders — mentioned by tMcGrathhttps://news.ycombinator.com/item?id=42496647urn:hackerpapers:mention:2407.14435:42496647Mon, 23 Dec 2024 18:45:36 GMT<p>We&#x27;re planning to release the weights once we do a moderation pass. Our SAE was trained on LMSys (you can see this in our accompanying post: https://www.goodfire.ai/papers/mapping-latent-spaces-llama/). No images in training - 3.3 70B is a text-only model so it wouldn&#x27;t have made sense. We&#x27;re exploring other modalities currently though. SAE is a basic ReLU one. This might seem a little backwards, but I&#x27;ve been concerned by some of the high-frequency features in TopK and JumpReLU SAEs and the recent SAE (https://arxiv.org/abs/2407.14435, Figure 14), and the recent SAEBench results…</p><p><a href="https://hackerpaper.kaipfister.de/p/2407.14435/">Paper and discussion history</a>. A mention is not an endorsement.</p>cs.LG