https://hackerpaper.kaipfister.de/p/2410.23506/Collected HN comments and submissions linking this paper.enMon, 02 Nov 2026 11:16:22 GMThttps://news.ycombinator.com/item?id=45698420urn:hackerpapers:mention:2410.23506:45698420Fri, 24 Oct 2025 19:48:46 GMT<p>There are many many problems with attention. The softmax has issues regarding attention sinks [1]. The softmax also causes sharpness problems [2]. In general this decision boundary being Euclidean dot products isn&#x27;t actually optimal for everything, there are many classes of problem where you want polyhedral cones [3]. Positional embedding are also janky af and so is rope tbh, I think Cannon layers are a more promising alternative for horizontal alignment [4]. I still think there is plenty of room to improve these things. But a lot of focus right now is unfortunately being spent on…</p><p><a href="https://hackerpaper.kaipfister.de/p/2410.23506/">Paper and discussion history</a>. A mention is not an endorsement.</p>cs.LGhttps://news.ycombinator.com/item?id=43202279urn:hackerpapers:submission:2410.23506:43202279Fri, 28 Feb 2025 06:12:32 GMT<p>Belief State Transformer</p><p></p><p>1 point; 0 discussion replies on HN. <a href="https://hackerpaper.kaipfister.de/p/2410.23506/">Paper and source history</a>. A submission is not an endorsement.</p>cs.LG