Scalable-Softmax Is Superior for Attention · HackerPapers discussions

Scalable-Softmax Is Superior for Attention — mentioned by spmurrayzzz

Sun, 06 Apr 2025 02:46:21 GMT

Not sure what you mean by summary paper, its a pretty dense topic that assumes a fair amount of prior knowledge of the fundamentals. But maybe the Meta blog post may suffice for that? Otherwise yes there are lots of papers on this and related topics, a few dozen in fact. But here are some notable ones, a couple of them are linked in their blog post. RoFormer: Enhanced Transformer with Rotary Position Embedding - https://arxiv.org/abs/2104.09864 Scaling Laws of RoPE-based Extrapolation - https://arxiv.org/abs/2310.05209 The Impact of Positional Encoding on Length Generalization in Transformers…

Paper and discussion history. A mention is not an endorsement.

Scalable-Softmax Is Superior for Attention — HN submission

Tue, 04 Feb 2025 13:57:32 GMT

Scalable-Softmax Is Superior for Attention

2 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Scalable-Softmax Is Superior for Attention — mentioned by TheGradfather

Mon, 03 Feb 2025 17:47:50 GMT

The OpenAI Deep Research graph showing tool calls vs pass rate reveals something fascinating about how these models handle increasing amounts of information. The relationship follows a logistic curve that plateaus around 16% pass rate, even as we allow more tool calls. This plateau behavior reflects something deeper about our current approach to AI. We've built transformer architectures partly inspired by simplified observations of human cognition - particularly how our brains use attention mechanisms to filter and process information. And like human attention, these models have inherent…

Paper and discussion history. A mention is not an endorsement.