Scaling Laws of RoPE-based Extrapolation · HackerPapers discussions

Scaling Laws of RoPE-based Extrapolation — mentioned by omneity

Mon, 14 Apr 2025 19:59:20 GMT

I'm not sure to which extent this opinion is accurately informed. It is well known that nobody trains on 1M token-long content. It wouldn't work anyway as the dependencies are too far fetched and you end up with vanishing gradients. RoPE (Rotary Positional Embeddings, think modulo or periodic arithmetics) scaling is key, whereby the model is trained on 16k tokens long content, and then scaled up to 100k+ [0]. Qwen 1M (who has near perfect recall over the complete window [1]) and Llama 4 10M pushed the limits of this technique, with Qwen reliably training with a much higher RoPE base, and…

Paper and discussion history. A mention is not an endorsement.

Scaling Laws of RoPE-based Extrapolation — mentioned by spmurrayzzz

Sun, 06 Apr 2025 02:46:21 GMT

Not sure what you mean by summary paper, its a pretty dense topic that assumes a fair amount of prior knowledge of the fundamentals. But maybe the Meta blog post may suffice for that? Otherwise yes there are lots of papers on this and related topics, a few dozen in fact. But here are some notable ones, a couple of them are linked in their blog post. RoFormer: Enhanced Transformer with Rotary Position Embedding - https://arxiv.org/abs/2104.09864 Scaling Laws of RoPE-based Extrapolation - https://arxiv.org/abs/2310.05209 The Impact of Positional Encoding on Length Generalization in Transformers…

Paper and discussion history. A mention is not an endorsement.