The Impact of Positional Encoding on Length Generalization in Transformers · HackerPapers discussions

The Impact of Positional Encoding on Length Generalization in Transformers — mentioned by krackers

Wed, 14 Oct 2026 06:28:14 GMT

>drops the positional encodings after pretraining, I thought with NoPE schemes you don't even need it during pretraining https://arxiv.org/abs/2305.19466 https://sebastianraschka.com/llm-architecture-gallery/nope/ Given the causal mask and the way attention works LLMs can implicitly propagate a signal for ordering

Paper and discussion history. A mention is not an endorsement.

The Impact of Positional Encoding on Length Generalization in Transformers — mentioned by krackers

Wed, 30 Jul 2025 01:12:29 GMT

The paper showing that dropping positional encoding entirely is feasible is https://arxiv.org/pdf/2305.19466 . But I was misremembering as to its long context performance, Llama 4 does use NoPE but it's still interleaved with RoPE layers. Just an armchair commenter though, so I may well be wrong. My intuition for NoPE was that the presence of the causal mask provides enough of a signal to implicitly distinguish token position. If you imagine the flow of information in the transformer network, tokens later on in the sequence "absorb" information from the hidden states of previous tokens, so in…

Paper and discussion history. A mention is not an endorsement.

The Impact of Positional Encoding on Length Generalization in Transformers — mentioned by spmurrayzzz

Sun, 06 Apr 2025 02:46:21 GMT

Not sure what you mean by summary paper, its a pretty dense topic that assumes a fair amount of prior knowledge of the fundamentals. But maybe the Meta blog post may suffice for that? Otherwise yes there are lots of papers on this and related topics, a few dozen in fact. But here are some notable ones, a couple of them are linked in their blog post. RoFormer: Enhanced Transformer with Rotary Position Embedding - https://arxiv.org/abs/2104.09864 Scaling Laws of RoPE-based Extrapolation - https://arxiv.org/abs/2310.05209 The Impact of Positional Encoding on Length Generalization in Transformers…

Paper and discussion history. A mention is not an endorsement.

The Impact of Positional Encoding on Length Generalization in Transformers — mentioned by miven

Sat, 05 Apr 2025 19:38:52 GMT

According to [0] it's partly due to a key change they introduced in interleaving layers that use standard RoPE positional encodings and layers using what's called NoPE [1], not encoding positions at all and letting the model to figure those out on its own (this exclusively works because the LLMs are autoregressive, so the model can recognize an input token as being the very first by there not yet being any other tokens to attend to, and recursively deriving the position of the subsequent ones from that base case) [0] https://ai.meta.com/blog/llama-4-multimodal-intelligence/ [1]…

Paper and discussion history. A mention is not an endorsement.

The Impact of Positional Encoding on Length Generalization in Transformers — HN submission

Sun, 18 Aug 2024 17:55:20 GMT

The Impact of Positional Encoding on Length Generalization in Transformers

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

The Impact of Positional Encoding on Length Generalization in Transformers — HN submission

Sat, 03 Jun 2023 05:58:48 GMT

The Impact of Positional Encoding on Length Generalization in Transformers

5 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.