Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings — mentioned by spmurrayzzz

Wed, 30 Jul 2025 14:54:13 GMT

Ah didn't realize you were referring to NoPE explicitly. And yea, the intuitions gained from that paper are pretty much what I alluded to above, you don't get away with never modeling the positional data, the question is how you model it effectively and from where do you derive that signal. NoPE never really took off more broadly in modern architecture implementations. We haven't seen anyone successfully reproduce the proposed solution to the long context problem presented in the paper (tuning the scaling factor in the attention softmax). There is a recent paper back in December[1] that…

Paper and discussion history. A mention is not an endorsement.

Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings — HN submission

Sat, 25 Jan 2025 13:41:59 GMT

Position Information Emerges in Causal Transformers Without Positional Encoding

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings · HackerPapers discussions

Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings — mentioned by spmurrayzzz

Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings — HN submission