RoFormer: Enhanced Transformer with Rotary Position Embedding · HackerPapers discussions

RoFormer: Enhanced Transformer with Rotary Position Embedding — mentioned by spott

Tue, 05 Aug 2025 18:43:52 GMT

All the Llamas have done it (well, 2 and 3, and I believe 1, I don't know about 4). I think they have a citation for it, though it might just be the RoPE paper (https://arxiv.org/abs/2104.09864). I'm not actually aware of any model that doesn't do positional embeddings on a per-layer basis (excepting BERT and the original transformer paper, and I haven't read the GPT2 paper in a while, so I'm not sure about that one either).

Paper and discussion history. A mention is not an endorsement.

RoFormer: Enhanced Transformer with Rotary Position Embedding — mentioned by spmurrayzzz

Sun, 06 Apr 2025 02:46:21 GMT

Not sure what you mean by summary paper, its a pretty dense topic that assumes a fair amount of prior knowledge of the fundamentals. But maybe the Meta blog post may suffice for that? Otherwise yes there are lots of papers on this and related topics, a few dozen in fact. But here are some notable ones, a couple of them are linked in their blog post. RoFormer: Enhanced Transformer with Rotary Position Embedding - https://arxiv.org/abs/2104.09864 Scaling Laws of RoPE-based Extrapolation - https://arxiv.org/abs/2310.05209 The Impact of Positional Encoding on Length Generalization in Transformers…

Paper and discussion history. A mention is not an endorsement.

RoFormer: Enhanced Transformer with Rotary Position Embedding — mentioned by jimmyl02

Sat, 05 Apr 2025 19:49:18 GMT

the large context windows generally involve RoPE[0] which is a trick that allows the training window to be smaller but expand larger during inference. it seems like they have a new "iRoPE" which might have better performance? [0]https://arxiv.org/pdf/2104.09864

Paper and discussion history. A mention is not an endorsement.

RoFormer: Enhanced Transformer with Rotary Position Embedding — mentioned by EGreg

Thu, 03 Apr 2025 02:14:22 GMT

LLaMa 3 already has RoPE encoding which can handle arbitrarily long contexts (within reason) https://arxiv.org/abs/2104.09864 The difference RoPE makes vs traditional positional encoding is that you just care about relative distances between tokens, and we can attenuate the attention over great distances. Instead of making the model look at every token in the entire sequence all at once (which gets expensive fast), you can break the text into logical chunks—like sentences or paragraphs—and run self-attention within each chunk. That keeps things efficient while still capturing local meaning.…

Paper and discussion history. A mention is not an endorsement.

RoFormer: Enhanced Transformer with Rotary Position Embedding — mentioned by GistNoesis

Wed, 09 Aug 2023 12:09:14 GMT

Cool, there are also additional issues with the RoPEAttention you might want to fix as well : The reference paper for rotary embedding is Roformer https://arxiv.org/pdf/2104.09864v4.pdf First you shouldn't rotate the values, only keys and queries. This is wrong : v_out = (torch.bmm(v.transpose(0,1), self.R[:m, ...])).transpose(0,1) Second you shouldn't apply multihead attention which as additional inner weights that will mess with the rotations you have just done. This is wrong : activations, attn_weights = self.multihead (q_out,k_out,v_out) Instead you should use…

Paper and discussion history. A mention is not an endorsement.

RoFormer: Enhanced Transformer with Rotary Position Embedding — mentioned by mfn

Mon, 10 Apr 2023 01:47:52 GMT

Sinusoidal positional embeddings have always seemed a bit mysterious - even more so since papers don't tend to delve much into the intuition behind them. For example, from Vaswani et al., 2017: > That is, each dimension of the positional encoding corresponds to a sinusoid. The wavelengths form a geometric progression from 2π to 10000 · 2π. We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k, PE(pos+k) can be represented as a linear function of PE(pos). Inspired largely by the RoFormer paper…

Paper and discussion history. A mention is not an endorsement.

RoFormer: Enhanced Transformer with Rotary Position Embedding — mentioned by sebzim4500

Sun, 12 Mar 2023 17:49:37 GMT

It doesn't really use them, it uses something called RoPE which is hardcoded rather than learned and is applied multiplicatively at every layer to both the key and the value. https://arxiv.org/abs/2104.09864

Paper and discussion history. A mention is not an endorsement.