Contextual Position Encoding: Learning to Count What's Important · HackerPapers discussions

Contextual Position Encoding: Learning to Count What's Important — mentioned by kouteiheika

Thu, 03 Apr 2025 03:56:18 GMT

This is another potential improvement to the transformer architecture from Facebook (the other one that comes to mind is this one from same authors: https://arxiv.org/abs/2405.18719), but note that it comes with a major problem that might not be obvious at first glance: it's just not usable in practice without a ton of work. It modifies the innards of the attention mechanism, so it is incompatible with Flash Attention (or any other optimized attention library), and you do not want to train anything beyond toy models without Flash Attention (the performance hit is just way too big). There's…

Paper and discussion history. A mention is not an endorsement.

Contextual Position Encoding: Learning to Count What's Important — mentioned by smougel

Thu, 24 Oct 2024 13:39:15 GMT

There is a paper about : https://arxiv.org/pdf/2405.18719

Paper and discussion history. A mention is not an endorsement.

Contextual Position Encoding: Learning to Count What's Important — HN submission

Sun, 02 Jun 2024 14:08:11 GMT

Contextual Position Encoding: Learning to Count What's Important

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Contextual Position Encoding: Learning to Count What's Important — HN submission

Thu, 30 May 2024 11:38:10 GMT

Meta FAIR: Contextual Position Encoding, Learning to Count What's Important

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.