On the Theoretical Limitations of Embedding-Based Retrieval · HackerPapers discussions

On the Theoretical Limitations of Embedding-Based Retrieval — mentioned by kingstnap

Fri, 24 Oct 2025 19:48:46 GMT

There are many many problems with attention. The softmax has issues regarding attention sinks [1]. The softmax also causes sharpness problems [2]. In general this decision boundary being Euclidean dot products isn't actually optimal for everything, there are many classes of problem where you want polyhedral cones [3]. Positional embedding are also janky af and so is rope tbh, I think Cannon layers are a more promising alternative for horizontal alignment [4]. I still think there is plenty of room to improve these things. But a lot of focus right now is unfortunately being spent on…

Paper and discussion history. A mention is not an endorsement.

On the Theoretical Limitations of Embedding-Based Retrieval — mentioned by ignobletruth

Tue, 16 Sep 2025 13:52:35 GMT

Question for you experts here. This article uses theory to imply a high bound for semantic capacity in a a vector space. However, this recent article (https://arxiv.org/pdf/2508.21038) empirically characterizes the semantic capacity of embedding vectors, finding inadequate capacity for some use cases. These two articles seem at odds. Can anyone help put these two findings in context and explain their seeming contradictions?

Paper and discussion history. A mention is not an endorsement.

On the Theoretical Limitations of Embedding-Based Retrieval — mentioned by kgeist

Fri, 12 Sep 2025 21:05:03 GMT

There was recently this paper: https://arxiv.org/abs/2508.21038 They show that with 4096-dimensional vectors, accuracy starts to fail at 250 mln documents (fundamental limits of embedding models). For 512-dim, it's just 500k. Is 1 bln vectors practical?

Paper and discussion history. A mention is not an endorsement.

On the Theoretical Limitations of Embedding-Based Retrieval — mentioned by gmueckl

Wed, 03 Sep 2025 19:40:43 GMT

Why not link the paper on arxiv? https://arxiv.org/abs/2508.21038

Paper and discussion history. A mention is not an endorsement.

On the Theoretical Limitations of Embedding-Based Retrieval — HN submission

Wed, 03 Sep 2025 19:09:29 GMT

The Theoretical Limitations of Embedding-Based Retrieval

36 points; 3 discussion replies on HN. Paper and source history. A submission is not an endorsement.

On the Theoretical Limitations of Embedding-Based Retrieval — HN submission

Sun, 31 Aug 2025 03:10:51 GMT

The Theoretical Limitations of Embedding-Based Retrieval

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

On the Theoretical Limitations of Embedding-Based Retrieval — HN submission

Fri, 29 Aug 2025 20:25:34 GMT

The Theoretical Limitations of Embedding-Based Retrieval

151 points; 38 discussion replies on HN. Paper and source history. A submission is not an endorsement.