Out-of-Domain Semantics to the Rescue! Zero-Shot Hybrid Retrieval Models · HackerPapers discussions

Out-of-Domain Semantics to the Rescue! Zero-Shot Hybrid Retrieval Models — mentioned by CShorten

Tue, 07 Mar 2023 16:18:18 GMT

1. Please expand on how you are defining full text search distinctly from sparse search to continue the discussion. My understanding is that full-text search is built on inverted indexing which is the intended meaning behind "sparse" search -- whether that be BM25 sparse or SPLADE sparse or exact keyword / phrase matching, my understanding is that it is the same underlying index algorithm with distinctions in scoring for BM25 / SPLADE. 2. The original argument references "combined system" in the sense of Hybrid Search (BM25 + Dense Vector Search). I don't think this is a fair comparison…

Paper and discussion history. A mention is not an endorsement.

Out-of-Domain Semantics to the Rescue! Zero-Shot Hybrid Retrieval Models — mentioned by trengrj

Tue, 07 Mar 2023 12:15:16 GMT

I work at Weaviate, a few comments on why we implemented hybrid search [1]. - Using two separate systems for traditional BM25 and vector search and keeping them in-sync is pretty difficult from an operations perspective. A combined system is much easier to manage and will have better end-to-end latency. - For combining scores, a linear combination like this article suggests is not recommended and instead rank fusion https://rodgerbenham.github.io/bc17-adcs.pdf (where you care what each method ranks first rather than the absolute score) is used. - The point of adding both search methods is for…

Paper and discussion history. A mention is not an endorsement.