BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models · HackerPapers discussions

BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — mentioned by svcrunch

Wed, 09 Nov 2022 20:43:57 GMT

This is an area that's dear to me, I'm the cofounder of Vectara and have been working with embedding-based semantic search, aka neural search or neural IR, since 2017. To whether Google uses semantic search, the answer is yes, very heavily [1][2]. Not only that, but they have led, and continue to lead, much of the pioneering research in NLP and neural IR for the past decade [3][4][5]. Technical challenges lie along a few primary dimensions. The first has been search quality, because, while early neural systems like Google Talk to Books [5][6] demonstrated the potential of these techniques…

Paper and discussion history. A mention is not an endorsement.

BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — mentioned by jkb79

Wed, 15 Dec 2021 10:15:15 GMT

Vector models are nothing but representation learning and applying the model out-of-domain usually gives worse results than plain old BM25. See https://arxiv.org/abs/2104.08663 A concrete example is DPR which is a state of the art dense retriever model for wikipedia for question answering, when applying that model on MS Marco passage ranking it performs worse than plain BM25.

Paper and discussion history. A mention is not an endorsement.

BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — mentioned by jkb79

Wed, 15 Dec 2021 08:51:21 GMT

>A much more interesting (and harder) problem is creating good vectors to begin >with. Indeed, this is the hardest problem. Vector search shines when used in-domain using deep representation learning, for example bi-encoders on top of transformer models for text domain. However, these models does not generalize well when used out of domain as the representations changes. Hence, in many cases, simple BM25 beats most of the dense vector models when used in a different domain. See https://arxiv.org/abs/2104.08663

Paper and discussion history. A mention is not an endorsement.