ColPali: Efficient Document Retrieval with Vision Language Models · HackerPapers discussions

ColPali: Efficient Document Retrieval with Vision Language Models — HN submission

Tue, 11 Mar 2025 15:40:23 GMT

Show HN: DataBridge: Open-Source Multi-Modal Parser and Retriever for LLMs

Hey HN! When testing complex PDFs (with diagrams, formulas, etc.), I found Gemini would straight up hallucinate, while GPT-4o asked me to run a script to parse the file: "It looks like you uploaded a PDF, but I can't directly analyze it unless I extract its contents. Do you want me to process the document and find the relevant data related to IRR (Internal Rate of Return)?"—but that script only handled plain text, missing tables, and images where the actual answer was. We're building DataBridge, an open-source parser and retriever using specialized multi-modal embeddings (ColPali-inspired)…

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

ColPali: Efficient Document Retrieval with Vision Language Models — mentioned by jonathan-adly

Tue, 14 Jan 2025 17:37:06 GMT

The “equivalent” here would be Jina-Clip (architecture-wise), not necessarily performance. The ColPali paper(1) does a good job explaining why you don’t really want to directly use vision embeddings; and how you are much better off optimizing for RAG with a ColPali like setup. Basically, it is not optimized for textual understanding, it works if you are searching for the word bird; and images of birds. But doesn’t work well to pull a document where it’s a paper about birds. 1. https://arxiv.org/abs/2407.01449

Paper and discussion history. A mention is not an endorsement.

ColPali: Efficient Document Retrieval with Vision Language Models — mentioned by jeffchuber

Thu, 10 Oct 2024 22:12:58 GMT

Colbert is great! - Check out https://github.com/AnswerDotAI/RAGatouille by the excellent https://x.com/bclavie Relatedly ColPali (https://arxiv.org/abs/2407.01449) is gaining a ton of steam in the IR community. In general we are skeptical of silver bullets in retrieval.

Paper and discussion history. A mention is not an endorsement.

ColPali: Efficient Document Retrieval with Vision Language Models — mentioned by reerdna

Tue, 30 Jul 2024 10:59:28 GMT

For use in retrieval/RAG, an emerging paradigm is to not parse the PDF at all. By using a multi-modal foundation model, you convert visual representations ("screenshots") of the pdf directly into searchable vector representations. Paper: Efficient Document Retrieval with Vision Language Models - https://arxiv.org/abs/2407.01449 Vespa.ai blog post https://blog.vespa.ai/retrieval-with-vision-language-models-... (my day job)

Paper and discussion history. A mention is not an endorsement.

ColPali: Efficient Document Retrieval with Vision Language Models — HN submission

Mon, 15 Jul 2024 17:09:30 GMT

ColPali: Efficient Document Retrieval with Vision Language Models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

ColPali: Efficient Document Retrieval with Vision Language Models — HN submission

Sun, 14 Jul 2024 21:19:08 GMT

ColPali: Efficient Document Retrieval with Vision Language Models

7 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

ColPali: Efficient Document Retrieval with Vision Language Models — HN submission

Tue, 02 Jul 2024 19:44:08 GMT

ColPali: Efficient Document Retrieval with Vision Language Models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.