Starling: An I/O-Efficient Disk-Resident Graph Index Framework for High-Dimensional Vector Similarity Search on Data Segment — mentioned by fzliu

Wed, 17 Jan 2024 17:37:28 GMT

You need different indexing algorithms for different use cases - brute-force indexing, for example, is "SOTA" when it comes to recall (100%). If you have multiple use cases or if you might have domain shift, you'll want a vector database that supports multiple indexes. Here's my 2¢: - If you're just playing around with vector search locally and have a very small dataset, use brute-force search. Don't worry about indexes until later. - If you have plenty of RAM and CPU cores and would like to squeeze out the most performance, use ScaNN or HNSW plus some form of quantization (product…

Paper and discussion history. A mention is not an endorsement.

Starling: An I/O-Efficient Disk-Resident Graph Index Framework for High-Dimensional Vector Similarity Search on Data Segment — mentioned by ccleve

Wed, 17 Jan 2024 17:24:15 GMT

There's a recently published paper on something called Starling. As far as I can tell, it's a more efficient DiskANN. It claims enormous performance benefits. The paper is tedious and near-unreadable, so I didn't finish reading it. Maybe there's something here, maybe not. https://arxiv.org/pdf/2401.02116.pdf

Paper and discussion history. A mention is not an endorsement.

Starling: An I/O-Efficient Disk-Resident Graph Index Framework for High-Dimensional Vector Similarity Search on Data Segment · HackerPapers discussions

Starling: An I/O-Efficient Disk-Resident Graph Index Framework for High-Dimensional Vector Similarity Search on Data Segment — mentioned by fzliu

Starling: An I/O-Efficient Disk-Resident Graph Index Framework for High-Dimensional Vector Similarity Search on Data Segment — mentioned by ccleve