Manu: A Cloud Native Vector Database Management System · HackerPapers discussions

Manu: A Cloud Native Vector Database Management System — mentioned by fzliu

Thu, 30 Mar 2023 23:12:06 GMT

The design of Milvus 2.x follows this paper we published a while back: https://arxiv.org/abs/2206.13843. In short, we used Pulsar to implement the write-ahead log, which provides coordination and a single source of truth across all Milvus components. You're right in that it's a bit heavyweight, so we're working to see how we can make pub/sub and other cluster components lighter and more efficient overall.

Paper and discussion history. A mention is not an endorsement.

Manu: A Cloud Native Vector Database Management System — mentioned by fzliu

Wed, 17 Aug 2022 04:44:56 GMT

On the topic of publishing, these two papers from the Milvus community may be of interest to some folks. SIGMOD'21 - https://www.cs.purdue.edu/homes/csjgwang/pubs/SIGMOD21_Milvu... This paper talks about the vector database vertical (compute core and user-facing API) VLDB'22 - https://arxiv.org/pdf/2206.13843.pdf This paper discusses the development of a cloud-native vector database. Reading through them should help folks understand where the novelty and difficulty in developing an full-fledged vector database comes from. Disclaimer: I'm a member of the Milvus community.

Paper and discussion history. A mention is not an endorsement.

Manu: A Cloud Native Vector Database Management System — mentioned by fzliu

Wed, 17 Aug 2022 00:29:53 GMT

Larger disk space does help effortlessly scale storage in single-node systems, but I agree with you that shared nothing (and/or shared something) is a necessary step for extracting maximum performance on a larger machine. When it comes to distributed architectures, shared nothingness is important as well. The decoupling of storage from compute and stateless from stateful helps minimize resource allocation when it comes to billion-scale vector storage, indexing, and search. Milvus 2.0 implements this type of architecture - here's a link to our VLDB 2022 paper, if you're interested…

Paper and discussion history. A mention is not an endorsement.