HackerPapers

arXiv:2308.05368cs.DB

Building a serverless Data Lakehouse from spare parts

Jacopo Tagliabue, Ciro Greco, Luca Bigon

Early discovery. Linked by one identifiable commenter. This paper is outside the main ranking; a mention is not an endorsement.

Abstract

The recently proposed Data Lakehouse architecture is built on open file formats, performance, and first-class support for data transformation, BI and data science: while the vision stresses the importance of lowering the barrier for data work, existing implementations often struggle to live up to user expectations. At Bauplan, we decided to build a new serverless platform to fulfill the Lakehouse vision. Since building from scratch is a challenge unfit for a startup, we started by re-using (sometimes unconventionally) existing projects, and then investing in improving the areas that would give us the highest marginal gains for the developer experience. In this work, we review user experience, high-level architecture and tooling decisions, and conclude by sharing plans for future development.

arXiv comments
Paper accepted for the Second International Workshop on Composable Data Management Systems (@ VLDB 2023)
Journal reference
DOI
Subjects
cs.DBDatabases cs.DCDistributed, Parallel, and Cluster Computing cs.SESoftware Engineering

On Hacker News

Linked by 1 identifiable commenter in 3 comments across 3 threads, from Nov 2023 to Apr 2026.

Comments linking this paper, per year.

Discussed in comments3 threads

  1. Show HN: Rocky – Rust SQL engine with branches, replay, column lineage 1 comment

    • Cool release. IMO, "Why it's distinctive" is a bit misleading on a few points: certainly the dbt and DX folks can add their POV, but even considering stuff I know / authored ;-), https://arxiv.org/pdf/2308.05368 from 2023 (and following releases) cover branches in a native way…

      jtagliabuetoosoView on HNParent context

  2. A Python-first data lakehouse 1 comment

    • Yeah, terms are confusing sometimes! "Data lakehouse" is weirdly enough a "technical term". The canonical reference is from CIDR https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf, but we have our own version from VLDB https://arxiv.org/pdf/2308.05368

      jtagliabuetoosoView on HNParent context

  3. Summing columns in remote Parquet files using DuckDB 1 comment

    • Quack-reduce author here. I think the most interesting thing about duckDB (for us) is that you can "take out the DB" from duckDB and use the built-in engine over a (stateless, ephemeral) table you got from wherever. We didn't do much map-reduce aside some quick tests (i.e. the…

      jtagliabuetoosoView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.