HackerPapers

arXiv:2410.17465cs.DB

Bauplan: zero-copy, scale-up FaaS for data pipelines

Jacopo Tagliabue, Tyler Caraza-Harter, Ciro Greco

Early discovery. Linked by one identifiable commenter. This paper is outside the main ranking; a mention is not an endorsement.

Abstract

Chaining functions for longer workloads is a key use case for FaaS platforms in data applications. However, modern data pipelines differ significantly from typical serverless use cases (e.g., webhooks and microservices); this makes it difficult to retrofit existing pipeline frameworks due to structural constraints. In this paper, we describe these limitations in detail and introduce bauplan, a novel FaaS programming model and serverless runtime designed for data practitioners. bauplan enables users to declaratively define functional Directed Acyclic Graphs (DAGs) along with their runtime environments, which are then efficiently executed on cloud-based workers. We show that bauplan achieves both better performance and a superior developer experience for data workloads by making the trade-off of reducing generality in favor of data-awareness

arXiv comments
Accepted for the 10th International Workshop on Serverless Computing (pre-print)
Journal reference
DOI
Subjects
cs.DBDatabases cs.LGMachine Learning cs.OSOperating Systems

On Hacker News

Linked by 1 identifiable commenter in 3 comments across 2 threads, from Apr 2025 to Jun 2025.

Comments linking this paper, per year.

Discussed in comments2 threads

  1. A Python-first data lakehouse 2 comments

    • Thanks for checking out bauplan (which also supports BYOC, so I guess it is indeed hostable by you in a sense!). We've done quite a lot of open source in our life, at Bauplan (you can check our github), and before (you can check me ;-)), so the comment seems unfair! We…

      jtagliabuetoosoView on HNParent context

    • Spark is technically not Python, even if we support PySpark with the relevant decorator but it's a very niche use case for us. As for all the other Python packages, including proprietary ones, the FaaS model is such that you can declare any package you want in a function as node…

      jtagliabuetoosoView on HNParent context

  2. Bauplan – Git-for-data pipelines on object storage 1 comment

    • Looking to get feedback for a code-first platform for data: instead of custom frameworks, GUIs, notebooks on a chron, bauplan runs SQL / Python functions from your IDE, in the cloud, backed by your object storage. Everything is versioned and composable: time-travel, git-like…

      jtagliabuetoosoView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.