Bauplan: zero-copy, scale-up FaaS for data pipelines · HackerPapers discussions

Bauplan: zero-copy, scale-up FaaS for data pipelines — mentioned by jtagliabuetooso

Sat, 21 Jun 2025 13:02:10 GMT

Spark is technically not Python, even if we support PySpark with the relevant decorator but it's a very niche use case for us. As for all the other Python packages, including proprietary ones, the FaaS model is such that you can declare any package you want in a function as node in the pipeline DAG, and any other in another: every function is fully isolated, and you can even selectively use pandas 1 in one, pandas 2 in another, or update the Python interpreter only in node X. If you're interested in containerization and FaaS abstractions, this is good deep dive…

Paper and discussion history. A mention is not an endorsement.

Bauplan: zero-copy, scale-up FaaS for data pipelines — mentioned by jtagliabuetooso

Sat, 21 Jun 2025 00:55:34 GMT

Thanks for checking out bauplan (which also supports BYOC, so I guess it is indeed hostable by you in a sense!). We've done quite a lot of open source in our life, at Bauplan (you can check our github), and before (you can check me ;-)), so the comment seems unfair! We understand the importance of being clear on how the platform works, and for that we have a long series of blog posts and, if you're so inclined, quite a few peer-reviewed papers in top conferences, ranging from low-level memory optimizations (https://arxiv.org/abs/2504.06151), columnar caching…

Paper and discussion history. A mention is not an endorsement.

Bauplan: zero-copy, scale-up FaaS for data pipelines — mentioned by jtagliabuetooso

Wed, 16 Apr 2025 14:25:31 GMT

Looking to get feedback for a code-first platform for data: instead of custom frameworks, GUIs, notebooks on a chron, bauplan runs SQL / Python functions from your IDE, in the cloud, backed by your object storage. Everything is versioned and composable: time-travel, git-like branches, scriptable meta-logic. Perhaps surprisingly, we decided to co-design the abstractions and the runtime, which allowed novel optimizations at the intersection of FaaS and data - e.g. rebuilding functions can be 15x faster than the corresponding AWS stack (https://arxiv.org/pdf/2410.17465). All capabilities are…

Paper and discussion history. A mention is not an endorsement.