Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie · HackerPapers discussions

Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie — mentioned by jtagliabuetooso

Sat, 21 Jun 2025 13:15:12 GMT

Thanks for the comment: your frustration is the default in the industry, and it's part of the reasons why Bauplan was built. "but it always takes forever and I'm never even able to articulate why." -> there are way more factors at play than DoFeatures unfortunately, see for example Table 1 (https://arxiv.org/pdf/2404.13682). Even knowing which data people have developed on is hard, which is why bauplan has git-for-data semantics built in: everyone works on production data, but safely and reliably, to avoid data skews. Each computer is different, which is why bauplan adopt FaaS with isolated…

Paper and discussion history. A mention is not an endorsement.

Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie — mentioned by jtagliabuetooso

Sat, 21 Jun 2025 13:09:28 GMT

Importantly, kedro does not run things for you, resulting in a suboptimal experience because the runtime and dsl are separated: in particular, it does not solve the problem of having K different systems with scattered logs and not easy to integrate APIs. If you want to dive deeper in one line reproducibility, you can chek our SIGMOD24 paper: https://arxiv.org/pdf/2404.13682. Let us know what you think!

Paper and discussion history. A mention is not an endorsement.

Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie — mentioned by jtagliabuetooso

Sat, 21 Jun 2025 00:55:34 GMT

Thanks for checking out bauplan (which also supports BYOC, so I guess it is indeed hostable by you in a sense!). We've done quite a lot of open source in our life, at Bauplan (you can check our github), and before (you can check me ;-)), so the comment seems unfair! We understand the importance of being clear on how the platform works, and for that we have a long series of blog posts and, if you're so inclined, quite a few peer-reviewed papers in top conferences, ranging from low-level memory optimizations (https://arxiv.org/abs/2504.06151), columnar caching…

Paper and discussion history. A mention is not an endorsement.