Towards Scalable Dataframe Systems · HackerPapers discussions

Towards Scalable Dataframe Systems — HN submission

Sun, 29 Mar 2026 17:42:04 GMT

Towards Scalable Dataframe Systems

6 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Towards Scalable Dataframe Systems — mentioned by lmwnshn

Thu, 11 Sep 2025 13:13:30 GMT

Check out the full version of Towards Scalable Dataframe Systems from VLDB 2020 [0]. They propose an algebra for dataframes and section 4.4's example succinctly describes the pivot operator. [0] https://arxiv.org/abs/2001.00888

Paper and discussion history. A mention is not an endorsement.

Towards Scalable Dataframe Systems — mentioned by devin-petersohn

Mon, 23 Jan 2023 19:44:22 GMT

The core abstraction of Modin is the dataframe algebra, published here: https://arxiv.org/pdf/2001.00888.pdf

Paper and discussion history. A mention is not an endorsement.

Towards Scalable Dataframe Systems — mentioned by devin-petersohn

Tue, 21 Jun 2022 15:09:12 GMT

There are loads of things that are not possible or are very cumbersome to write in SQL, but that pandas and many other dataframe systems allow. Examples are dropping null values based on some threshold, one-hot encoding, covariance, and certain data cleaning operations. These are possible in SQL but very cumbersome to write. There are also things that are outright impossible in a relational database related to metadata manipulation. SQL is super expressive, but I think pandas gets a bad rap. At it's core the data model and language can be more expressive than relational databases (see [1]). I…

Paper and discussion history. A mention is not an endorsement.

Towards Scalable Dataframe Systems — mentioned by devin-petersohn

Tue, 22 Mar 2022 15:17:09 GMT

This blog post is a great way to learn about the theory underlying dataframes without reading a bunch of technical jargon from our academic paper[1]. AMA! [1] https://arxiv.org/pdf/2001.00888

Paper and discussion history. A mention is not an endorsement.

Towards Scalable Dataframe Systems — mentioned by chubot

Tue, 01 Feb 2022 15:42:23 GMT

Very nice idea, although the first PDF I tried wasn't available, and the second one seems to be missing a lot: https://arxiv.org/pdf/2001.00888.pdf (19 pages) https://ar5iv.org/html/2001.00888 (missing content) This is no doubt a hard problem ...

Paper and discussion history. A mention is not an endorsement.

Towards Scalable Dataframe Systems — mentioned by devin-petersohn

Fri, 17 Dec 2021 01:10:05 GMT

This was my PhD focus. We identified a core "dataframe algebra"[1] that encompasses all of pandas (and R/S data.frames): a total of 16 operators that cover all 600+ operators of pandas. What you describe was exactly our aim. It turns out there are a lot of operators that are really easy to support and make fast, and that gets you about 60% or so of the way to supporting all of pandas. Then there are really complex operators that may alter the schema in a way that is undeterminable before the operation is carried out (think a row-wise or column-wise `df.apply`). The flexibility that pandas…

Paper and discussion history. A mention is not an endorsement.

Towards Scalable Dataframe Systems — mentioned by devin-petersohn

Tue, 09 Mar 2021 18:47:25 GMT

I'm one the maintainers of Modin, so I can chime in here. Dataframes are the focus of my PhD thesis, and Modin started as my PhD project. Most of the differences come down to functionality and support. Truthfully, the goals of the projects are quite different so it's a bit of apples-to-oranges. As a part of developing Modin, we identified a low-level algebra and data model that both generalizes and encompasses all of the pandas and R dataframe functionalities. Modin is an implementation of this data model and algebra[1]. Based on our studies, Vaex's architecture can support somewhere in the…

Paper and discussion history. A mention is not an endorsement.