HackerPapers

arXiv:2001.00888cs.DB

Towards Scalable Dataframe Systems

Devin Petersohn, Stephen Macke, Doris Xin, William Ma, Doris Lee, Xiangxi Mo, Joseph E. Gonzalez, Joseph M. Hellerstein, Anthony D. Joseph, Aditya Parameswaran

Abstract

Dataframes are a popular abstraction to represent, prepare, and analyze data. Despite the remarkable success of dataframe libraries in Rand Python, dataframes face performance issues even on moderately large datasets. Moreover, there is significant ambiguity regarding dataframe semantics. In this paper we lay out a vision and roadmap for scalable dataframe systems. To demonstrate the potential in this area, we report on our experience building MODIN, a scaled-up implementation of the most widely-used and complex dataframe API today, Python's pandas. With pandas as a reference, we propose a simple data model and algebra for dataframes to ground discussion in the field. Given this foundation, we lay out an agenda of open research opportunities where the distinct features of dataframes will require extending the state of the art in many dimensions of data management. We discuss the implications of signature data-frame features including flexible schemas, ordering, row/column equivalence, and data/metadata fluidity, as well as the piecemeal, trial-and-error-based approach to interacting with dataframes.

arXiv comments
Journal reference
DOI
Subjects
cs.DBDatabases

On Hacker News

Linked by 3 identifiable commenters in 7 comments across 7 threads, from Mar 2021 to Sep 2025. Found in 1 HN submission from Mar 2026 to Mar 2026. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. DOOMscrolling: The Game 1 comment

    • Check out the full version of Towards Scalable Dataframe Systems from VLDB 2020 [0]. They propose an algebra for dataframes and section 4.4's example succinctly describes the pivot operator. [0] https://arxiv.org/abs/2001.00888

      lmwnshnView on HNParent context

  2. Run Pandas on 1TB+ Directly in Your Data Warehouse 1 comment
  3. One-liner for running queries against CSV files with SQLite 1 comment

    • There are loads of things that are not possible or are very cumbersome to write in SQL, but that pandas and many other dataframe systems allow. Examples are dropping null values based on some threshold, one-hot encoding, covariance, and certain data cleaning operations. These…

      devin-petersohnView on HNParent context

  4. We parallelized 600 pandas functions with Modin 1 comment

    • This blog post is a great way to learn about the theory underlying dataframes without reading a bunch of technical jargon from our academic paper[1]. AMA! [1] https://arxiv.org/pdf/2001.00888

      devin-petersohnView on HNParent context

  5. Tell HN: Replace the X with a 5 in arXiv.org to display a paper in HTML 1 comment

    • Very nice idea, although the first PDF I tried wasn't available, and the second one seems to be missing a lot: https://arxiv.org/pdf/2001.00888.pdf (19 pages) https://ar5iv.org/html/2001.00888 (missing content) This is no doubt a hard problem ...

      chubotView on HNParent context

  6. Polars: Fast DataFrame library for Rust and Python 1 comment

    • This was my PhD focus. We identified a core "dataframe algebra"[1] that encompasses all of pandas (and R/S data.frames): a total of 16 operators that cover all 600+ operators of pandas. What you describe was exactly our aim. It turns out there are a lot of operators that are…

      devin-petersohnView on HNParent context

  7. A Hybrid Apache Arrow/Numpy DataFrame with Vaex Version 4.0 1 comment

    • I'm one the maintainers of Modin, so I can chime in here. Dataframes are the focus of my PhD thesis, and Modin started as my PhD project. Most of the differences come down to functionality and support. Truthfully, the goals of the projects are quite different so it's a bit of…

      devin-petersohnView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.