Semantic Operators: A Declarative Model for Rich, AI-based Data Processing · HackerPapers discussions

Semantic Operators: A Declarative Model for Rich, AI-based Data Processing — mentioned by anishathalye

Mon, 15 Sep 2025 15:42:24 GMT

Yeah, that was just a design choice that I made: I wanted a library that worked with `Iterator`s, felt more lightweight to me / fit my immediate needs better. I'm personally not a huge fan of Pandas DataFrames for certain applications. LOTUS (by Liana Patel et al., folks from Stanford and Berkeley; https://arxiv.org/abs/2407.11418) extends Pandas DataFrames with semantic operators, you could check out their open-source library: https://github.com/lotus-data/lotus Semlib does batch requests, that was one of the primary motivations (I wanted to solve some concrete data processing tasks, started…

Paper and discussion history. A mention is not an endorsement.

Semantic Operators: A Declarative Model for Rich, AI-based Data Processing — mentioned by anishathalye

Mon, 15 Sep 2025 14:55:29 GMT

That was a small self-contained example that fit above the fold in the README (and fwiw even last year’s models like GPT-4o give the right output there). That `sort` is based on pairwise comparisons, which is one of the best ways you can do it in terms of accuracy (Qin et al., 2023: https://arxiv.org/abs/2306.17563). I think there are many real use cases where you might want a semantic sort / semantic data processing in general, when there isn’t a deterministic way to do the task and there is not necessarily a single right answer, and some amount of error (due to LLMs being imperfect) is…

Paper and discussion history. A mention is not an endorsement.