HackerPapers

arXiv:2504.07096cs.CL

OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens

Jiacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, YenSung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Cheng, Karen Farley, Sruthi Sreeram, Taira Anderson, David Albright, Carissa Schoenick, Luca Soldaini, Dirk Groeneveld, Rock Yuren Pang, Pang Wei Koh, Noah A. Smith, Sophie Lebrecht, Yejin Choi, Hannaneh Hajishirzi, Ali Farhadi, Jesse Dodge

Abstract

We present OLMoTrace, the first system that traces the outputs of language models back to their full, multi-trillion-token training data in real time. OLMoTrace finds and shows verbatim matches between segments of language model output and documents in the training text corpora. Powered by an extended version of infini-gram (Liu et al., 2024), our system returns tracing results within a few seconds. OLMoTrace can help users understand the behavior of language models through the lens of their training data. We showcase how it can be used to explore fact checking, hallucination, and the creativity of language models. OLMoTrace is publicly available and fully open-source.

arXiv comments
ACL 2025 demo track
Journal reference
DOI
Subjects
cs.CLComputation and Language

On Hacker News

Linked by 3 identifiable commenters in 4 comments across 3 threads, from Apr 2025 to Sep 2026.

Comments linking this paper, per year.

Discussed in comments3 threads

  1. On the Navier–Stokes Millennium Prize Problem 2 comments

    • Yeah, I'm sure it's completely automated. But that doesn't preclude being able to index and track what the sources of data are. For your data sets, I would hope you are including source information for where the data came frome. And at OpenAI's scale, I would presume they are…

      lambdaView on HNParent context

    • So, one way to prove that the data played no part is to trace and show that it wasn't used in the training process at all. If the data was never used in training, then it couldn't have played a part in the training process. You're right; if the data was used in training, then it…

      lambdaView on HNParent context

  2. Judge said Meta illegally used books to build its AI 1 comment

    • A difficult, but not intractable problem: OLMoTrace claims to be able to trace from output to training data in seconds [1]. Notably, it can do this because OLMo itself was intentionally designed to be open and transparent [2]; it was trained on 4.6 trillion tokens of entirely…

      ijkView on HNParent context

  3. OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens 1 comment

    • Paper: https://arxiv.org/pdf/2504.07096 (original blog title was too long so I used the paper’s title) HF: https://huggingface.co/allenai/OLMo-2-0325-32B-Instruct GH: https://github.com/allenai/infinigram-api

      erlend_shView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.