HackerPapers

arXiv:2301.05062cs.LG

Tracr: Compiled Transformers as a Laboratory for Interpretability

David Lindner, János Kramár, Sebastian Farquhar, Matthew Rahtz, Thomas McGrath, Vladimir Mikulik

Abstract

We show how to "compile" human-readable programs into standard decoder-only transformer models. Our compiler, Tracr, generates models with known structure. This structure can be used to design experiments. For example, we use it to study "superposition" in transformers that execute multi-step algorithms. Additionally, the known structure of Tracr-compiled models can serve as ground-truth for evaluating interpretability methods. Commonly, because the "programs" learned by transformers are unknown it is unclear whether an interpretation succeeded. We demonstrate our approach by implementing and examining programs including computing token frequencies, sorting, and parenthesis checking. We provide an open-source implementation of Tracr at https://github.com/google-deepmind/tracr.

arXiv comments
Presented at NeurIPS 2023 (Spotlight)
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.AIArtificial Intelligence stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 5 identifiable commenters in 5 comments across 5 threads, from Feb 2023 to Nov 2026.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. The 'Astra can read and write weights to improve Qwen' post is misleading 1 comment

    • Here's the work that shows how rule-based logic can be compiled into transformer weights: * Thinking Like Transformers” by Weiss, Goldberg & Yahav (ICML 2021).: https://arxiv.org/abs/2106.06981 * Tracr: Compiled Transformers as a Laboratory for Interpretability…

      syllogismView on HNParent context

  2. Astra can read and write weights to improve Qwen 1 comment

    • I don't think I am. I have cited ROME as an early example of weight editing in large language models, the research has moved on since then. I think you are misinformed or don't know what has been done previously and extrapolating the results in this paper. In exp 1, you can…

      yurimoView on HNParent context

  3. Phi-2: The surprising power of small language models 1 comment

    • There are papers written about this. https://openreview.net/pdf/0c9a0f445d430a4d2bf9a08078c435091... Without a loop - an LLM takes a fixed set of tokens and produces a fixed set of tokens. It has some internal state. So without a loop an LLM cannot be a general purpose computer…

      nojvekView on HNParent context

  4. I made a transformer to predict a simple sequence manually 1 comment

    • A related line of work is "Thinking Like Transformers" [1]. They introduce a primitive programming language, RASP, which is composed of operations capable of being modeled with transformer components, and demonstrate how different programs can be written with it, e.g.…

      teddykokerView on HNParent context

  5. Why Can't Little Chatty Do Math? 1 comment

    • Perhaps GPT-like models are already capable enough to do math, but they need to store what we call mathematical reasoning as one of many distinct processing pathway and tap into it whenever the context is appropriate. Easy to say obviously but there's some promising work in this…

      rsliceView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.