HackerPapers

arXiv:1705.05937cs.PL

Engineering Record And Replay For Deployability: Extended Technical Report

Robert O'Callahan, Chris Jones, Nathan Froyd, Kyle Huey, Albert Noll, Nimrod Partush

Abstract

The ability to record and replay program executions with low overhead enables many applications, such as reverse-execution debugging, debugging of hard-to-reproduce test failures, and "black box" forensic analysis of failures in deployed systems. Existing record-and-replay approaches limit deployability by recording an entire virtual machine (heavyweight), modifying the OS kernel (adding deployment and maintenance costs), requiring pervasive code instrumentation (imposing significant performance and complexity overhead), or modifying compilers and runtime systems (limiting generality). We investigated whether it is possible to build a practical record-and-replay system avoiding all these issues. The answer turns out to be yes - if the CPU and operating system meet certain non-obvious constraints. Fortunately modern Intel CPUs, Linux kernels and user-space frameworks do meet these constraints, although this has only become true recently. With some novel optimizations, our system 'rr' records and replays real-world low-parallelism workloads with low overhead, with an entirely user-space implementation, using stock hardware, compilers, runtimes and operating systems. "rr" forms the basis of an open-source reverse-execution debugger seeing significant use in practice. We present the design and implementation of 'rr', describe its performance on a variety of workloads, and identify constraints on hardware and operating system design required to support our approach.

arXiv comments
This extended technical report is based on our previous arXiv paper arXiv:1610.02144 but contains much deeper technical detail and a "lessons learned" section
Journal reference
DOI
Subjects
cs.PLProgramming Languages

On Hacker News

Linked by 8 identifiable commenters in 11 comments across 6 threads, from Nov 2018 to Mar 2025. Found in 2 HN submissions from May 2017 to Jun 2020. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Show HN: CodeTracer – A new time-traveling debugger implemented in Nim and Rust 1 comment

    • We are currently working on two "backends" where recording works in different ways. For the scripting languages and smart contract/ZK languages, we instrument the interpreters using high level hooking API-s or direct patches, and we produce a trace. For system languages, we…

      alehander42View on HNParent context

  2. rr – record and replay debugger for C/C++ 3 comments

    • Yeah, the reason it only works for these coarser race conditions is that RR only has one thread executing at a time. Chaos mode randomizes the durations of time allotted to each thread before it is preempted. This may be out of date. I believe I read it in the Extended Technical…

      bqmjjx0kacView on HNParent context

    • https://arxiv.org/abs/1705.05937

      khueyView on HNParent context

    • The two approaches are completely different. gdb singlesteps the program and, before each instruction, records the state of registers and memory that will be changed by that instruction --- an undo log. Then you can reverse-execute an instruction by restoring the state from the…

      rocaView on HNParent context

  3. What rr does 1 comment

    • No, that has been done but is much slower. It records all communication with the external world. The full answer is well described in https://arxiv.org/pdf/1705.05937.pdf which I'll quote here: > We identify a boundary around state and computation, record all sources of…

      sfinkView on HNParent context

  4. Time Traveling Linux Bug Reporting: Coming in Julia 1.5 1 comment

    • Some small-ish items: -- Virtualization of rr's required PMU features on Azure and GCP -- Virtualization of Intel's CPUID faulting on AWS and other cloud providers (Linux KVM has it, but it's not enabled in AWS) -- Fix the reliability bug(s) in the retired-conditional-branches…

      rocaView on HNParent context

  5. Debuggers Suck 1 comment

    • By capturing all external inputs to application, it can replay it many times and results will be always identical. https://arxiv.org/pdf/1705.05937

      garaetjjteView on HNParent context

  6. rr: lightweight recording and deterministic debugging 4 comments

    • The basic idea is the same as ReVirt, but there are a lot of different details because rr has to run as a pure user-space Linux application whereas ReVirt was baked into the hypervisor. For rr to run efficiently we have to use Linux APIs in creative ways.…

      rocaView on HNParent context

    • rr has an "extended technical report" at https://arxiv.org/pdf/1705.05937.pdf which explains the principles it uses. In general they are inherently rather complicated -- in order to get to anything resembling useful speed of execution, you need to play clever tricks of one kind…

      pm215View on HNParent context

    • Being slower is not, I think, the major downside. It is that an entire class of errors - race conditions - are basically outside of the scope of the tool. Which is understandable! Race conditions are hard, and when I read about the tool, my first thought was "How are they…

      scott_sView on HNParent context

    • You should read https://arxiv.org/abs/1705.05937 so you don't need to speculate. rr absolutely does guarantee that threads are scheduled the same way during replay as during recording, otherwise it wouldn't work at all on applications like Firefox which use a lot of threads.…

      rocaView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.