HackerPapers

arXiv:1602.00602cs.PL

Virtual Machine Warmup Blows Hot and Cold

Edd Barrett, Carl Friedrich Bolz-Tereick, Rebecca Killick, Sarah Mount, Laurence Tratt

Abstract

Virtual Machines (VMs) with Just-In-Time (JIT) compilers are traditionally thought to execute programs in two phases: the initial warmup phase determines which parts of a program would most benefit from dynamic compilation, before JIT compiling those parts into machine code; subsequently the program is said to be at a steady state of peak performance. Measurement methodologies almost always discard data collected during the warmup phase such that reported measurements focus entirely on peak performance. We introduce a fully automated statistical approach, based on changepoint analysis, which allows us to determine if a program has reached a steady state and, if so, whether that represents peak performance or not. Using this, we show that even when run in the most controlled of circumstances, small, deterministic, widely studied microbenchmarks often fail to reach a steady state of peak performance on a variety of common VMs. Repeating our experiment on 3 different machines, we found that at most 43.5% of <VM, benchmark> pairs consistently reach a steady state of peak performance.

arXiv comments
40 pages, 11 figures, 9 tables, 3 listings
Journal reference
Object-Oriented Programming, Systems, Languages & Applications. October 2017, Pages 52:1--52:27
DOI
doi:10.1145/3133876
Subjects
cs.PLProgramming Languages

On Hacker News

Linked by 5 identifiable commenters in 10 comments across 10 threads, from Feb 2016 to Dec 2025. Found in 1 HN submission from Feb 2016 to Feb 2016. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments10 threads

  1. We "solved" C10K years ago yet we keep reinventing it (2003) 1 comment

    • JITs are a great magic trick but it's nowhere near guaranteed you'll get good steady performance out of one, especially when the workload is wide not narrow. https://arxiv.org/abs/1602.00602v1

      yencabulatorView on HNParent context

  2. ClickHouse Keeper: A ZooKeeper alternative written in C++ 1 comment
  3. More evidence for problems in VM warmup 1 comment

    • I not only welcome reasonable scepticism, but I do my best to facilitate it. I have accrued sufficient evidence over time of my own fallibility, and idiocy, that I now try to give people the opportunity to spot mistakes so that I might correct them. As a happy bonus, this also…

      ltrattView on HNParent context

  4. Big changes ahead for Deno 1 comment

    • https://arxiv.org/abs/1602.00602 is a good exploration of why this just doesn't work out, and at best tends to lead to weird stuttering performance.

      yencabulatorView on HNParent context

  5. Jitterdämmerung (2015-2021) 1 comment

    • Virtual Machine Warmup Blows Hot and Cold https://arxiv.org/abs/1602.00602 > [...] even when run in the most controlled of circumstances, small, deterministic, widely studied microbenchmarks often fail to reach a steady state of peak performance on a variety of common VMs.

      yencabulatorView on HNParent context

  6. Energy Efficiency Across Programming Languages 1 comment

    • > Are they likely to be expert in all those programming languages? No, but again, I take paper results very seriously. Misleading results are bad for science, period. Here is an example of more rigorous research: https://arxiv.org/pdf/1602.00602.pdf > If Lua wasn't included…

      etieneView on HNParent context

  7. Nectar: A Native JavaScript Compiler Inspired by Crystal Lang and Nim Lang 1 comment

    • However, the JIT compilation process itself is expensive, as is the interpretation and tracing. Since JIT-optimisation happens at run-time, often sub-optimal but fast optimisations are chosen (e.g. linear scan register alloc instead of graph-colouring). Indeed the performance…

      mafribeView on HNParent context

  8. How to speed up the Rust compiler 1 comment

    • > allowed to do its work < It's not like the programs are run with -Xint :-) It's not like a panacea -- https://arxiv.org/abs/1602.00602

      igouyView on HNParent context

  9. CS143: Compilers (2011) 1 comment

    • I agree with you that idealised compilers have a beautiful pipeline structure. Finding this structure Is one of the great achievement of the early computing pioneers. turn it into a laughably simple chain. By this reasoning, any computable problem is trivially simple, because I…

      mafribeView on HNParent context

  10. Akka, Haskell, Erlang, Go and .NET Core compared on 1M threads 1 comment

    • How do you know when warmup is finished? Warmup's a tricky issue, see this brand new paper: http://arxiv.org/abs/1602.00602

      mafribeView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.