HackerPapers

arXiv:1608.04295cs.PF

Robust benchmarking in noisy environments

Jiahao Chen, Jarrett Revels

Abstract

We propose a benchmarking strategy that is robust in the presence of timer error, OS jitter and other environmental fluctuations, and is insensitive to the highly nonideal statistics produced by timing measurements. We construct a model that explains how these strongly nonideal statistics can arise from environmental fluctuations, and also justifies our proposed strategy. We implement this strategy in the BenchmarkTools Julia package, where it is used in production continuous integration (CI) pipelines for developing the Julia language and its ecosystem.

arXiv comments
7 pages, 5 figures, Proceedings of the 20th Annual IEEE High Performance Extreme Computing Conference, 2016
Journal reference
DOI
Subjects
cs.PFPerformance

On Hacker News

Linked by 2 identifiable commenters in 2 comments across 2 threads, from Jan 2020 to Sep 2023. Found in 1 HN submission from Aug 2016 to Aug 2016. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments2 threads

  1. Julia and Mojo Mandelbrot Benchmark 1 comment

    • I think you might not be familiar with the package used to benchmark Julia [1]. It does not fix processes to CPU's, or set kernel governor to performance, and there are fluctuations from usage of the computer. But it does run the function for several seconds and returns the…

      suavesitoView on HNParent context

  2. Measure code execution time accurately in Python 1 comment

    • https://arxiv.org/abs/1608.04295 this paper is the statistical basis of time measuring in Julia. It is, at least a good read when trying to measure the execution of some program seriously at low level. It uses also a predictor of the estimate number of evaluations via a logistic…

      longemen3000View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.