Robust benchmarking in noisy environments · HackerPapers discussions

Robust benchmarking in noisy environments — mentioned by suavesito

Sat, 09 Sep 2023 10:16:13 GMT

I think you might not be familiar with the package used to benchmark Julia [1]. It does not fix processes to CPU's, or set kernel governor to performance, and there are fluctuations from usage of the computer. But it does run the function for several seconds and returns the distribution of the runs (the little graphics underneath the benchmarks). It calculates standard deviation and if some runs are too small (sub-nano seconds) it emits warnings saying the results might be caused by inlining and constant propagation. The differences in runtimes you refer to are from use of different machines…

Paper and discussion history. A mention is not an endorsement.

Robust benchmarking in noisy environments — mentioned by longemen3000

Sat, 18 Jan 2020 19:52:31 GMT

https://arxiv.org/abs/1608.04295 this paper is the statistical basis of time measuring in Julia. It is, at least a good read when trying to measure the execution of some program seriously at low level. It uses also a predictor of the estimate number of evaluations via a logistic function

Paper and discussion history. A mention is not an endorsement.

Robust benchmarking in noisy environments — HN submission

Tue, 16 Aug 2016 03:32:43 GMT

Robust benchmarking in noisy environments

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.