HackerPapers

arXiv:1702.01284cs.DS

New cardinality estimation algorithms for HyperLogLog sketches

Otmar Ertl

Abstract

This paper presents new methods to estimate the cardinalities of data sets recorded by HyperLogLog sketches. A theoretically motivated extension to the original estimator is presented that eliminates the bias for small and large cardinalities. Based on the maximum likelihood principle a second unbiased method is derived together with a robust and efficient numerical algorithm to calculate the estimate. The maximum likelihood approach can also be applied to more than a single HyperLogLog sketch. In particular, it is shown that it gives more precise cardinality estimates for union, intersection, or relative complements of two sets that are both represented by HyperLogLog sketches compared to the conventional technique using the inclusion-exclusion principle. All the new methods are demonstrated and verified by extensive simulations.

arXiv comments
Corrected typos
Journal reference
DOI
Subjects
cs.DSData Structures and Algorithms

On Hacker News

Linked by 3 identifiable commenters in 4 comments across 4 threads, from Jun 2017 to Mar 2021.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. WyHLL: The most accurate 3-bits HyperLogLog 1 comment
  2. HyperMinHash: Bringing intersections to HyperLogLog 1 comment

    • In my benchmarks, HyperLogLogs unilaterally outperform MinHash* by a constant factor, while Bloom Filters of the same size become more accurate than hyperloglogs in the 100-500kB range (though this number increases with the cardinalities of sets being compared). The biggest…

      stochastic_monkView on HNParent context

  3. Show HN: HyperLogLog intersections 1 comment

    • Great work! I haven't messed around with the count-min augmentation for HLLs, but I've had a lot of success in practice. I've experimented with using HLL sketches for set operations for genome comparisons. The great thing about these operations is that the cost of operations…

      stochastic_monkView on HNParent context

  4. Show HN: An improved version of HyperLogLog 1 comment

    • How does the Xiao et al. paper (which introduces TailCut, "Better with Fewer Bits: Improving the Performance of Cardinality Estimation of Large Data Streams", <http://cse.seu.edu.cn/PersonalPage/csqjxiao/csqjxiao_files/p...) compare with Ertl's "New cardinality estimation…

      jbappleView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.