Distinct Elements in Streams: An Algorithm for the (Text) Book · HackerPapers discussions

Distinct Elements in Streams: An Algorithm for the (Text) Book — mentioned by abetusk

Tue, 11 Feb 2025 06:51:52 GMT

Ok, big shout out to monort [0] for the link to the video [1]. This is just a quick overview from a single viewing of the video, but it's called "funnel hashing". The idea is to split into exponentially smaller sub arrays, so the first chunk is n/m, the second is n/(m^2), etc. until you get down to a single element. Call them A0, A1, etc., so |A0| = n/m, |A1| = n/(m^2) etc., k levels in total. Try inserting into A0 c times. If it fails, try inserting into A1 c times. If it fails, go down the "funnel" until you find a free slot. Call \delta the fraction of slots that are empty (I'm unclear if…

Paper and discussion history. A mention is not an endorsement.

Distinct Elements in Streams: An Algorithm for the (Text) Book — mentioned by kuldeepmeel

Sat, 18 May 2024 19:39:06 GMT

We are very grateful for the interest, and I thought I would link to some relevant resources. Paper: https://arxiv.org/pdf/2301.10191 Knuth's note: https://cs.stanford.edu/~knuth/papers/cvm-note.pdf Talk Slides: https://www.cs.toronto.edu/~meel/Slides/meel-distinct.pdf Talk Video: https://www.youtube.com/watch?v=K_ugk7OW0bI The talk also discusses the general settings where our algorithm resolved the open problem of estimation of the union of high dimensional rectangles.

Paper and discussion history. A mention is not an endorsement.

Distinct Elements in Streams: An Algorithm for the (Text) Book — mentioned by jtanderson

Fri, 17 May 2024 13:52:34 GMT

Given the topic of the paper[0], the footnote is especially charming: > the authors decided to forgo the old convention of alphabetical ordering of authors in favor of a randomized ordering, denoted by r⃝. The publicly verifiable record of the randomization is available at https://www.aeaweb.org/journals/policies/random-author-order... [0]: https://arxiv.org/pdf/2301.10191 edit: formatting

Paper and discussion history. A mention is not an endorsement.

Distinct Elements in Streams: An Algorithm for the (Text) Book — mentioned by vitus

Fri, 17 May 2024 13:04:25 GMT

From the paper [0]: > We state the following well-known concentration bound, Chernoff bound, for completeness. Which variant of the Chernoff bound is this? This is almost the (looser variant of the) multiplicative form, but it's not quite right (per the use of 1+delta instead of a single parameter beta). In particular, that bound is only guaranteed to hold for delta >= 0 (not beta = 1 + delta > 0 as asserted in the paper) [0] https://arxiv.org/pdf/2301.10191 [1] https://en.wikipedia.org/wiki/Chernoff_bound#Multiplicative_... edit: to be clear: I'm not sure at all whether this breaks the proof…

Paper and discussion history. A mention is not an endorsement.

Distinct Elements in Streams: An Algorithm for the (Text) Book — HN submission

Fri, 17 May 2024 12:29:51 GMT

Distinct Elements in Streams: An Algorithm for the (Text) Book∗ [pdf]

3 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Distinct Elements in Streams: An Algorithm for the (Text) Book — mentioned by aspenmayer

Fri, 17 May 2024 12:11:46 GMT

Link to abstract: https://arxiv.org/abs/2301.10191

Paper and discussion history. A mention is not an endorsement.

Distinct Elements in Streams: An Algorithm for the (Text) Book — mentioned by usgroup

Fri, 17 May 2024 07:56:33 GMT

I found the paper took about as long to read as the blog post and is more informative: https://arxiv.org/pdf/2301.10191 It is about estimating the cardinality of a set of elements derived from a stream. The algorithm is so simple, you can code it and play with it whilst you read the paper. The authors are explicit about the target audience and purpose for the algorithm: undergraduates and textbooks.

Paper and discussion history. A mention is not an endorsement.

Distinct Elements in Streams: An Algorithm for the (Text) Book — mentioned by seanhunter

Fri, 17 May 2024 07:43:37 GMT

The proof is pretty straightforward and is included in the paper https://arxiv.org/abs/2301.10191

Paper and discussion history. A mention is not an endorsement.

Distinct Elements in Streams: An Algorithm for the (Text) Book — mentioned by svat

Tue, 28 Nov 2023 11:44:34 GMT

Yeah as you say it's just opinion/preference, but re: > source code is the primary source of understanding a concept, and English text is just an accompanying explanation, not the other way round note that most algorithms in TAOCP are only described in English (pseudocode), no (MIX) source code. (So if you want source code as primary you simply won't find it… but then again, a typical algorithm textbook like CLRS also would have only pseudocode). The TAOCP pseudocode is also in a unique style that's not similar to modern programming languages — e.g. here's a note that Knuth wrote just earlier…

Paper and discussion history. A mention is not an endorsement.