Decoding billions of integers per second through vectorization · HackerPapers discussions

Decoding billions of integers per second through vectorization — mentioned by ashtonsix

Sun, 05 Oct 2025 14:21:28 GMT

Thank you so much for attempting a reproduction! (I posted this on Reddit and most commenters didn't even click the link) For the baseline you need SIMDe headers: https://github.com/simd-everywhere/simde/tree/master/simde. These alias x86 intrinsics to ARM intrinsics. The baseline is based on the previous State-of-The-Art (https://arxiv.org/abs/1209.2137) which happens to be x86-based; using SIMDe to compile was the highest-integrity way I could think of to compare with the previous SOTA. Note: M1 chips specifically have notoriously bad small-shift performance, so the benchmark results will…

Paper and discussion history. A mention is not an endorsement.

Decoding billions of integers per second through vectorization — mentioned by ot

Fri, 18 Jun 2021 13:28:27 GMT

> For my purpose, I want to get as high a compression rate as possible, and I need to store just the list of integers Then PFor(Delta) is not the best option, that class of algorithms is intentionally a trade-off between compression rate and decoding speed, as it's meant to store posting lists in inverted indexes. If you want higher compression, a better option would be Binary Interpolative Coding [1], which has some of the highest compression rate among the schemes that are relatively simple to implement. Then you can go down the rabbit hole and start to use entropy compression, complex…

Paper and discussion history. A mention is not an endorsement.

Decoding billions of integers per second through vectorization — mentioned by jeffffff

Thu, 28 Sep 2017 21:05:04 GMT

if you want bit level granularity you can use either SIMD-BP128 or SIMD-FastPFOR from https://arxiv.org/pdf/1209.2137.pdf

Paper and discussion history. A mention is not an endorsement.

Decoding billions of integers per second through vectorization — mentioned by DannyBee

Mon, 27 Feb 2017 06:30:35 GMT

IMHO: Using varint for this is silly, and just wastes decoding time. (Doing because proto/etc did it, when they have wildly different use cases, it a bad idea.) The example they purport to give even proves it: " For example, a 100 TB file in IPFS may have as many as 400 million subobjects, which would mean 400 million hashes. 400,000,000 hashes * (7 - 2) bytes = 2 GB" Right, so uh, that's 0.002% of that 100tb. (IE who cares) (This already appears to be the difference between what it would normally take, at 7 bytes, and what it takes with varint, which is ~17 bits per int) But assume we go…

Paper and discussion history. A mention is not an endorsement.

Decoding billions of integers per second through vectorization — mentioned by gwern

Tue, 09 Jul 2013 04:00:46 GMT

You should probably read the original research paper if you're curious: http://arxiv.org/abs/1209.2137

Paper and discussion history. A mention is not an endorsement.

Decoding billions of integers per second through vectorization — mentioned by cube13

Wed, 12 Sep 2012 16:35:25 GMT

It's buried in the article, but here's the paper: http://arxiv.org/abs/1209.2137 Here's the source: https://github.com/lemire/FastPFor

Paper and discussion history. A mention is not an endorsement.

Decoding billions of integers per second through vectorization — mentioned by kevingadd

Wed, 12 Sep 2012 16:34:13 GMT

Look for the section that says 'We posted our paper online together with our software.' The word 'paper' is a link to http://arxiv.org/abs/1209.2137 and 'our software' is a link to https://github.com/lemire/FastPFor

Paper and discussion history. A mention is not an endorsement.