Stream VByte: Faster Byte-Oriented Integer Compression · HackerPapers discussions

Stream VByte: Faster Byte-Oriented Integer Compression — mentioned by mananaysiempre

Sun, 11 Aug 2024 21:26:56 GMT

Stream VByte[1,2,3] is not far away from that idea, storing the packed lengths in a separate stream from the data bytes. (There’s also a more conventional interleaved design in varint-G8IU, but Stepanov decided to fuck everyone over and patented that.) [1] https://lemire.me/blog/2017/09/27/stream-vbyte-breaking-new-... [2] https://arxiv.org/abs/1709.08990 [3] https://github.com/lemire/streamvbyte

Paper and discussion history. A mention is not an endorsement.

Stream VByte: Faster Byte-Oriented Integer Compression — mentioned by lifthrasiir

Thu, 23 May 2024 23:59:57 GMT

Yeah, see Stream VByte [1] for more information. [1] https://arxiv.org/abs/1709.08990

Paper and discussion history. A mention is not an endorsement.

Stream VByte: Faster Byte-Oriented Integer Compression — mentioned by danking00

Thu, 23 May 2024 19:50:19 GMT

If I’m following, this presents an alternative format and implementation to LEB128 which encodes and decodes substantially faster. Notably, the implementation is quite simple. Cool! And agreed that modern CPUs really suffer from branches. Should I interpret the plot to mean the average elapsed wall clock time per integer decoded/encoded? And can I conclude the throughput is the reciprocal? So about 100,000 integers per second or around a 1 MB/s of decompressed data. I know this is a bit unfair because the implementation is much more complex, but my first thought is why I would use vu128…

Paper and discussion history. A mention is not an endorsement.