Validating UTF-8 In Less Than One Instruction Per Byte · HackerPapers discussions

Validating UTF-8 In Less Than One Instruction Per Byte — mentioned by 3pm

Thu, 16 May 2024 21:10:38 GMT

Good paper on UTF-8 validation performance: https://arxiv.org/pdf/2010.03090 The relatively simple algorithm (lookup) can be several times faster than conventional algorithms at a common task using nothing more than the instructions available on commodity processors. It requires fewer than an instruction per input byte in the worst case.

Paper and discussion history. A mention is not an endorsement.

Validating UTF-8 In Less Than One Instruction Per Byte — mentioned by striking

Sat, 20 Jan 2024 20:52:01 GMT

From https://github.com/websockets/utf-8-validate/tree/master/dep... > This C++ library is part of the JavaScript package utf-8-validate. The utf-8-validate package is routinely downloaded more than a million times per week. > If you are using Node JS (19.4.0 or better), you already have access to this function as buffer.isUtf8(input). And at https://github.com/websockets/utf-8-validate/tree/master/dep... > # Reference > John Keiser, Daniel Lemire, [Validating UTF-8 In Less Than One Instruction Per Byte](https://arxiv.org/abs/2010.03090), Software: Practice & Experience 51 (5), 2021

Paper and discussion history. A mention is not an endorsement.

Validating UTF-8 In Less Than One Instruction Per Byte — mentioned by tlb

Sat, 24 Sep 2022 18:23:30 GMT

Modern CPUs can look ahead in the instruction stream and run 4 or more instructions simultaneously. This isn't easy: you have to respect data dependencies where one instruction depends on the output of a previous one. When something depends on a load instruction that missed in the cache, CPUs can keep going farther ahead and do some other work while waiting for load to complete. This "speculative out-of-order execution" requires a huge number of transistors to consider various combinations of future instructions it might be able to execute every clock cycle, and burns some extra power doing…

Paper and discussion history. A mention is not an endorsement.

Validating UTF-8 In Less Than One Instruction Per Byte — HN submission

Tue, 20 Oct 2020 22:47:53 GMT

Validating UTF-8 in Less Than One Instruction per Byte

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Validating UTF-8 In Less Than One Instruction Per Byte — HN submission

Tue, 13 Oct 2020 22:59:53 GMT

Validating UTF-8 in Less Than One Instruction per Byte

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.