Silent Data Corruptions at Scale · HackerPapers discussions

Silent Data Corruptions at Scale — HN submission

Mon, 29 Jun 2026 17:29:55 GMT

Silent Data Corruptions at Scale

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Silent Data Corruptions at Scale — mentioned by philjohn

Mon, 05 Aug 2024 09:57:18 GMT

It's pretty widely publicised that even on older Intel CPU's there is a non zero number of servers giving unexpected results at scale - https://arxiv.org/pdf/2102.11245

Paper and discussion history. A mention is not an endorsement.

Silent Data Corruptions at Scale — mentioned by dist-epoch

Fri, 23 Feb 2024 14:49:42 GMT

You should go work for Facebook. At their scale they are encountering CPU bugs daily: > This has resulted in hundreds of CPUs detected for these errors https://arxiv.org/abs/2102.11245

Paper and discussion history. A mention is not an endorsement.

Silent Data Corruptions at Scale — HN submission

Sun, 18 Feb 2024 06:06:03 GMT

Silent data corruptions at scale (2021)

84 points; 39 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Silent Data Corruptions at Scale — mentioned by sitkack

Sun, 22 Jan 2023 01:27:00 GMT

Read the paper(s) below. Absolutely frightening, we all basically swimming in quicksand. ECC should be mandatory! Not using it is like huffing leaded gas. Silent Data Corruptions at Scale https://arxiv.org/abs/2102.11245 Harish Dixit has two other papers available https://arxiv.org/search/cs?searchtype=author&query=Dixit%2C... Revisiting Memory Errors in Large-Scale Production Data Centers: Analysis and Modeling of New Trends from the Field https://users.ece.cmu.edu/~omutlu/pub/memory-errors-at-faceb... Justin Meza has a similar body of research.…

Paper and discussion history. A mention is not an endorsement.

Silent Data Corruptions at Scale — mentioned by _rtld_global_ro

Sun, 01 May 2022 14:17:22 GMT

Given Intel's AVX extension could cause silent failures on servers (very high work load for prolonged time, compare to end user computers), I'm not sure it would be a big win for servers either: https://arxiv.org/pdf/2102.11245.pdf.

Paper and discussion history. A mention is not an endorsement.

Silent Data Corruptions at Scale — mentioned by nixgeek

Sun, 04 Jul 2021 13:28:22 GMT

It could be a CPU issue, and ECC doesn’t save you from all of these: https://engineering.fb.com/2021/02/23/data-infrastructure/si... https://arxiv.org/pdf/2102.11245

Paper and discussion history. A mention is not an endorsement.

Silent Data Corruptions at Scale — HN submission

Sun, 04 Jul 2021 11:46:39 GMT

Silent Data Corruptions at Scale

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Silent Data Corruptions at Scale — mentioned by fulafel

Sun, 04 Jul 2021 11:43:41 GMT

These things exist, using trade names like chipkill or lockstep memory. Though they don't need to sacrifice half of the memory chips to get good error recovery properties. Note that this is still not end-to-end protection of data integrity. Bit flips happen in networking, storage, buses between everything, caches, CPUs, etc. See eg [1] [1] https://arxiv.org/abs/2102.11245 Silent Data Corruptions at Scale (based on empirical data at Facebook)

Paper and discussion history. A mention is not an endorsement.

Silent Data Corruptions at Scale — HN submission

Sat, 26 Jun 2021 17:51:09 GMT

Silent Data Corruptions at Scale

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Silent Data Corruptions at Scale — mentioned by freehrtradical

Thu, 03 Jun 2021 12:47:51 GMT

The article references Dixit et al. for an example of a root cause investigation of a CEE which is an interesting read: https://arxiv.org/pdf/2102.11245.pdf > After a few iterations, it became obvious that the computation of 𝐼𝑛𝑡(1.153)=0 as an input to the math.pow function in Scala would always produce a result of 0 on Core 59 of the CPU. However, if the computation was attempted with a different input value set 𝐼𝑛𝑡(1.152)=142 the result was accurate.

Paper and discussion history. A mention is not an endorsement.

Silent Data Corruptions at Scale — HN submission

Wed, 03 Mar 2021 22:33:55 GMT

Silent Data Corruptions at Scale

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.