HackerPapers

arXiv:2102.11245cs.AR

Silent Data Corruptions at Scale

Harish Dattatraya Dixit, Sneha Pendharkar, Matt Beadon, Chris Mason, Tejasvi Chakravarthy, Bharath Muthiah, Sriram Sankar

Abstract

Silent Data Corruption (SDC) can have negative impact on large-scale infrastructure services. SDCs are not captured by error reporting mechanisms within a Central Processing Unit (CPU) and hence are not traceable at the hardware level. However, the data corruptions propagate across the stack and manifest as application-level problems. These types of errors can result in data loss and can require months of debug engineering time. In this paper, we describe common defect types observed in silicon manufacturing that leads to SDCs. We discuss a real-world example of silent data corruption within a datacenter application. We provide the debug flow followed to root-cause and triage faulty instructions within a CPU using a case study, as an illustration on how to debug this class of errors. We provide a high-level overview of the mitigations to reduce the risk of silent data corruptions within a large production fleet. In our large-scale infrastructure, we have run a vast library of silent error test scenarios across hundreds of thousands of machines in our fleet. This has resulted in hundreds of CPUs detected for these errors, showing that SDCs are a systemic issue across generations. We have monitored SDCs for a period longer than 18 months. Based on this experience, we determine that reducing silent data corruptions requires not only hardware resiliency and production detection mechanisms, but also robust fault-tolerant software architectures.

arXiv comments
8 pages, 3 figures, 33 references
Journal reference
DOI
Subjects
cs.ARHardware Architecture cs.DCDistributed, Parallel, and Cluster Computing

On Hacker News

Linked by 7 identifiable commenters in 7 comments across 6 threads, from Jun 2021 to Aug 2024. Found in 5 HN submissions from Mar 2021 to Jun 2026. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Puget Systems' Perspective on Intel CPU Instability Issues 1 comment

    • It's pretty widely publicised that even on older Intel CPU's there is a non zero number of servers giving unexpected results at scale - https://arxiv.org/pdf/2102.11245

      philjohnView on HNParent context

  2. Intel Processor Instability Causing Oodle Decompression Failures 1 comment

    • You should go work for Facebook. At their scale they are encountering CPU bugs daily: > This has resulted in hundreds of CPUs detected for these errors https://arxiv.org/abs/2102.11245

      dist-epochView on HNParent context

  3. ECC RAM should be a human right 1 comment

    • Read the paper(s) below. Absolutely frightening, we all basically swimming in quicksand. ECC should be mandatory! Not using it is like huffing leaded gas. Silent Data Corruptions at Scale https://arxiv.org/abs/2102.11245 Harish Dixit has two other papers available…

      sitkackView on HNParent context

  4. Removing characters from strings faster with AVX-512 1 comment

    • Given Intel's AVX extension could cause silent failures on servers (very high work load for prolonged time, compare to end user computers), I'm not sure it would be a big win for servers either: https://arxiv.org/pdf/2102.11245.pdf.

      _rtld_global_roView on HNParent context

  5. Single random bit flip causes error in certificate transparency log 2 comments

    • These things exist, using trade names like chipkill or lockstep memory. Though they don't need to sacrifice half of the memory chips to get good error recovery properties. Note that this is still not end-to-end protection of data integrity. Bit flips happen in networking…

      fulafelView on HNParent context

    • It could be a CPU issue, and ECC doesn’t save you from all of these: https://engineering.fb.com/2021/02/23/data-infrastructure/si... https://arxiv.org/pdf/2102.11245

      nixgeekView on HNParent context

  6. Cores that don’t count [pdf] 1 comment

    • The article references Dixit et al. for an example of a root cause investigation of a CEE which is an interesting read: https://arxiv.org/pdf/2102.11245.pdf > After a few iterations, it became obvious that the computation of 𝐼𝑛𝑡(1.153)=0 as an input to the math.pow function in…

      freehrtradicalView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.