Silent Data Errors Redefine Test Coverage And Fleet Maintenance Strategies

Key Takeaways:

A chip can pass ATPG, transition-fault, stuck-at, and at-speed structural testing and still produce an incorrect arithmetic result that propagates until a wrong output surfaces.
Targeted functional tests at the system level and in the field can catch more of these stealth defects. Broad and deep functional test content and stress tests can identify SDEs in commercial processors.
At the fleet level, in-field hardware testing and application testing reduce the operational impact of SDE escapes.

Testing and server-mitigation strategies are struggling to keep pace with the damage caused by silent data errors (SDEs), stealth defects that can wreak havoc on long-duration programs like AI training runs. Shrinking design rules, increasing complexity, and chiplet-based packages only add fuel to the fire already raging around silent data errors, also known as silent data corruption (SDC) failures.

Though SDEs are technically rare, data corruption damage is common in large server fleets running at high utilization rates and containing leading-node processors. Early analyses by Google and Meta determined that SDEs afflict one in a thousand servers, which equates to a high defective parts per million level of 100-1,000 DPPM. Engineers running data centers tend to describe failure rates in terms of failures-in-time, where 1 FIT equates to one failure for every billion hours of device operation. But for an impressive rate of only 10 FIT, scaling to 10 million devices in a fleet means that an SDE failure occurs every 4 days, which is untenable.

“Historically, the higher reliability end of commercial electronics requirements has been around 100 to 300 defective parts per million (dppm) over the first year of life in the field. This is for a full reticle SoC, which was mostly a single chip or a relatively low number of chips that was expected to run workloads for hours to weeks,” said John Carulli, director for strategic partnership development at Advantest Innovation Center. “Now we need to adjust our capabilities to meet automotive quality sub-1 dppm for chips on the latest technology nodes for silicon and package with the latest design architectures with latest EDA tools and with little history.”

The high cost of silent data errors mainly arises from subtle manufacturing defects that can become failures at any point in the chip’s lifecycle. “SDC-related failures may result in the loss of data, lack of access, or disruption of services, forcing system companies running data centers to divert their workloads, shut them down, or even risk losing user trust. This results in significant business implications, often in billions of dollars,” said Nilanja Mukherjee, vice president of engineering, Tessent, at Siemens EDA in a recent presentation. [1] “With miniaturization and the introduction of new technologies such as gate-all-around and nanosheet transistors, the exposure of subtle defects that may result from physical imperfections during silicon manufacturing can’t be overstated. Not only do we need to improve test quality to reduce test escapes, on-chip test and monitoring capabilities are also needed to continuously evaluate how silicon is performing, thereby preventing failures.”

As AI needs change, so do the silicon testing requirements. “With agentic AI, the role of the CPU changes from the traditional orchestrator of accelerator activity to an active executor that continuously plans, reasons, calls tools, manages and initiates new operations,” explained Noam Brouchard, vice president of solutions engineering at proteanTecs. “When combined with rapid and unpredictable workload transitions, understanding what the silicon is actually experiencing under those workloads becomes increasingly important. That is why deep monitoring of timing margin, voltage, workload stress, temperature and degradation is critical for this new class of CPUs.”

SDEs are becoming an increasing problem partly because SDE faults are much more likely to happen at advanced process nodes, where process margins are slimmer, connections between materials are more likely to exhibit variability, and interconnects are smaller and therefore more resistive.

For all these reasons, engineers from design to test to system integration are calling for an “all hands on deck” collaborative approach to solving silent data corruption, which is proving more intractable than originally conceived. Because an estimated 80% of these corrupt execution errors are attributed to time-0 test escapes, better screening now includes more critical-path monitoring and expanded functional testing at the system and fleet levels. For the remaining 20% of SDEs, which manifest as intermittency and aging, a physical model of transistor-level reliability failure mechanisms, such as gate oxide breakdown or random telegraph noise, may help identify signatures associated with corrupt execution errors. [2]

At the same time, SDEs are redefining what it means to have a known-good die. “The traditional objective of test was to identify devices that fail. The new objective is increasingly to identify devices that may compute incorrectly while passing conventional tests,” said Chen He, fellow and senior director of automotive microcontrollers and microprocessors at NXP Semiconductor. “As a result, many organizations are adding deep functional testing specifically designed to verify computational correctness rather than simply checking for structural defects. Meanwhile, the industry is increasingly moving beyond wafer sort and final test toward more extensive system-level test as well as mission-mode and workload-aware testing.”

In other words, under the old definition, test coverage was defined as the percentage of modeled manufacturing faults that could be detected. Under this new definition, test coverage includes the probability of detecting an incorrect answer from a device calculation under realistic operating conditions and workloads.

But even with advanced screening and extended testing in-system, not every resistive open, bridge defect or small-delay defect — how SDEs often manifest — can be caught. In fact, because silent data corruption failures can depend on voltage, frequency, temperature, aging, and workload, exhaustive testing is impractical. For these reasons, ongoing system checks at the fleet level are now a necessity.

“Efficient SDE detection during both manufacturing test and periodic in-situ data center screening is essential for uninterrupted and reliable at-scale compute,” stated Intel’s Manu Shamsa in a recent paper. [3]

Companies like Google, Meta, AWS, and others are implementing fleet-wide screening, redundant computations, application-level SDE detection, and software-based detection methods to identify and quarantine corrupted cores.

What causes SDEs?
Since Google and Meta first sounded the alarm about SDEs in 2021, the industry has scrambled to provide solutions through better testing methods, improved DFT, expanded system-level functional testing, and in-system testing. “Perhaps the most profound emerging change is the idea that testing should not stop at shipment,” said He. “Several industry initiatives are focused on in-system test and silicon lifecycle management approaches that periodically evaluate silicon health after deployment. The rationale is straightforward: some SDEs originate from aging, wear-out mechanisms, or field-induced degradation that manufacturing cannot detect. Rather than just testing before shipment, engineers are now verifying quality throughout the product lifecycle.”

SDCs originate from subtle timing violations, aging effects, or marginal defects that escape standard semiconductor testing and data center monitoring. “Sources of SDC errors include permanent, intermittent, transient, and degrading faults,” said Jyotika Athavale, director of Silicon Lifecycle Management and RAS Architecture at Synopsys. “Root causes can be extrinsic manufacturing defects, intrinsic silicon aging, or transient errors. Severe defects are easily detectable during manufacturing test. If the defects are weak, they can create circuit marginalities that fail only under certain operating conditions. Latent defects are not symptomatic until after the components have been operational for a certain duration. Monitoring environmental changes in the silicon, as well as application stress and tracking timing margin changes for memory and logic paths over time, allows for prediction of an SDC error before it manifests.”

Silent data errors cause incorrect calculations in arithmetic operations such as 10÷2=4.

These faults are “silent” because they leave no record or trace in the system logs. They can lead to incorrect numerical calculations or “not-a-number,” an undefined mathematical result. That corrupt value then cascades through large data centers and is typically identified only after a training run returns a result that is inconsistent with expectations.

Once errors are identified at a system level, it is incredibly difficult to trace the failure back to the device’s defect. “The system-to-ATE correlation has always been a challenge, and it has been a key issue in bring-up forever,” said Carulli. “On the system side, engineers are speaking in firmware with register loads, register writes, register reads, and maybe we can get an understanding of this or that design block is failing. But it’s really hard to then map that understanding back to the structural test world where we’ve got a lot more granularity in understanding loading scan chains and understanding what cone of logic I’m interacting with and what number of patterns might be failing to all the diagnostics that take you down to a defect picture in failure analysis where “x” marks the spot. That analysis can take weeks, involving teams from different disciplines of EDA, test, FA, and system levels, all speaking their own language.”

So far, the industry has identified the most common physical causes of SDEs to include:

Resistive opens: A partially connected metal or via interconnect. The node may still function but it frequently exhibits increased RC delay or voltage degradation.
Resistive shorts / bridging defects: Unintended weak connections between nets, producing incorrect logic under a particular voltage, temperature, or switching condition.

In addition to such extrinsic manufacturing defects, SDEs can also be caused by electrical marginalities, radiation, and faults caused by intrinsic reliability mechanisms.

Another way to look at this is by mapping these failures onto the bathtub curve that chips follow over their lifetime (see Figure 1). During initial production, early life failures occur and many SDEs are part of this population. After a certain duration, the chip’s useful life begins, characterized by a low, constant random failure rate. Eventually, device wear-out mechanisms kick in, where chips fail due to intrinsic negative bias temperature instability (NBTI), hot carrier injection (HCI), and time-dependent dielectric breakdown (TDDB) mechanisms.

Fig. 1: Semiconductor failure rate over the lifetime of a device. Source: Siemens EDA

How fleets manage SDCs
Hyperscalers use a layered software approach to detect operational inconsistencies caused by silent data corruption, isolate the server or device, run diagnostics to confirm the error (typically confined to one core), and then quarantine the afflicted chip until it can be replaced.

Meta has published three ways it controls SDCs in its fleets. [4] First, its Fleetscanner takes a server out of production and runs targeted computational tests, often during preventive maintenance procedures. The tests select inputs with known results. If a CPU produces the wrong result, the machine can be quarantined and investigated. The company recently increased its testing frequency to cover the fleet more aggressively.

Secondly, during regular production, Meta’s Ripple program runs short test patterns and operations (milliseconds to seconds) and checks the results. The third approach, Hardware Sentinel, analyzes application exceptions and system behavior to identify core-level anomalies without allocating dedicated test workloads. Meta describes this as a test- and architecture-agnostic approach that improves detection by 40% versus testing-based approaches across architectures, applications, and data centers.

Google monitors its hardware for corrupting CPUs or cores, as well as its software for symptoms of corruption. [5] Google’s multi-layered software defense strategy against SDCs includes:

Performing end-to-end checksums;
Replicated computations;
Comparing replicas;
Invariant/assertion checks;
Checksums on data moving through the system; and
Periodic verification of data at rest.

The company’s Spanner application-level telemetry approach detects corruption and removes faulty machines from the fleet. For field returns, Google modifies its screening methods, enabling proactive identification of problematic cores before failure.

Lots of tests catch lots of SDEs
If there is one silver lining regarding SDEs it is that they are not associated with any new device failure modes. SDEs also do not discriminate by device type. They are just as likely to be found in logic as in memory circuitry and also appear in clock distribution hardware. SDEs are equally likely to occur in data paths as in control paths on the chip. [6]

Nonetheless, there is no equivalent to error correction code, which is so effective in memory hardware, for logic hardware. “ECC and similar techniques are extremely valuable for protecting stored and communicated data, but they do not protect every logic path in a processor,” said proteanTecs’ Bouchard. “An SDC can result from an instruction being executed incorrectly while still producing a perfectly plausible output. That is why root cause analysis can be extraordinarily difficult.”

In one example, Intel engineers traced an SDE to marginal gate-work-function metal step coverage at the transistor level. [6] Transistors with good step coverage exhibited normal operating behavior, but when the gate metal coverage was poor, probing showed a 30% higher threshold voltage and 30% lower drain-to-source drive current. The same study found that typical SDE defects were resistive opens, most easily detected by system-level functional tests during write operations.

To adequately screen defects that manifest as SDEs, Intel developed a large set of functional tests designed to verify that every operation and computation delivers the correct result. [7] This test suite includes checking core-to-core and socket-to-socket communications; exercising the various on-die caches; and executing nearly all floating-point, integer, and data manipulation instructions as part of the company’s data center diagnostic (DCDiag) test suite.

While developing tests for 1.2 million processors across five CPU generations, Intel determined that over 1,000 functional tests in DCDiag suite and 5,000 synthetic stress tests were needed to catch all SDEs. “Results on five generations of Intel Xeon processors demonstrate that a large and diverse set of system-level functional tests is necessary to screen defects that manifest as SDEs. Every product we have evaluated requires a unique combination of tests for effective screening, necessitating thousands of individual SoCs to be characterized to construct optimal test recipes,” stated Intel’s David Lerner in the report. [7]

“While about 50% of the defective parts may be identified with just 5% of the tests, over half of the (1,000) tests are required to screen 90% of the SDEs,” as shown in Figure 2. Unfortunately, Intel found that the bespoke test suite for one product generation did not help develop the test suite for the next generation, so existing product data does not provide a good basis for selecting tests for new designs.

Fig. 2: Testing 1.2 million processors across 1,000 tests determined that over 70% of defects are only detected by a single test. Source: Intel

Similarly, Intel employed some 5,000 synthetic stress tests designed to expose worst-case timing paths while checking every calculated result. Again, many faults were detected by only a single stress test, so massive numbers of tests are needed to catch SDEs comprehensively. These tests are architecture-aware and target failure mechanisms the DCDiag test suite misses.

One reason SDEs escape traditional testing is the methodology used to catch small delay defects. “Most companies rely heavily on the conventional transition fault model,” said Mukherjee. “Although defect models such as timing-aware and cell-aware timing are gaining momentum, they rely on single input switching (SIS), meaning a transition is allowed to propagate through only one input, while the other inputs are held at a constant path. However, we know that SIS doesn’t necessarily mimic how an IC operates in functional mode. Additionally, literature shows that multiple input switching (MIS) can result in larger propagation delays compared to SIS. In other words, preventing multi-input switching results in inadequate testing of the path delays, leaving room for small delay defects to escape.”

Mukherjee further emphasized the need to extend conventional stress test methods. “Another factor that is greatly overlooked is the improper integration of current test methodologies into a multi-corner, multi-mode design paradigm, where delay defects must be targeted under different PVT corners and test patterns optimized to work across all such corners,” he said. “Only this way will we be able to close the gap with the operating conditions that the IC is subjected to when running in-system.”

The industry also needs intelligent voltage- and current-based stress tests. “Conventional stress test methodologies that are dependent on burning or voltage-related tests are falling short of eradicating latent effects that show up as early life failures,” Mukherjee continued. “For HTOL testing, most companies today rely on either toggle coverage or ad hoc methods to generate patterns without ways to measure their impact on the overall test quality. We strongly believe that intelligent voltage- and current-based stress test solutions are necessary. Once we have the ability to measure stress test coverage, smart ATPG techniques can be developed to cover the holes to guarantee a more holistic coverage of the entire design.” Mukherjee highlights developments in activity-based and defect-based stress test models that simulate device behavior under stress and generate patterns that activate such conditions. “Finally, we need to provide a DFT infrastructure that would enable one to apply appropriate manufacturing test patterns for in-system testing in the field.”

AI also helps. “The ability to identify outliers in datasets and diagnose symptoms such as time delays or voltage degradation can help expose early indications of impending SDC errors,” said Synopsys’ Athavale. “AI/ML algorithms could flag when certain conditions are met that show early signs of SDC. Silicon lifecycle management is a solution that allows chip designers to monitor, analyze, and optimize their semiconductor devices throughout their life. This makes it easier for designers to track and gain actionable insights on their devices in real time and ultimately, detect silent data corruption before it’s too late.”

Improved outlier detection provides another way of detecting silent defects. “During production, our outlier detection methodology uses parametric measurements together with machine learning to model a device’s expected profile and then flag devices that diverge from that profile, providing another layer of defect filtering,” said proteanTecs’ Brouchard. “The approach includes timing margin-based and IDDQ-based methods. The analytics platform trains these models, which are deployed on ATEs for inline decision-making. At system-level test, time series analysis can reveal intermittent margin drops under functional workloads that a shorter test might miss.”

Conclusion
Though the semiconductor industry has come a long way in understanding silent data corruption and how it manifests, developing the most efficient testing strategies is still a work in progress. “We have learned that SDEs are not one problem but a collection of failure mechanisms that evade traditional detection methods and often manifest in unexpected ways,” said NXP’s He. “Engineers now recognize that silent corruption can originate from manufacturing defects, process variation, timing marginalities, aging mechanisms, voltage fluctuations, thermal effects, and design interactions that escape conventional validation and production testing. One of the most challenging aspects of SDE debugging is that the symptom may appear far removed from the underlying hardware defect.”

Often, the processor itself reports no fault. “Instead, engineers may observe corrupted databases, abnormal AI model behavior, incorrect analytics results, software crashes, or inconsistent application outputs,” said He. Because no single method can catch all SDEs, He suggests the industry is turning to a combination of:

In-process monitors or on-chip sensors to reveal timing margins;
DFT enhancements such as mission-mode and path-delay testing;
Deep functional testing including randomized data streams;
Fleet-level monitoring for outlier behavior;
Diverse instruction combinations, cross-core comparisons, and redundant execution testing; and
System-level test and in-system test to improve SDE detection.

Generally, semiconductor testing is changing from one-time test-at-shipment mode to continuous quality monitoring. “Testing increasingly cannot be viewed as something that ends when the chip ships,” said Bouchard. “A device can have healthy margin in production and later lose that margin because of aging, workload stress, environmental conditions, or an external issue such as power or clock delivery. The industry needs visibility across those stages rather than isolated snapshots. For instance, in the field, real-time health monitoring tracks timing margin continuously, providing a performance index that reflects how close a device is to failure, enabling more proactive maintenance.”

But wider data sharing will be essential to step-function improvements in quality. “While the engineering teams understand that collaboration is needed, our respective business and legal teams balance the financial implications,” said Advantest’s Carulli. “So, for cross-business/cross-domain problems, progress will be slower without some change to an updated approach. We need to get better at sharing data across the companies in the ecosystem and with the universities so that we can get ahead on this important topic.”

References

N. Mukherjee, “Silent Data Corruption in AI Datacenters: Causes, Detection and Mitigation – Tessent,” https://www.youtube.com/watch?v=kAA0I5msaDs.
D. Sangani et al., “Possible Origins, Identification, and Screening of Silent Data Corruption in Data Centers,” 2024 IEEE International Reliability Physics Symposium (IRPS), Grapevine, TX, USA, 2024, pp. 1-7, doi: 10.1109/IRPS48228.2024.10529436.
M. Shamsa et al., “Improved Silent Data Error Detection Through Test Optimization Using Reinforcement Learning,” 2025 IEEE International Reliability Physics Symposium (IRPS), Monterey, CA, USA, 2025, pp. 8C.1-1-8C.1-5, doi: 10.1109/IRPS48204.2025.10983294.
H.D. Dixit and S. Sankar, “How Meta Keeps Its AI Hardware Reliable,” July 22, 2025, https://engineering.fb.com/2025/07/22/data-infrastructure/how-meta-keeps-its-ai-hardware-reliable/
“Silent Data Corruption,” https://support.google.com/cloud/answer/10759085?hl=en&sjid=1247380514469191753-NC
M. Shamsa and D. Lerner, “Defect Mechanisms Responsible for Silent Data Errors,” 2024 IEEE International Reliability Physics Symposium (IRPS), Grapevine, TX, USA, 2024, pp. 1-5, doi: 10.1109/IRPS48228.2024.10529392.
D. Lerner et al., “Screening for Manufacturing Defects that Manifest as Silent Data Errors in Data Center Processors,” 2025 IEEE International Electron Devices Meeting (IEDM), San Francisco, CA, USA, 2025, pp. 1-4, doi: 10.1109/IEDM50572.2025.11353795.

Related Articles
Identifying Sources Of Silent Data Corruption
Rooting out the causes of silent data corruption errors will require testing improvements and much more.

Silent Data Errors Still Slipping Through The Cracks
Expanded DFT and test strategies are catching more SDEs, but this rare problem in server fleets is far from solved.

Strategies For Detecting Sources Of Silent Data Corruption
Manufacturing screening needs improvement, but that won’t solve all problems. SDCs will require tools and methodologies that are much broader and deeper.

The post Silent Data Errors Redefine Test Coverage And Fleet Maintenance Strategies appeared first on Semiconductor Engineering.