Building More with Less: How to Navigate the Memory Chip Shortage

Every major AI buildout has multiple moving parts, but many operators are running into the same constraint. Demand for high-bandwidth memory (HBM) has outpaced what manufacturers can produce. That supply crunch is now spilling over into the conventional dynamic random access memory (DRAM) chip that every server depends on, and the numbers show how tight the market has become, resulting in shortages across industries.

The Center for Strategic and International Studies found that 50% to 70% of all memory chips produced this year will be consumed by data centers. The U.S. market relies on a relatively small number of manufacturers and building new factories to increase that supply can take years. This creates a problem that goes beyond a component. Operators can have GPUs, servers, and data center capacity ready to go, but a shortage of memory can still hold up deployment. At the same time, higher memory prices raise the cost of building AI infrastructure.

Reclaiming memory from decommissioned servers and reusing them in new AI systems is how data center operators have tried to solve memory chip shortages. Organizations that adopt this approach as a design requirement can benefit in three ways.

Adding capacity without waiting on new fabs

The most immediate benefit is that operators are relying on a pre-existing supply. Large server fleets hold vast quantities of DRAM from earlier upgrade cycles, which operators can redeploy through Compute Express Link (CXL) memory expansion devices. These devices attach memory over PCIe instead of through the CPU's native memory channels, so a newer DRAM-based server can use older generations of memory for additional capacity. A single expansion controller can host a dozen reclaimed modules and more than one terabyte of memory, with some controllers adding in-line compression that nearly doubles usable capacity.

Reuse becomes far simpler when memory compatibility is built into a platform from the start. Systems that support open interconnect standards and leave room in their layout for expansion devices give reclaimed modules a ready place to land, so each new server generation does not require a one-off adapter project.

Every module recovered this way stays out of the competition for fab output. No new memory hardware has to be produced, and operators are less dependent on distant fabrication plants.

Stretching scarce HBM further

AI servers draw on memory in tiers, each trading speed for cost. HBM sits beside the GPU or accelerator, where a single HBM stack can deliver over 1 terabyte per second of bandwidth. Conventional DRAM costs far less, with a DDR4 module topping out at 25.6 gigabytes per second. CXL, an open interconnect standard built on PCIe, adds a further tier of expanded and pooled memory while keeping data coherent across the hierarchy. That tier is where older modules fit naturally.

HBM is the most constrained memory in the data center, with its supply not being able to flex quickly. Its vertically stacked design costs more and takes longer to manufacture than conventional memory. There is no alternative for systems/GPUs designed to use HBM.

That scarcity makes every HBM-equipped accelerator too valuable to spend on work that doesn't need it. Caching, data preprocessing, retrieval, and other supporting services can run on reclaimed memory, leaving HBM free for latency-sensitive training and real-time inference. Right-sizing systems to fit their workloads helps avoid overprovisioning expensive components across massive server fleets.

Tiering does require close attention and care. CXL-attached memory adds some latency compared to directly attached DRAM, making it essential for operators to test configurations against real workloads before committing. Qualifying older modules on new platforms ahead of time means capacity is ready the moment procurement tightens.

Breaking the full-system refresh cycle

The third benefit is structural. Servers are built from subsystems that improve at different rates, yet many refresh cycles still retire an entire system when one component ages. Modular, disaggregated designs let operators replace GPUs, CPUs, and memory chips on independent schedules. A well-designed server can accommodate several generations of components, so memory that's already in the fleet stays in service as accelerators turn over.

Decoupling memory from every accelerator upgrade lowers the amount of new memory an operator needs to buy at shortage pricing or wait due to extended lead times. It also cuts electronic waste and Scope 3 emissions tied to manufacturing and sourcing new components, which supports the sustainability goals many operators have laid out for themselves.

Building for what comes next

Across a multi-year buildout, these gains compound. Operators who plan for reuse lower total infrastructure spend when supply chains are least forgiving, and they gain more control over their own timelines, since fewer deployments hinge on a single allocation from a single supplier.

Memory supply will eventually catch up with demand. Until then, operators and vendors that treat hardware reusability as a first-order engineering priority will be the ones that keep building without disruption.

About the author: Michael McNerney is senior vice president of marketing and network security at Supermicro. McNerney has over two decades of experience working in the enterprise hardware industry, with a proven track record of leading product strategy and software design. Prior to Supermicro, he also held leadership roles at Sun Microsystems and Hewlett-Packard.

The post Building More with Less: How to Navigate the Memory Chip Shortage appeared first on Embedded.