SemiAnalysis sees memory doubling, CPO delay, and CPU catch-up in AI stack

Memory prices have 2x to 3x more upside as AI inference demand for KV cache creates a multi-year structural shortage that supply growth of 20% to 30% per year cannot match, according to SemiAnalysis founder Dylan Patel.

"Memory is not a short-term shortage — it is a multi-year structural deficit," Patel said in a podcast interview. "Supply grows 20% to 30% per year, while AI demand is doubling and doubling again. The gap keeps widening."

Patel's firm, which has grown from a Substack newsletter to a 90-person research operation tracking the full AI supply chain, identified the memory inflection point in December 2024 after OpenAI released its o1 reasoning model. The shift from short-context chat interactions to long-context reasoning workloads has exploded demand for KV cache — the memory buffer that stores relationships between tokens during inference. A single reasoning session can consume 10 times the memory of a standard chat interaction, and the trend is accelerating as models from OpenAI, Anthropic and DeepSeek push deeper into agentic workflows.

The structural imbalance is already reshaping downstream markets. Patel said Chinese mid-tier smartphone shipments have dropped 40% as memory costs squeeze lower-margin segments, and he predicted Apple will raise iPhone and MacBook prices next year. "Memory prices will keep rising until consumer electronics are compressed to a new equilibrium and AI gets the capacity it needs," he said. "From trough to trough, the long-term growth is undeniable."

CPU demand is surging, but the rally has limits. Patel described the current CPU boom as a "mini-cycle" driven by two forces: a structural shift as reinforcement learning and agentic workloads require more CPU cycles for environment validation and tool calling, and a massive catch-up effect as millions of AI accelerators shipped over the past three years lacked adequate CPU pairing.

"The catch-up effect is real," Patel said. "Once the historical backlog is filled, only incremental demand remains." He cautioned against extrapolating the current growth rates indefinitely. In dollar terms, a fully loaded Blackwell system costs about $50,000 per GPU, while a companion CPU runs roughly $5,000. "Memory and AI accelerators are the big numbers. CPU was underpriced and is now repriced, but it will not outgrow AI chips forever," he said.

Nvidia has guided to $20 billion in CPU revenue for its Vera line, while Arm, AMD and Intel have all benefited from the procurement wave. Amazon's Graviton chips are seeing surging rental demand, and Nvidia's Vera CPU — with fewer than 100 cores but faster single-thread performance than AMD's 256-core flagship — targets workloads where AI compute stalls waiting for CPU responses.

Co-packaged optics mass production is pushed to 2029, extending the copper cable window. Patel said CPO — a technology that integrates optical transceivers directly with switch silicon to reduce power and latency — will not reach volume production until late 2028 to 2029, later than the 2027 timeline many investors expect.

"Manufacturing yields are not there, chip designs are not optimized, and the supply chain is not mature," Patel said. Nvidia's Rubin and Rubin Ultra GPUs will use all-copper interconnects, and even the subsequent Feynman architecture is not fully committed to CPO. The delay benefits copper cable suppliers such as Amphenol, which Patel said will outperform expectations as the CPO transition slips. SemiAnalysis recently published a report to institutional clients that is "constructive on copper and non-CPO optics, cautious on CPO itself."

Behind-the-meter power is becoming the default for new data centers. Patel projected 20 gigawatts of new data center capacity this year, rising to 30 GW next year and 50 GW the year after. Within a few years, half of incremental demand will be served by on-site generation rather than grid power, he said.

The shift is driving demand for combined-cycle gas turbines from GE Vernova, Mitsubishi and Siemens, as well as unconventional solutions including repurposed marine, rail and truck engines. "It sounds crude, but it runs, and people are already using it," Patel said. Solar-plus-storage is expected to undercut gas within two years, and longer-term, Patel flagged orbital data centers as a potential solution where solar panels operate without atmospheric interference.

SemiAnalysis's largest research team is now its Data Centers, Energy and Industrial unit — not semiconductors — tracking every data center and power plant deployment globally. The power conversion supply chain, spanning IGBTs, silicon carbide, gallium nitride MOSFETs, solid-state transformers and supercapacitors, is undergoing rapid innovation as data centers demand cleaner power delivery at higher voltages.

Anthropic's profitability counters the AI ROI skepticism. Patel disclosed that Anthropic turned free cash flow positive in the second quarter, with April and May both profitable and June tracking similarly. The company's annualized recurring revenue has surpassed $50 billion with gross margins above 70%. OpenAI's revenue is also accelerating as Codex adoption grows.

Patel cited his own firm as a case study: SemiAnalysis's annualized AI spending surged from under $100,000 last November to $11 million today for a 90-person team, with AI costs now exceeding one-third of total employee costs. "The ROI is enormous because we can build products and improve everyone's efficiency," he said. "Companies that cut AI budgets will fall behind."

This article is for informational purposes only and does not constitute investment advice.