A Google Deepmind engineer explains the memory crisis
A report from FMS 2026
As you know, since April we considered memory one of the easier trades in AI, and expressed a preference for NAND over DRAM.
In July, Sandisk and Kioxia drew down dramatically, more than six months earlier than when NAND prices are widely thought to peak (~H2 2027). This was largely attributed to some investors being forced to sell significant holdings, as well as the “second derivative”, the rate of change of NAND price increase, slowing down.
Since then, Jukan received a bit of heat for this tweet which was viewed 3.6 million times. Those angry about Jukan’s views fail to appreciate that being bullish about memory market fundamentals does necessarily mean being bullish about the price action in the short or medium term.
This is why we can, without contradiction, be bullish about s/d dynamics while also have reservations about pressure on the stock because of a turnover in the shareholder base and because it will take some time for people to persuaded that “this time is different”.
In this note, we summarise one of Dr Xiaoyu Ma’s presentations at the Future of Memory and Storage Convention 2026.
Dr Xiaoyu Ma is a Senior Software Engineer at Deepmind. He created the concept of high bandwidth flash with David Patterson in 2023, and has published numerous important papers.
Dr Ma explains the demand vectors driving the LLM inference / memory crisis. We also flag some honest questions about NAND arising from his arguments.
First, Dr Ma’s explanation of the basic flow of LLM inference.
There are two stages, prefill and decode. Prefill is the stage where you process all the input tokens (the basic units of text or data) all at once.
This can be hundreds, thousands, or even up to millions of tokens all at once. Prefill is inherently parallel, so it’s compute-bound and runs great on TPUs and GPUs.
The second decode stage, where we generate tokens step by step, one token at a time, is problematic. This sequential step is called autoregressive, and that's the fundamental reason why the decode phase is memory-hungry.
There’s a data structure called KV cache that connects the two stages. The size of the KV cache is proportional to the input sequence length plus the output sequence length. While the auto-regressive decode stage creates a memory bandwidth challenge, the KV cache creates both a bandwidth and capacity challenge.
These challenges are made a lot of more difficult by the six trends below.
First, Mixture of Experts (MoE). Transformers began eight years ago as dense architectures; the industry has since shifted to sparse MoE, moving from low tens of experts to hundreds or even thousands in today’s state-of-the-art models.
The consequence is a blow-up in the size of model weights.
Editor’s note: Model weights currently sit in HBM, so this point is not relevant to SSDs. However, if HBF is successfully commercialised there is a good argument for storing model weights on HBF. HBF could tighten supply for NAND in the same way that HBM has tightened supply for DRAM. We’ll park this argument for now until we have better visibility on HBF - keep a lookout for the Sandisk investor day on 13 August.
Second, multi-modality.
Models have moved from text generation to audio and video generation, and per-token storage scales accordingly: a text token is a couple of bytes, an audio token roughly a thousand bytes (~1,000x text), and video tokens hundreds of bytes up to megabytes — another hundred-to-thousand-fold increase. More bytes per token means a proportionally larger KV cache for the same sequence length.
Editors note: This is indeed a well understood bull case for NAND demand. We are also in the very early innings of this playing out. See tweet below.
Third, reasoning.