A new paper from Google DeepMind, authored by Xiaoyu Ma and Turing Award winner David Patterson, argues that the biggest obstacle to scaling large language model inference is memory, not raw computing power. The research states that the decode phase of autoregressive LLMs is fundamentally memory-bound and interconnect-limited, causing expensive chips to sit idle while waiting for data. The paper, titled “Challenges and Research Directions for Large Language Model Inference Hardware” and submitted to arXiv in January 2026, identifies trends like Mixture-of-Experts models with up to 256 experts, long reasoning chains, extended context windows, multimodal inputs, and retrieval-augmented generation as adding pressure on memory. Despite inference hardware sales projected to grow 4 to 6 times, the authors warn that service costs could jeopardize economic sustainability. To address these constraints, the paper proposes High Bandwidth Flash (HBF), a memory technology designed to deliver roughly 10 times the capacity of current solutions, aiming for approximately 512 GB of capacity compared to about 48 GB for HBM4. The paper acknowledges challenges like write endurance and page-read latencies, framing HBF as a research direction. Ma and Patterson also call for rethinking datacenter network designs and interconnect topologies, and argue that the industry’s standard benchmark, peak FLOPS, is the wrong yardstick for measuring accelerator performance.