As hyperscalers move from building AI infrastructure to monetizing it, tokens per watt helps to reflect if revenue is scaling and if profitability is improving, with tokens representing the revenue side of the inference equation and power consumption representing associated costs. Nvidia’s offload-engine approach is designed to address accelerators sitting idle, waiting for KV cache, memory or data movement, thereby increasing tokens per watt by improving utilization and inference revenue from the same power footprint. Power is no longer an abundant resource, and inference growth will be constrained by how efficiently hyperscalers use existing power and accelerators. Memory is the biggest constraint for improving tokens per watt, as GPUs or XPUs not supplied with sufficient memory become underutilized. Offload engines are being designed into rack-scale architectures of AI infrastructure leaders like Nvidia, while others build open-standard solutions through Compute Express Link (CXL). Nvidia’s latest offload engines target GPU memory to accelerate inference, addressing the growing size of model KV caches that consume the majority of working memory during inference. Ideally, the KV cache would be stored in high-bandwidth memory (HBM) to minimize latency and maximize tokens per watt, but as models increase in complexity for multi-step reasoning or agentic deployments, KV cache can exceed HBM capacity, leading to offloading to slower memory tiers or recomputation.