NVIDIA has introduced a new host offloading technique for JAX-based large language model (LLM) training, addressing GPU high-bandwidth memory (HBM) bottlenecks. Leveraging the latest NVIDIA Blackwell architecture, this approach enables larger batch sizes and faster training throughput by moving selected activations to CPU memory during the forward pass and streaming them back during the backward pass. NVIDIA’s host offloading solution, detailed in a company blog post published on July 10, 2026, offers an alternative to activation rematerialization. The Blackwell GPU, paired with NVIDIA’s Grace CPU, achieves up to 900 GB/s bidirectional bandwidth via NVLink-C2C. On NVIDIA’s forthcoming Vera and Rubin platforms, this bandwidth doubles to 1.8 TB/s. Tests using the JAX-based MaxText framework on the dense Llama 3.1 (405B parameters) and the sparse DeepSeek-V3 (671B parameters) showed that for DeepSeek-V3, host offloading with pipelined transfers achieved 908.2 TFLOPs/s/device—a 57% improvement over activation rematerialization and a 67.7% boost compared to non-pipelined offloading, increasing GPU memory utilization to 165.2 GiB. For Llama 3.1, LHS-enabled QKV offloading improved throughput by 2.9%.