Published on 18 Sep 2026, efficient large language model training can exceed consumer GPU memory limits, as a 7B model alone requires substantial memory for weights, gradients, optimizer states, and activations, making intelligent memory management essential. QLoRA stores base model weights in 4-bit precision while training low-rank adapter matrices, and DoRA separates magnitude and directional updates, though dynamic dequantization can reduce training throughput. GaLore projects large gradient matrices into smaller low-rank spaces, enabling full-parameter learning with less optimizer-state memory, but periodic projection calculations add computation. FSDP and ZeRO-3 distribute model parameters, gradients, and optimizer states across GPUs while optionally moving inactive data into system memory, though frequent PCIe transfers can bottleneck hardware lacking high-bandwidth interconnects. Selective activation checkpointing discards selected intermediate tensors during forward passes and recomputes them during backpropagation, useful for long context windows at the cost of additional computation. FlashAttention-2 processes attention in smaller tiles and avoids materializing the full attention matrix, while FP8 training reduces memory and bandwidth requirements on supported hardware, though compatibility and numerical stability require careful implementation. RingAttention distributes long sequences across multiple devices and overlaps communication with attention computation, extending context windows beyond individual GPU memory limits without specialized interconnects, but consumer PCIe and network connections can introduce communication delays when small batches leave insufficient computation to hide transfer latency.