Together AI has introduced a new autoscaling framework designed to optimize large language model (LLM) inference, addressing the challenges of managing GPU-intensive workloads. The system allows deployments to scale dynamically based on metrics such as in-flight requests, GPU utilization, and token throughput, improving performance under fluctuating demand while minimizing costs. Unlike traditional web services, LLM workloads are latency-sensitive and GPU-bound, with cold starts taking several minutes as new replicas load model weights into VRAM. Together AI’s approach focuses on leading indicators like queue pressure to preemptively scale before user-facing performance degrades. The system mitigates risks by allowing users to fine-tune scaling policies, including setting replica bounds, choosing scaling metrics, and defining timing windows. For example, time-to-first-token (TTFT) can balloon from 200 milliseconds to over 15 seconds under heavy loads. The platform supports eight autoscaling metrics, including inflight_requests, which provides a leading indicator of demand, and SLO-driven metrics like TTFT for low latency. An experiment highlighted in Together AI's blog showed that under identical traffic conditions, a deployment scaling on inflight_requests dynamically added replicas, reducing latency spikes, while policies based on TTFT and GPU utilization failed to scale. This announcement comes as enterprises increasingly move AI systems into production, with market research from 2026 emphasizing that efficient autoscaling is now a cornerstone of enterprise AI strategy.