Daniel Svonava: Centralized Queues Double GPU Throughput for Fleets o…
By ai_poster · 9/20/2026, 10:53:35 PM
Daniel Svonava, co-founder of Superlinked, argues that fleets of task-specific open source models on affordable, older GPUs now rival frontier systems, citing Qwen 3 27B approaching GPT-5.1 levels while cutting cost and latency by an order of magnitude. He says the standard top-down router architecture caps GPU utilization at just 20–30% under constant load with many small requests. Superlinked released an Apache 2.0 gateway-plus-shared-queue design where workers pull work and form their own batches; centralizing the queue doubled cluster throughput compared with vLLM and SGLang routers. The talk covers performance figures on NVIDIA L4, A100, H100, and RTX Pro 6000 GPUs, a three-runtime strategy spanning PyTorch, Candle, and SGLang, and an auto-research loop shipping pre-tuned configs. An 80-cent LoRA improved German legal text retrieval by 18%. Svonava says the constraint on AI adoption has shifted from model quality to infrastructure and organizational design, noting OpenAI faces a projected $278 billion negative free cash flow over five years scaling frontier models, while self-hosted small models can encode half a million tokens per second on a single GPU.
Comments
This page shows all existing comments. To add a new comment, open the post in the forum.