NVIDIA’s NeMo-Tron labs have unveiled Dory, a 4-billion-parameter looped transformer model designed to address scalability and memory efficiency challenges in AI systems. According to Sam Witteveen, Dory organizes its layers into input, recurrent and output blocks, with the recurrent block using a looping mechanism to iteratively refine token representations, achieving greater computational depth without increasing memory demands. Inspired by ResNet models, Dory reintroduces input data during each iteration to enhance stability and maintain performance across tasks. Its elastic inference compute feature allows users to dynamically adjust loop counts to balance speed and accuracy. Dory outperforms larger models on benchmarks including instruction-following and long-context understanding despite its compact size, though its smaller size limits performance on knowledge-intensive tasks. The model faces limitations such as increased KV cache memory requirements at higher loop counts. Developed under NVIDIA’s NeMo-Tron initiative, Dory was pre-trained on an extensive dataset of 6 trillion tokens and employs supervised fine-tuning (SFT) and reinforcement learning. Released under an Open MDW license, Dory is accessible for both research and commercial use.