A news summary on model distillation explains that it involves a large, expensive "teacher" model—smart, slow, high-capacity, and costly to run—teaching a smaller, cheaper "student" model that is faster and easier to deploy. The core question of distillation is whether the student can learn not only from the original dataset but also from the teacher’s behavior, effectively training the small model on reality as interpreted by the big model.