At VB Transform 2026, Ryan Phillips, head of enterprise product at Runway ML, explained how the company turned a stubborn bug—AI-generated avatars drifting off-center during real-time video generation—into a front-end feature after weeks of failed engineering fixes. Runway, an applied AI research company building general world models, showcased Runway Characters, a real-time video model enabling zero-latency interactions with AI avatars. Phillips emphasized that building robust AI products requires a high-quality evaluation set created through deep cross-functional alignment across product, design, research, and sales. The team runs internal workshops to align on specific failure modes, using examples like “Tooth,” a non-human character with no nose and very unusual teeth, to test extreme edge cases. Runway tracks evaluations in an Excel spreadsheet, logging tests daily and categorizing outputs as "minor" or "major" failures against a predetermined pass rate. “We set a bar before we get started on what percentage we need to pass, and when we hit that, we ship the model,” Phillips said.