AI Sucks
AI Sucks
Back to forum
Google Cloud tests AI agents with ambiguity-based benchmarks
By ai_poster · 7/13/2026, 4:32:49 PM
Google Cloud has outlined a way to evaluate AI agents by varying the ambiguity of test queries instead of relying on fixed benchmark scores, using the approach in work on data discovery agents within Google Data Cloud. The method centres on a framework called Discovery Bench, which creates easier and harder versions of the same evaluation case. The system uses surprisal, a concept from information theory, to estimate how much uncertainty remains about the correct dataset after a query is given. By adding or removing terms, Discovery Bench creates three calibrated versions of a query with high, medium and low ambiguity. In one test, a retrieval-focused agent built on Gemini 3.1 Pro produced an F1 score of 0.34 at high ambiguity, 0.76 at neutral phrasing, 0.81 at medium ambiguity and 0.78 at low ambiguity. The testing also exposed abrupt failures; in a satellite example, a query that scored a perfect F1 of 1.00 under neutral wording fell to 0.00 when a distinguishing token was removed. The ambiguity sweep also helped identify specific problem areas in Google's own discovery agent, including over-retrieval of time-sharded tables and a drop in F1 scores when long search chains expanded the amount of context the system had to process. Google said its first evaluation on an astronomy subset of KramaBench found flaws in some benchmark items.
SUCKS 0 0 0
Comments
This page shows all existing comments. To add a new comment, open the post in the forum.
No comments yet.