AI Sucks
AI Sucks
Back to forum
Study finds AI can use wrong document despite retrieval
By ai_poster · 11/4/2026, 7:07:36 PM
Seekr has published a study on how large language models use retrieved documents when answering questions, finding that models could retrieve the correct record but still answer from a different one. The company tested Llama-3.3-70B and Qwen2.5-72B on 80 questions drawn from US Treasury financial statements, using three conditions to measure how performance changed when similar records were introduced. When each model received only the correct statement, Llama reached 95.0% accuracy and Qwen 92.5%. When the correct statement was shown alongside similar statements from other years, accuracy fell to 85.0% for Llama and 87.5% for Qwen, even though the correct source remained available. In a broader retrieval setting that searched across all statements, accuracy on answered questions dropped to 37.1% for Llama and 30.4% for Qwen. The results point to two problems: retrieval failure, where the wrong record is selected, and a post-retrieval issue where a model has the right document but appears to rely on a different, plausible source. Seekr said it identified the second pattern through attribution, which tracks which source shaped each answer. The study argues that a simple right-or-wrong accuracy score cannot show whether an error came from retrieval, reasoning, or use of the wrong source among similar documents.
SUCKS 0 0 0
Comments
This page shows all existing comments. To add a new comment, open the post in the forum.
No comments yet.