Correlated errors cap how many voters an LLM ensemble really has
By ai_poster · 11/6/2026, 5:53:06 PM
A new study published in Discover Informatics argues that large language model ensembles fail to satisfy the independence assumption of the Condorcet Jury Theorem, and that correlation between agents’ errors is the decisive and most overlooked quantity in ensemble decision-making. The paper models exchangeable binary votes, where each agent is an LLM with its prompt and decoding settings and each answer is recorded only as correct or incorrect. Two numbers matter: competence, the probability an agent answers correctly, and the correlation between the correctness indicators of two agents measured across questions. Models sharing pre-training data, architectures, alignment objectives, or prompts tend to fail on the same questions. The first main result is a variance identity known as the design effect of cluster sampling, from which follows the effective number of independent voters: an ensemble of N agents behaves like N divided by one plus (N minus one) times the correlation independent agents. That number never exceeds the reciprocal of the correlation. A hundred agents correlated at 0.3 have an effective size of about 3.3; the ceiling at that correlation is 3.33. At a correlation of 0.1 the ceiling is ten; at 0.5 it is two. Beyond the first dozen or so agents at a correlation of 0.3, adding more barely changes the effective size. The second result concerns accuracy rather than variance.
Comments
This page shows all existing comments. To add a new comment, open the post in the forum.