In the rush to automate evaluation, from grading student code to ranking research papers, we have embraced Large Language Models as judges. They are fast. These units are cheap. They scale. However, at a workshop at DHS 2026, Bhaskarjit Sarmah made a point that stuck with me: “you can’t trust LLM as a judge. I avoid using them.” LLMs pick up information from the web, which is filled with biased data created by humans and other biases that we will cover in this article.
Why judges are biased
Every bias we are going to explore has the same root cause: the judge answers from its own priors instead of from the evidence in front of it.
Moreover, it learned from human-written text where longer answers signal effort, citations signal rigor, consensus signals correctness, and certain country and brand names carry decades of loaded coverage. Alignment training then rewarded whatever human raters liked, such as polished, confident, and thorough-looking answers. Consequently, the model learned that “this looks good” is a proxy for “this is good,” and you have now hired that proxy as your grader.
In particular, one finding reframes everything: bias bites hardest when two answers are close in quality. When one is obviously better, bias cannot overcome the gap. When they are comparable, which is the normal production case, bias becomes the deciding factor.
The different biases
1. Position bias
For example, consider what happens when you show a judge two answers labeled A and B and ask which is better. Then, you run it again with the exact same two answers, but you swap the labels so what was A is now B. The verdict flips. Nothing about the content changed. Instead, only the slot it occupied changed.
Why does this happen? A model does not attend to every part of its input equally. Furthermore, the model weights the beginning and end of a prompt more heavily than the middle. In addition, human writing has conventions about where the winner goes. We lead with our strongest point, or we build to the best for last. The model absorbed those conventions as a prior about where good answers live, and it applies that prior even when position is arbitrary.
How bad is it? On simple two-way swaps, weaker judges land barely above a coin flip. It degrades fast with more options too. Give a judge three or four candidates instead of two, and most models flip the winner more than half the time when you reshuffle the order.
Testing for position bias
Now swap the two texts so the good one is labeled B, and run it again. Do this ten times per ordering. If the good answer does not win at roughly the same rate in both positions, you have measured position bias.
2. Verbosity bias
What happens: Between two answers that are both correct, the longer one gets the higher score, even when the extra words add nothing.
Why: In the model’s training data, longer answers genuinely were more often the better ones. Detailed writing correlates with effort and expertise, so “long equals thorough” is a real pattern it learned. The trouble is that the correlation collapses the moment someone pads an answer, which takes about ten seconds. The judge cannot tell depth from filler.
Why it matters practically: If anyone knows your judge rewards length, your evaluation is now trivially gameable. If you are feeding judge scores back as a training signal, you are actively teaching your model to waffle.
Test it:
3. Self-preference bias
What happens: A model scores its own writing higher than another model’s writing of equal quality. The unsettling part is that this holds even when you strip out any label saying who wrote what.
Furthermore, the model’s training shaped its sense of “well-written.” The model feels text is more fluent from the inside when it reads text matching its own phrasing, rhythm, and structure, and it quietly promotes fluency to quality. It is not recognizing a byline. It is recognizing itself in the prose.
How bad is it? Measured self-inflation, which is how much a model’s score for its own anonymized answer exceeded what other models gave that same answer, ran as high as roughly 16% for some models. The best performers stayed near 1 to 2%.
The rule this creates: never use the same model to generate answers and to grade them. It is the single most-violated rule in production evaluation pipelines, purely because it is convenient to use one API key.
4. Entity and tone bias
Entity framing: the same factual claim receives a different verdict depending on which country, company, or person the text names. The model learned sentiment associations for those names from an internet whose coverage of different entities is anything but neutral. When it reads “North Korea,” the geopolitical baggage attached to that token bleeds into what is supposed to be a content-only judgment. It cannot cleanly separate “is this claim positive” from “do I have positive associations with this name.”
Impact of emotional tone
Emotional tone: take a correct answer and rewrite it to sound angry or anxious. Same facts, same conclusion, just a different mood. Judges abandon it roughly 70 to 75% of the time. The model produces a one-sided effect. Adding emotion to a bad answer barely helps it, but adding emotion to a good answer wrecks its credibility. Cheerful is the safest tone. Fear and anger do the most damage.
There is a documented case where a judge stated in its own reasoning that the angry-toned answer was more direct and more technically precise, then picked the other one anyway because it worried the confrontational tone might put users off. The model told you it knew which answer was better and chose the other one.
Test it:
Rerun with “North Korea”, then with “Country X”. That last one is your control. It tells you the baseline sentiment for that exact sentence with no entity association attached. Any deviation from it in the other two runs is bias you have measured.
5. Identity bias
What happens: Tell the judge something about who submitted an answer, such as their gender, religion, health status, or immigration status. Its verdict on the identical text shifts.
Why: This uses the same machinery as entity bias. It is just pointed at people instead of countries. The model absorbed patterns in how different groups get described and evaluated across its training corpus. Those patterns leak into judgments that should be about content alone.
How bad: The spread between models here is enormous. The most robust judges barely moved. Weaker ones flipped their verdict on roughly a third of items purely on identity information.
6. Authority bias
What happens: Attach a citation to an answer and its score goes up. This happens even when the citation is completely fabricated.
Why: In training data, references correlate with rigor. The model learned to treat the appearance of a scholarly source as a credibility signal. It has no way to check whether that source exists.
The documented case is the damning part. Researchers took a pair of answers where a judge had correctly picked the better one. They then appended an invented URL to the worse answer. The judge reversed its verdict and named the fake reference as its reason. It did not just get fooled. It explained its reasoning in terms of the thing that fooled it. Fabricated book citations and famous-person quotes worked even better than URLs did.
7. Bandwagon bias
What happens: Tell the judge that most people preferred Answer A and it drifts toward A. There is no new information about the answers themselves.
Why: Deference to consensus is everywhere in human text. It is often a reasonable heuristic. The model picked it up as a general reasoning shortcut and applies it where it does not belong.
The odd finding: The stated percentage barely matters. Claiming 60% agreement moves the judge about as much as claiming 90%. It is not weighing how strong the consensus is. It is reacting to the fact that a consensus was mentioned at all.
What this means for you: audit what is actually sitting in your judge’s context window. Upvote counts, prior model rankings, reviewer tallies, or “this response was preferred by X” metadata. None of it belongs in front of a judge.
8. Distraction bias
What happens: Slip in a completely irrelevant detail about one answer and the verdict moves. The test case researchers used was a note that one assistant enjoys pasta and desserts.
Why: A transformer treats everything in its context as potentially relevant. If the judge cannot cleanly separate “the thing I’m evaluating” from “text that happens to be nearby,” the noise gets folded into the score.
The pattern worth noting: just like with tone, the damage is asymmetric. Irrelevant noise hurts the evaluation of good answers more than bad ones. Good answers are fragile under perturbation. Bad answers are already at the floor and have nowhere left to fall.
9. Fallacy-oversight bias
What happens: An answer reaches the correct conclusion through completely broken logic. The judge scores it as though the reasoning were sound.
Why: Checking whether a final answer is right is cheap. Auditing every step of someone else’s reasoning for validity is expensive. This applies to models and humans alike. The judge takes the shortcut available to it: verify the destination, skip the path.
Where this bites: models handle it reasonably well on simple factual questions where broken logic is obvious. It gets much worse when the flaw is subtle. This is the case for grading student work, reviewing code, or evaluating math. In all three, how someone got there is the entire point of the exercise.
Test it:
A’s reasoning is nonsense that happens to land on the right answer. See whether the judge catches that, or just ratifies the shared conclusion.
The fix: ground the judge
Every bias above comes from the judge free-forming a verdict. So stop asking it to.
Grounding replaces the open question, “is this good?”, which can only be answered from priors, with a closed one: “is this claim supported by this document?” That is textual entailment, a far narrower problem. Narrowness is what leaves bias nowhere to operate.
Notice what disappears. We check claims independently, so the model takes no position. Each claim is one sentence, so the model provides no length signal. We extract claims rather than quoting them, so the model adopts no tone. The checker is not the generator, so the model shows no self-preference. Engineers remove the bias surface rather than prompting it away.
Grounded models you can use
You don’t need to build this yourself. Purpose-built evaluators exist.
- Lynx (Patronus AI) is a Llama-3-Instruct fine-tune, 8B and 70B, that scores whether an answer is faithful to a given document. Trained on hard cases from PubmedQA, DROP, and FinanceBench, so it catches subtle stuff. The 70B beat GPT-4 across a range of hallucination scenarios and shows its reasoning.
- Bespoke-MiniCheck is the cheap, fast option. MiniCheck-FT5 is 770M params with GPT-4-level accuracy at roughly 400x less cost. The 7B version leads LLM-AggreFact at 77.4% and replies in about 200ms, fast enough for a live guardrail.
Also worth a look: Prometheus 2, GLIDER, Vectara HHEM.
Conclusion
LLM judges often score from absorbed priors rather than evidence. Small changes in position, length, tone, names, or framing can shift verdicts, especially when answers are similarly good. Some of the strongest biases stay hidden because the explanation never reveals what actually moved the score.
A better approach is to ask narrower questions: “Is this claim supported by this document?” and use grounded tools such as Lynx or MiniCheck. When grounding is impossible, use a diverse jury and stress-test the judge before trusting the result.
Read more: A Complete Guide to AI Red-Teaming (With Garak Tutorial)
Frequently Asked Questions
A. Models learn from training data where length often correlates with effort and expertise, leading them to mistake wordiness for higher quality.
A. It occurs when a model’s preference changes simply because the order of the provided answers is swapped, despite the content remaining identical.
A. You should never use the same model to both generate the content and grade it, as models tend to favor text that mimics their own style.
I am a Data Science Trainee at Analytics Vidhya, passionately working on the development of advanced AI solutions such as Generative AI applications, Large Language Models, and cutting-edge AI tools that push the boundaries of technology. My role also involves creating engaging educational content for Analytics Vidhya’s YouTube channels, developing comprehensive courses that cover the full spectrum of machine learning to generative AI, and authoring technical blogs that connect foundational concepts with the latest innovations in AI. Through this, I aim to contribute to building intelligent systems and share knowledge that inspires and empowers the AI community.
Login to continue reading and enjoy expert-curated content.
