November 5, 2026

Who checks the AI that checks your translation?

A quality score on an AI translation looks like a fact. Someone, or something, had to grade the translation first, and that grader is usually another AI model. If you rely on the score to decide whether to publish, send or sign, the grader's reliability matters as much as the translator's.

I want to look at that grader. Intento's 2026 benchmark of 229 AI translation solutions found that its own panel of AI judges could not reach enough agreement to assess quality in some languages. Our own scores have limits too, and I will lay those out as well, because I would rather you know them than assume them away.

Key takeaways

  • Intento's 2026 benchmark accepted a quality result only when three AI judges agreed on at least 90% of sentences, and in 14 of 96 languages fewer than half of the shortlisted solutions passed that check.
  • In Shona, the three judges reached that level of agreement for only 1 of the 11 solutions checked.
  • Changing the prompt moved one judge's results sharply: it flagged errors in 36% of sentences without examples and in 7% with them.
  • Only 43.7% of AI model outputs on MachineTranslation.com carried a quality score from July 12 to August 8, 2026, and the share ranged from 22.5% to 54.8% depending on the model.
  • In a small sample of rated translations, 1-star and 5-star translations averaged almost the same machine score, 9.08 and 9.10.

Who grades an AI translation?

Usually another AI model. An AI judge reads the source and the translation, marks errors by type and severity, and those marks become a score. Intento's 2026 benchmark used three judges from three vendors and a fourth model to combine their findings, a design meant to avoid trusting any single grader.

Scoring this way exists because nobody can have a person review every translation. For background on human and automatic methods, our guide to how machine translation evaluation works covers the basics. Scores are good for comparing systems at scale. From where I sit, the useful question about any score is how often independent graders would give the same number.

Do AI judges agree with each other?

Not reliably, because Intento accepted a result only when its three judges agreed on at least 90% of sentences, and in 14 of the 96 languages it checked, fewer than half of the shortlisted solutions passed. Intento says it could not assess quality there.

What Intento tested                                                                                           Result (Intento, 2026)
Minimum agreement before a result is acceptedThree judges agree on at least 90% of sentences
Languages where fewer than half of shortlisted solutions pass14 of 96
Shona: solutions that reach 90% judge agreement1 of 11
One judge's flagged sentences, without vs. with examples in the prompt36% vs. 7%

Shona shows how this looks in one language. Intento's Shona page reports that the judges reached 90% agreement for only one of the 11 solutions it checked, so it does not give a quality result for most of them.

The prompt result is the one that worries me. Intento reports that adding examples to the prompt changed how often one judge flagged an error from 36% of sentences to 7%, while the other two judges barely moved. The same judge gave very different results under different instructions. Intento also reports that the judges differ in strictness, and that they rated an English term left untranslated in a Ukrainian translation as minor while human reviewers rated it major.

Why do AI judges struggle in some languages?

Probably for the same reason AI translators do: they have seen far less text in those languages. Intento does not state a cause, so this is my inference. A judge that reads a language poorly can miss real errors or flag correct text, and the two cases are hard to tell apart without human reviewers.

I would argue that the rarer the language, the less a score should be taken at face value. The same limit applies to any automated score, including ours, which is why the next section sets out what ours leaves out.

What do our own quality scores leave out?

Quite a lot. In the 28 days from July 12 to August 8, 2026, only 43.7% of AI model outputs on MachineTranslation.com carried a quality score, and the share ranged from 22.5% to 54.8% depending on the model. Our scores are automated, so the same questions apply to them.

Scores also cluster near the top of the scale, with an average of 9.003 across that window, so small differences are easy to over-read. The sharper test is whether a score tracks what customers experience. In a small sample of translations that customers rated, those rated 1 star averaged 9.08 and those rated 5 stars averaged 9.10.

05109.08Rated 1 star9.10Rated 5 starsAverage machine quality score (0 to 10)

Average machine quality score by customer star rating, small sample of rated translations, MachineTranslation.com, 2026.

I read that as a reminder that a machine score and a customer's experience measure different things, and I cannot say from this data why they do not move together. A star rating can reflect more than accuracy, and the sample is small. Even so, it tells me not to treat our score as proof that a translation worked for the person who received it.

Is a jury of AI models better than a single judge?

Probably, because a 2026 EAMT paper from Smartling researchers argues that a single AI judge carries the bias of its own model, and it proposes combining judgments from several models instead. Intento's three-judge design points the same way, with a fourth model to combine their findings.

That is the logic behind SMART, which applies it to producing the translation: it runs the text through 22 AI models and returns the version most of them agree on, as explained in our piece on what SMART does differently. I think the best way to evaluate translation resembles the best way to produce it: several independent models, with disagreement treated as a signal. A jury still has limits, and Intento's Shona results show one. Judges that share the same blind spot will agree with each other and still be wrong.

How do you check a translation when you cannot rely on the score?

Compare several independent AI models, look at where they disagree, and have a speaker of the target language review anything that matters. Use a score as one input, never as the only check. The steps below apply this to everyday use, from reading disagreement to adding a professional review.

  1. Look at the disagreement. When SMART runs a translation through 22 AI models, the Key Term Translations panel shows the percentage of models that agreed on each term. Low agreement marks the passages worth reading closely.
  2. Treat rarer languages as higher risk. Intento's Shona results show why: for most solutions the judges could not agree enough to grade the output at all.
  3. Add a human check where it counts. With Human Verification, a professional translator reviews the AI output and returns it with a 100% accuracy guarantee.
  4. Do not stop at one model. A single AI model produces errors it cannot detect in its own output, and 34% of our customers said they were not confident enough to publish one AI translation unchecked. Our piece on why one AI model's output needs a cross-check covers the evidence.

Intento's 2025 benchmark showed how far AI translation has come for widely used languages, and our write-up of that benchmark covers it. For a rare language, I would ask a speaker of that language to check the output, and I would treat any score, including ours, as a prompt for questions.

Frequently asked questions

1. Does AI translate languages well?

Yes for most widely used languages, but how well varies by language, and the checks that measure it are weaker in rarer ones. In Intento's 2026 benchmark, AI judges could not reach enough agreement to assess quality in 14 of 96 languages. Review anything important before you rely on it.

2. How to check translation accuracy?

Compare the output across several independent AI models, look at where they disagree, and have a speaker of the target language review anything that matters. Treat a quality score as one input, never the only check. For high-stakes text such as legal or medical content, add a professional translator's review.

3. Can I trust AI to tell the truth?

Not on its own, because a single AI model cannot reliably spot every error in its own output. In our customer data, 34% said they were not confident enough to publish one AI translation without checking it. Compare several models, and have a person review text that matters.

4. Can an AI judge hallucinate?

Yes, an AI judge can give unreliable results. In Intento's 2026 benchmark, one judge flagged errors in 36% of sentences without examples in its prompt and 7% with them, and the judges rated an untranslated English term in Ukrainian as minor where human reviewers rated it major.

Photo of Rachelle Garcia

By Rachelle Garcia

Connect on LinkedIn

Rachelle leads product and AI at Tomedes, where she runs the experiments that turn internal data into better translation experiences. She writes about what actually happens when you build AI products such as MachineTranslation.com — the numbers, the surprises, and the parts that don't go to plan.

Share: