I ran the same German idiom through every GPT and Claude version we support. Here's what I found.

I test our platform constantly. That comes with building it — I use it the way our users do, on real content, across language pairs I care about. And I pay attention to where the models agree and where they don't, because disagreement is usually where the interesting things happen.

Last week I ran a test I keep thinking about. Not because the result was dramatic. Because it was counterintuitive in a way that I think matters for anyone making decisions about which AI model to use for translation, especially if you're choosing based on cost.

Here's what prompted it. I was running the German idiom "ich verstehe nur Bahnhof" through MachineTranslation.com, watching the individual model outputs appear. Five GPT versions. Three Claude versions. Eight models from two of the most widely-used AI families in the world, all translating the same four words simultaneously.

Seven of them got it right. One from each family got it wrong. And they made the exact same mistake.

The phrase I used and why it matters

"Ich verstehe nur Bahnhof" literally translates to "I only understand train station." In German, it means the complete opposite of what those words say — it is an idiom expressing total incomprehension, the equivalent of "it's all Greek to me" or "I don't understand a word of it."

The phrase is not obscure. It is common, widely used, and any competent German speaker would recognise it immediately as figurative rather than literal. It also has no direct English equivalent that maps word-for-word, which means a model translating it correctly has to understand the intent behind the phrase, not just the words.

That is why I find it useful as a test. It is not a trick. It is the kind of language that appears in business emails, casual professional exchanges, customer messages, and informal documentation every single day. If a model handles it badly, it will handle a wide range of similarly figurative language badly — and do so silently, with no indication that anything went wrong.

The results: Two models, two families, the exact same mistake

Here is what every model returned:

Two models returned "I only understand train station." — GPT-4O-MINI and Claude Haiku. Both went completely literal. The German word Bahnhof means train station, and that is exactly what both models translated.

The remaining six models understood that this was an idiom and rendered the meaning rather than the words. The SMART consensus correctly selected "I don't understand a thing" — the output that the majority of the model pool agreed on.

What struck me was not that two models failed. It was which two. GPT-4O-MINI is not the smallest model in the test — GPT-4.1-NANO is smaller, and it got the idiom right. Claude Haiku failed while Claude Sonnet and Opus succeeded. The failure did not correlate with model size the way I would have predicted. It correlated with something else.

Then I tested a standard business phrase, and everything changed

Before drawing any conclusions, I wanted to make sure I was being fair. The idiom test measures a specific capability, figurative language comprehension. I needed to know whether GPT-4O-MINI and Claude Haiku were generally worse models, or whether their weakness was specific to nuanced content.

So I ran a standard business phrase: "Please confirm receipt of this document" into German.

Every single model returned the same output.

GPT-4O-MINI: identical.

Claude Haiku: identical.

The SMART score was 9.5 — near-perfect consensus across the board, with only minor punctuation variation between some models.

This is an important result. It means GPT-4O-MINI and Claude Haiku are not bad models. For standard professional language (the kind of text that makes up the majority of business translation volume), they perform at exactly the same level as GPT-5.4 and Claude Opus. The quality gap is not general. It is specific to content where meaning is carried by convention, not by the literal value of the words.

That specificity is the problem. It means you can use a lower-tier model for months and see perfectly acceptable results — right up until your content contains an idiom, a cultural reference, a figurative expression, or a piece of specialised terminology that diverges from its everyday meaning. At that point, the model returns a fluent, grammatically correct output that is simply wrong. And nothing in the output signals that anything went wrong.

What this actually tells you about model versions — it's not about size, it's about generation

The most interesting finding from this test is the GPT-4.1-NANO result. This is the smallest model in the GPT-4.1 family. Its name implies it should be the least capable. And yet it correctly identified the idiom where GPT-4O-MINI (a model that sounds more substantial) did not.

The reason, as best I can interpret it, is architectural generation rather than size. The GPT-4.1 family was trained differently from the GPT-4O family. The O-MINI variant is a compressed version of an older architecture. The 4.1-NANO, despite being tiny, inherits the training approach of the 4.1 generation — which appears to handle figurative language better, even at small scale.

The same logic applies to the Claude family. Haiku is Claude's lightweight, cost-optimised tier. Sonnet and Opus are the mid and high-end versions. On standard text, Haiku holds its own. On figurative language, the gap between Haiku and Sonnet becomes visible — not because Haiku is a bad model, but because the kind of understanding required to decode "ich verstehe nur Bahnhof" is a capability that scales with model tier within the Claude architecture.

As Techidemics noted in their analysis of why every AI model has a hidden range of answers you never see, every model operates within a confidence envelope — a range of content types and complexity levels where it performs reliably and a boundary beyond which its outputs become less predictable. The idiom test is a clean illustration of where that boundary falls for GPT-4O-MINI and Claude Haiku.

Why this matters if you're using AI translation at scale

Most businesses that integrate AI translation via API do not run this kind of test. They evaluate the model on a sample of their content (usually the most common, most straightforward content) confirm the quality looks acceptable, and deploy. The decision to use GPT-4O-MINI or Claude Haiku is often driven by cost: they are significantly cheaper per token than their higher-tier counterparts, and for the majority of translation volume, the quality is genuinely equivalent.

The problem surfaces later, on specific content types. A customer service message that contains a colloquial expression. A business email that uses industry jargon with a non-literal meaning. A contract clause that uses a legal formula specific to one jurisdiction. A marketing headline built around a cultural reference. These are exactly the cases where the lower-tier model quietly diverges from the correct translation, and where the cost of the error is highest.

TechBullion's analysis of why the same phrase produces four different translations across AI models documents the same pattern across a broader model set: the divergence between model outputs is not random. It is systematic, and it concentrates on content types that require contextual and cultural interpretation rather than compositional translation.

The businesses most exposed to this risk are those translating content at volume (customer support, e-commerce product descriptions, HR communications, marketing localisation) where individual outputs are rarely reviewed before they reach the end user.

What I do about it at MachineTranslation.com

The reason I can show you this test is that MachineTranslation.com runs all of these model versions simultaneously on every translation. You do not have to choose between GPT-4O-MINI and GPT-4.1-NANO. You do not have to decide whether your content is the kind that needs Claude Opus or whether Haiku will do. The SMART engine runs them all (up to 22 models at once) and the output you receive is the translation that the majority of the pool agreed on.

When GPT-4O-MINI and Claude Haiku both returned "I only understand train station," they were in the minority. Six other models identified the idiomatic meaning. SMART selected the consensus: "I don't understand a thing." The two failing models were visible in the panel (you can see them, see their outputs, see where they diverged) but their error did not make it into the final result.

That is the practical value of running a consensus rather than a single model. More than 1,500,000 users rely on MachineTranslation.com precisely because this cross-verification catches the specific failures that single-model platforms return with full confidence and no warning. According to our internal benchmarking, the SMART consensus approach cuts translation error risk by 90% compared to single-model output, and tests like this one show exactly where that risk lives.

For content where even the consensus is not enough (high-stakes documents, content that will be published, anything with legal or regulatory implications), the platform also offers a human verification option directly within the interface. A professional translator reviews and verifies the AI output with a one-year guarantee. The banner appears automatically after translation precisely because there are cases where the right answer is not to trust any model alone.

I run this test, and variations of it, regularly. Not because I distrust the platform, but because the data it produces is the clearest argument I know for why running one model at a time is not enough.

This post is based on internal testing on MachineTranslation.com's development platform. The multi-model version testing shown here is not yet publicly available. I'm sharing the finding because I think the observation is useful regardless of where you run your translations.