How to Compare Translations Using Metrics and Human Review

Two teams run the same MT shootout on the same 300 segments and reach opposite conclusions. One picks System A on a 2-point BLEU lead. The other picks System B after a linguist spots three dropped legal clauses that BLEU never penalized. Both decisions look defensible on paper, and one of them ships a translation error into a contract. The gap is not the tools. It is the absence of a rule for when a number is trustworthy and when a human has to look.

The reliable way to compare translations is a two-stage pipeline: run automated metrics first to narrow thousands of segment pairs down to a review set, then confirm the meaningful differences with targeted side-by-side (S×S) human review using an MQM-style rubric. Metrics tell you where to look. Human review tells you who wins.

Threshold orientation: Metric gaps are not equally trustworthy. Neural metrics (COMET, CometKiwi QE) separate systems reliably at much smaller gaps than BLEU, which needs a wide margin before a lead means anything. Treat the point values in this article as directional guidance, not fixed constants: if the gap between two systems is narrow on your metric, escalate to human S×S before declaring a winner. The idea of mapping a metric gap to an estimated probability that a human would agree is well established in metric-evaluation research (Kocmi et al., WMT 2021).

Key Takeaways

The most defensible way to compare translations is to combine automated metric pre-screening with targeted S×S human review, using metric-gap-to-accuracy reasoning to decide when a lead is real and when it needs a human.

What is the fastest workflow to compare translations and pick a winner?

A repeatable six-step process covers every translation comparison task, from a two-system MT shootout to a vendor SLA audit.

1. Define scope and stakes. Identify file formats (XLIFF, TMX, JSON), language pairs, and high-risk field types (legal clauses, numeric thresholds, calls to action). Treat version-vs-version consistency as the unit of comparison, because per-language proofreading misses inconsistencies in shared fields like deadlines and eligibility criteria.
2. Run automated metrics in batch. Execute sacreBLEU and chrF for lexical coverage, COMET or BLEURT for supervised correlation, and CometKiwi QE for reference-free scoring. Log metric versions and parameters for reproducibility.
3. Sort segment pairs by gap size and risk. Rank by the metric with the strongest human-correlation track record for your language pair, then flag segments in high-risk fields regardless of gap size.
4. Sample for S×S human review. Pull the top-gap segments plus a stratified random sample. Aim for at least 500 segments total to keep variance manageable.
5. Apply the decision rule. Use the confidence thresholds and adjudication policy from the decision-rule section below. Two annotators vote; a third adjudicates ties.
6. Export results and update TM and glossary. Tag error spans with MQM categories, push fixes to your translation memory, and log the comparison run for audit.

Pro tip: Spend human review hours on segments with legal language, numeric values, and calls to action. A mistranslated dosage or contract clause costs far more to fix after publication than a stylistic preference does.

Which automatic metrics should you run, and how do you read the gaps?

Metrics fall into four families, each with different input requirements and human-correlation profiles.

Lexical metrics (BLEU, chrF) count n-gram overlaps against a reference translation. BLEU uses modified n-gram precision plus a corpus-level brevity penalty, so scores shift with the number of references and the testset composition. chrF extends this to character n-grams, which helps for morphologically rich languages. Both are fast and free but correlate poorly with human judgment when comparing systems that differ substantially in approach.

Embedding-based metrics (BERTScore, BARTScore) match tokens in semantic space rather than exact strings, catching paraphrases that lexical metrics penalize. They require no training data but can reflect the biases of the underlying language model.

Supervised regression metrics (BLEURT, COMET family) are trained on human ratings and generally show the highest correlation with human judgment. BLEURT has reported relatively high correlation with human judgments in benchmark experiments.

Reference-free QE (CometKiwi QE) scores translations using only the source sentence, which makes it practical when no reference exists.

Key point: Neural metrics separate systems at smaller gaps than BLEU, which needs a wide margin before a lead is meaningful. Running a neural metric alongside BLEU gives you a fast filter and a higher-confidence signal in one pass. Treat any single gap threshold as directional and confirm close calls with human review.

When should you use Quality Estimation and LLM-based evaluators?

Quality Estimation (QE) produces quality scores without any reference translation. The MT quality-estimation literature documents three method families: handcrafted features, deep learning with pre-trained language models, and LLM-centered approaches such as GEMBA, EAPrompt, and KPE.

Use QE or LLM evaluators when:

• No reference translation exists (new content, low-resource languages).
• You need continuous deployment scoring at high throughput.
• You want explainable, span-level quality signals without full MQM annotation.
• You are running iterative system selection and need fast directional signals.

Avoid relying on them alone when:

• Content carries legal or safety consequences.
• The language pair or domain is underrepresented in the evaluator’s training data.
• You have not validated the evaluator against an internal gold set.

LLM-based QE approaches show promise for zero-shot and explainable predictions, but they have not yet universally surpassed pre-trained language model approaches. Validate any LLM-based estimator against your own gold data before using it for production decisions.

Pro tip: Neural QE misses some mechanical errors like dropped tags and placeholders. A two-line script that checks tag and placeholder integrity catches them in seconds, before you spend a metric run on them.

How do you run MQM, S×S MQM, and S×S Relative Ranking effectively?

MQM (Multidimensional Quality Metrics) is the industry standard for structured error annotation. Annotators mark error spans, assign them to categories (accuracy, fluency, terminology, style, locale convention), and rate severity (minor, major, critical). The output is a weighted error count per segment, which rolls up to a document-level score.

S×S MQM places two translation outputs side by side against the same source. Annotators mark errors in both at once, which forces direct comparison and reduces the cognitive drift that inflates variance in point-wise annotation. Research published at ACL 2025 reports that S×S MQM improves inter-translation error-marking consistency by about 38.5% on average for explicitly compared systems.

S×S Relative Ranking (RR) asks annotators to pick the better translation for each segment pair without marking individual errors. It is faster than S×S MQM and produces stable system rankings, which makes it the right tool for iterative system selection when you do not yet need root-cause error data.

Practical setup checklist:

• Align source segments and both outputs in a shared spreadsheet or CAT tool export (XLIFF preferred).
• Provide a one-page annotation guide covering MQM categories, severity definitions, and tie-breaking rules.
• Recruit bilingual subject-matter experts, not generalist translators, for technical or legal content.
• Set an adjudication policy: two annotators vote; a third resolves disagreements above a defined severity threshold.
• Export annotation logs in XLIFF or JSON for downstream TM integration.

Pro tip: Use S×S RR for fast system ranking in early evaluation rounds. Switch to S×S MQM only when you need error categories to drive remediation, such as identifying which terminology gaps to fix in your glossary.

How many segments do you need for a reliable comparison?

Variance in metric scores rises sharply for testsets under roughly 500 segments. A comparison run on 200 segments may show the same gap as one on 1,000 segments, yet the smaller run carries far wider confidence intervals, because statistical power grows with more data. The gap can stay stable while the p-value shifts with sample size.

Sampling strategies that improve reliability:

• Stratify by content type (UI strings, legal clauses, marketing copy) so each stratum is represented proportionally.
• Oversample high-risk fields: numeric values, dates, legal thresholds, and product names.
• Cover all language pairs in scope, not just the highest-volume one.
• Build balanced gap bins: include segments where systems agree as well as segments where they diverge, to avoid selection bias.

Paired sampling (matching each source segment to both system outputs) is more statistically powerful than random independent sampling because it controls for segment-level difficulty.

Testset size guidance: Below 500 segments, treat metric-gap conclusions as directional only and require human S×S confirmation before any vendor or model decision.

How do you turn metric gaps and human votes into a defensible decision?

A decision rule needs four components working together.

Decision checklist:

• Gap threshold: Is the metric gap large enough to be meaningful for your metric? Neural metrics clear this at smaller gaps than BLEU; set the bar per metric and treat close calls as ties until a human breaks them.
• Minimum sample: At least 500 segments, or document the confidence interval explicitly.
• Human-vote margin: System A wins if it takes more than 60% of S×S RR votes in the sample.
• Adjudication policy: All critical-severity MQM errors in the losing system must be logged, regardless of the overall vote.

The reasoning behind a gap threshold is a mapping from a raw score difference to an estimated probability that a human annotator would prefer the higher-scoring system. Metric-evaluation research fits this relationship empirically, and the consistent finding is that lexical metrics like BLEU need a much larger gap than neural metrics to reach the same confidence.

Practical threshold: Set your gap thresholds per metric against your own historical human-agreement data rather than borrowing fixed numbers. As a rule of thumb, a neural metric (COMET, CometKiwi QE) will reach your target confidence at a smaller gap than BLEU, so a BLEU lead of a point or two should trigger human review, not an automatic decision.

What tools and file formats support bulk comparison workflows?

File formats to prioritize:

• XLIFF for segment-aligned bilingual exports from any major TMS or CAT tool; it preserves inline tags and placeholders.
• TMX for translation memory exchange and batch re-scoring against updated references.
• JSON and CSV for segment IDs, metric scores, and flag columns in pipeline scripts.

Metric runners and integrations:

• sacreBLEU and chrF scripts (pip-installable, reproducible with explicit tokenization flags).
• COMET and BLEURT Python wrappers for supervised metric scoring; containerize for reproducible runs.
• CometKiwi QE via the Unbabel COMET library for reference-free scoring at scale.
• TMS and CAT export hooks for XLIFF round-trips; most enterprise TMS platforms support XLIFF 2.0.

For scale, prefer API-first metric runners and containerized environments with pinned model versions. Log sacreBLEU parameters (tokenizer, lowercase flag, reference count) in every run so results are reproducible across teams. For non-Latin scripts, verify tokenization settings before running BLEU; character-level chrF is more robust for languages like Chinese or Arabic. Translation quality also affects downstream multilingual SEO performance, so consistent metric logging across language pairs pays dividends beyond QA.

How do you recruit and train annotators for MQM and S×S tasks?

Annotator quality is the single largest source of variance in human evaluation. A well-designed recruitment and calibration process cuts that variance before it contaminates results.

Recruitment and training steps:

• Recruit bilingual subject-matter experts, not generalist translators, for technical, legal, or medical content.
• Run a structured training module: annotation exercises on pre-labeled segments, followed by a calibration session where annotators discuss disagreements.
• Provide a gold-standard set of 50 to 100 adjudicated segments for ongoing accuracy checks.
• Measure inter-annotator agreement using Cohen’s kappa or pairwise accuracy against the gold set before scaling to the full batch.
• S×S setups typically show higher agreement than point-wise MQM because direct comparison reduces individual scoring drift.

Pro tip: Run a triply annotated pilot batch of 50 to 100 segments before committing to the full run. If S×S disagreement concentrates on style rather than accuracy or terminology, tighten the rubric’s style-severity definitions before scaling. Catching rubric gaps at 100 segments costs far less than re-annotating 2,000.

What belongs in the comparison report, and how do you act on it?

A comparison report is only useful if stakeholders can read it and engineers can act on it.

Minimum report elements:

• Scope: language pairs, content types, file formats, and system versions compared.
• Sample size and sampling strategy, with a confidence interval or variance note if under 500 segments.
• Metrics run, with versions and parameter settings (sacreBLEU flags, COMET model version).
• Metric gaps per system pair, with the estimated human-agreement level for each gap.
• Human-vote tallies (S×S RR) and MQM error counts by category and severity.
• Recommended action: promote System A, re-evaluate, or escalate to a full MQM audit.

Export MQM error spans as tagged XLIFF or JSON and route critical-severity errors directly to remediation tickets. Prioritize fixes by severity first, then by volume: a single critical terminology error in a legal clause outranks fifty minor fluency issues in UI strings. Bilingual side-by-side review catches omissions and meaning shifts that automated checks miss, so include at least one human-reviewed sample in every report for high-stakes content.

Pro tip: Append a one-page gap-to-agreement mapping table to every report, so a reader can see at a glance which leads were decided by metrics and which needed a human.

What are the most common mistakes when comparing translations?

• Over-relying on BLEU for unrelated systems. BLEU penalizes valid paraphrases and correlates poorly across systems with different generation strategies. Use COMET or BLEURT as the primary signal; treat BLEU as a sanity check.
• Small-sample false confidence. A 100-segment comparison with a clean gap still carries wide confidence intervals. Document the interval or increase the sample before reporting a winner.
• Ignoring inline tags and placeholders. A translation that drops {product_name} or <b> tags scores well on fluency metrics but breaks the product. Pre-filter for tag integrity before running any metric.
• Conflating fluency with fidelity. A fluent translation can omit a key clause. MQM’s accuracy category and targeted S×S checks on high-risk fields catch this; aggregate metric scores do not.
• Skipping numeric and legal field checks. Dates, prices, and legal thresholds need a dedicated pass. A cross-language comparison table that enumerates concrete fields spots inconsistencies that single-language review misses.

The case for S×S human review as the final arbiter

The conventional wisdom in localization QA is to trust the metric with the highest reported correlation. That instinct is understandable but incomplete. Metrics measure what they were trained to measure, and no single metric captures the full range of errors that matter in production: a dropped legal clause, a culturally inappropriate register, a numeric transposition.

The more defensible approach is to use automated metrics for what they do well, which is fast, cheap, large-scale pre-screening, and then reserve human judgment for the decisions that carry risk. S×S annotation is the right human step because it forces direct comparison, reduces individual scoring drift, and produces agreement levels that point-wise annotation cannot match.

The gap-to-agreement mapping matters just as much. Converting a raw score difference into an estimated human-agreement percentage turns an opaque number into a decision you can defend in a review meeting.

How Lara Translate fits into your comparison pipeline

Running a defensible translation comparison takes fast batch translation, clean bilingual exports, and a coordinated human review step. Lara Translate covers those three inputs to your pipeline. It is a translation platform, not an MT-metric scorer, so it produces the outputs and the human validation you feed into your BLEU, COMET, or CometKiwi runs rather than computing those scores itself.

Relevant capabilities for your comparison workflow:

• API for batch translation across 200+ languages and 61 document formats, with XLIFF exports for direct pipeline integration. See the supported file formats for the full list.
• AI plus human validation that coordinates professional linguist review on the same segments your automated metrics flagged, so S×S human review is a built-in step rather than an external handoff.
• Glossaries and translation memory that carry approved terminology across comparison runs, which reduces noise from inconsistent term choices.
• Incognito mode and enterprise-grade security for sensitive legal, medical, or financial content that cannot leave a controlled environment.

Teams that need a quick start for English-centric comparisons can try Lara Translate’s English translation workflow right away. For enterprise integrations with human S×S review built in, the AI plus human validation page covers the full service model and available trial options.

FAQ

What is the fastest way to compare translations automatically?

Run sacreBLEU and CometKiwi QE in batch against your segment pairs. CometKiwi QE needs no reference translation and separates systems at smaller score gaps than BLEU, which makes it the faster path to a defensible automated signal.

How many segments do you need to compare translations reliably?

Aim for at least 500 segments. Below that, metric variance is high enough that the same gap can flip direction with a different random sample, so treat smaller runs as directional only.

When does side-by-side human review outperform automated metrics?

Side-by-side human review catches omissions, meaning shifts, and cultural register errors that no metric reliably detects. It is the required step for legal, medical, or safety-critical content, and for any comparison where the automated metric gap is too small to be confident.

What is MQM and why does it matter for translation comparison?

MQM (Multidimensional Quality Metrics) is a structured annotation framework that categorizes errors by type (accuracy, fluency, terminology) and severity (minor, major, critical). It produces weighted error counts that support root-cause analysis and direct remediation, unlike aggregate metric scores.

Can Lara Translate support a full comparison workflow?

Lara Translate provides API-based batch translation, XLIFF exports for pipeline integration, and an AI plus human validation service that coordinates professional linguist review on flagged segments. It supplies the translation and human-review inputs to a comparison workflow; you run the BLEU, COMET, or CometKiwi scoring in your own pipeline.

This article is about

This article is about how to compare machine translation outputs reliably and pick a winner you can defend: which automated metrics to run (BLEU, chrF, COMET, BLEURT, CometKiwi QE), how to read the score gaps between systems, when to escalate to side-by-side human review with an MQM rubric, how many segments a comparison needs, and how to turn metric gaps and human votes into a documented decision. It also shows where Lara Translate fits, supplying batch translation, XLIFF exports, and AI plus human validation to feed the workflow.

Sources

• Enhancing Human Evaluation in Machine Translation with Comparative Judgement (ACL 2025)
• To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation (Kocmi et al., WMT 2021)
• A Survey on Evaluation Metrics for Machine Translation
• Translation Quality Assurance: Side-by-Side Expert Review (Algoran)

Recommended

• AI + Human Translation | Lara Translate
• Adaptive Translation API | Lara Translate
• 200+ languages | Lara Translate
• Translate to and from English with Lara Translate