How to Compare Translations Using Metrics and Human Review

Two translations of the same file can both read fine and still be far apart on the thing that matters: a dropped legal clause, a flipped number, a term that quietly breaks your glossary. Eyeballing them does not scale, and trusting a single score does not hold up when a decision gets questioned. The reliable path is a two-stage one, machines to narrow the field, humans to make the call.

To compare translations reliably, run automated metrics first (BLEU, chrF, COMET, CometKiwi QE 22), map the resulting deltas to estimated pairwise human accuracy, then validate meaningful differences with targeted side-by-side (S×S) human review using an MQM-style rubric.

Two quick orientation points before the workflow:

• Automated pre-screening narrows thousands of segment pairs to a manageable review set. BLEU and chrF are fast but require a reference translation; CometKiwi QE 22 runs reference-free and reaches high pairwise accuracy at much smaller numeric deltas than BLEU.
• S×S human review is where defensible decisions get made. Research published at ACL 2025 shows that S×S MQM has been shown to improve inter-annotator error-marking consistency substantially compared to point-wise MQM alone.

Threshold orientation: A delta of ~1.3 BLEU points corresponds to roughly 70% estimated pairwise accuracy; CometKiwi QE 22 reaches 90% accuracy at a delta of approximately 0.9 points. If your delta falls below those thresholds, escalate to human S×S before declaring a winner.

Key Takeaways

The most defensible way to compare translations is to combine automated metric pre-screening with targeted S×S human review, using metric-delta-to-accuracy mapping to set escalation thresholds.

What is the fastest workflow to compare translations and pick a winner?

A repeatable six-step process covers every translation comparison task, from a two-system MT shootout to a vendor SLA audit.

1. Define scope and stakes. Identify file formats (XLIFF, TMX, JSON), language pairs, and high-risk field types (legal clauses, numeric thresholds, calls to action). Cross-language side-by-side review should treat version-vs-version consistency as the unit of comparison, because per-language proofreading misses inconsistencies in shared fields like deadlines and eligibility criteria.
2. Run automated metrics in batch. Execute sacreBLEU/chrF for lexical coverage, COMET or BLEURT for supervised correlation, and CometKiwi QE 22 for reference-free scoring. Log metric versions and parameters for reproducibility.
3. Sort segment pairs by delta magnitude and risk. Rank by the metric with the highest human-correlation track record for your language pair, then flag segments in high-risk fields regardless of delta size.
4. Sample for S×S human review. Pull the top-delta segments plus a stratified random sample. Aim for at least 500 segments total to keep variance manageable.
5. Apply the decision rule. Use pairwise accuracy thresholds (see the decision-rule section below) and a simple adjudication policy. Two annotators vote; a third adjudicates ties.
6. Export results and update TM/glossary. Tag error spans with MQM categories, push fixes to your translation memory, and log the comparison run for audit.

Pro Tip: Focus human review hours on segments with legal language, numeric values, and calls to action. A mistranslated dosage or contract clause costs far more to fix post-publication than a stylistic preference.

Which automatic metrics should you run, and how do you read the deltas?

Metrics fall into four families, each with different input requirements and human-correlation profiles.

Lexical metrics (BLEU, chrF) count n-gram overlaps against a reference translation. BLEU’s original design uses modified n-gram precision plus a corpus-level brevity penalty, which means scores shift with the number of references and testset composition. chrF extends this to character n-grams, which helps for morphologically rich languages. Both are fast and free but correlate poorly with human judgment when comparing systems that differ substantially in approach.

Embedding-based metrics (BERTScore, BARTScore) match tokens in semantic space rather than exact strings, catching paraphrases that lexical metrics penalize. They require no training data but can reflect the biases of the underlying language model.

Supervised regression metrics (BLEURT, COMET family) are trained on human ratings and generally show the highest correlation with human judgment. BLEURT has reported relatively high correlations with human judgments in benchmark experiments.

Reference-free QE (CometKiwi QE 22) scores translations using only the source sentence, making it practical when no reference exists.

Key figure: Empirical mapping research shows CometKiwi QE 22 reaches 90% estimated pairwise accuracy at roughly 0.9 points, while BLEU needs a delta of ~1.3 just to reach 70%. Running both together gives you a fast filter and a high-confidence signal in one pass.

When should you use Quality Estimation and LLM-based evaluators?

Quality Estimation (QE) produces real-time quality scores without any reference translation. The MTQE survey documents three method families: handcrafted features, deep learning with pre-trained language models, and LLM-centered approaches such as GEMBA, EAPrompt, and KPE.

Use QE or LLM evaluators when:

• No reference translation exists (new content, low-resource languages).
• You need continuous deployment scoring at high throughput.
• You want explainable, span-level quality signals without full MQM annotation.
• You are running iterative system selection and need fast directional signals.

Avoid relying on them alone when:

• Content carries legal or safety consequences.
• The language pair or domain is underrepresented in the evaluator’s training data.
• You have not validated the evaluator against an internal gold set.

LLM-based QE approaches show promise for zero-shot and explainable predictions, but the MTQE survey notes they have not yet universally surpassed pre-trained language model approaches. Validate any LLM-based estimator against your own gold data before using it for production decisions.

Pro Tip: CometKiwi QE 22 misses some mechanical errors, like a dropped placeholder or a transposed number. A two-line script that checks tag and number integrity catches them in seconds, so run it alongside your QE pass.

How do you run MQM, S×S MQM, and S×S Relative Ranking effectively?

MQM (Multidimensional Quality Metrics) is the industry standard for structured error annotation. Annotators mark error spans, assign them to categories (accuracy, fluency, terminology, style, locale convention), and rate severity (minor, major, critical). The output is a weighted error count per segment, which rolls up to a document-level score.

S×S MQM places two translation outputs side by side against the same source. Annotators mark errors in both simultaneously, which forces direct comparison and reduces the cognitive drift that inflates variance in point-wise annotation. ACL 2025 findings confirm that S×S MQM improves inter-translation error-marking consistency by ~38.5% for explicitly compared systems.

S×S Relative Ranking (RR) asks annotators to pick the better translation for each segment pair without marking individual errors. It is faster than S×S MQM and produces stable system rankings, making it the right tool for iterative system selection when you do not yet need root-cause error data.

Practical setup checklist:

• Align source segments and both outputs in a shared spreadsheet or CAT tool export (XLIFF preferred).
• Provide a one-page annotation guide covering MQM categories, severity definitions, and tie-breaking rules.
• Recruit bilingual subject-matter experts, not generalist translators, for technical or legal content.
• Set an adjudication policy: two annotators vote; a third resolves disagreements above a defined severity threshold.
• Export annotation logs in XLIFF or JSON for downstream TM integration.

Pro Tip: Use S×S RR for fast system ranking in early evaluation rounds. Switch to S×S MQM only when you need error categories to drive remediation, such as identifying which terminology gaps to fix in your glossary.

How many segments do you need for a reliable comparison?

Variance in metric scores increases sharply for testsets under roughly 500 segments. Empirical sampling research confirms that metric deltas stay relatively stable across sample sizes, but p-values shift with sample size because statistical power grows with more data. A comparison run on 200 segments may show the same delta as one on 1,000 segments, yet the smaller run carries far wider confidence intervals.

Sampling strategies that improve reliability:

• Stratify by content type (UI strings, legal clauses, marketing copy) so each stratum is represented proportionally.
• Oversample high-risk fields: numeric values, dates, legal thresholds, and product names.
• Cover all language pairs in the scope, not just the highest-volume one.
• Build balanced delta bins: include segments where systems agree as well as segments where they diverge, to avoid selection bias.

Paired sampling (matching each source segment to both system outputs) is more statistically powerful than random independent sampling because it controls for segment-level difficulty.

Testset size guidance: Below 500 segments, treat metric-delta conclusions as directional only and require human S×S confirmation before any vendor or model decision.

How do you turn metric deltas and human votes into a defensible decision?

A decision rule needs four components working together.

Decision checklist:

• Delta threshold: Is the metric delta above the threshold for your target pairwise accuracy? (e.g., >0.9 CometKiwi delta for 90% accuracy, >1.3 BLEU for 70%)
• Minimum sample: At least 500 segments, or document the confidence interval explicitly.
• Human-vote margin: System A wins if it receives more than 60% of S×S RR votes in the sample.
• Adjudication policy: All critical-severity MQM errors in the losing system must be logged, regardless of the overall vote.

The metric-delta-to-accuracy mapping uses fitted sigmoid curves to convert a raw delta into an estimated probability that a human annotator would prefer the higher-scoring system.

Practical threshold: To reach 80% estimated pairwise accuracy, you typically need a COMET-family delta of roughly 0.5 or a CometKiwi QE 22 delta of roughly 0.7. BLEU requires a substantially larger delta to reach the same confidence level.

What tools and file formats support bulk comparison workflows?

File formats to prioritize:

• XLIFF for segment-aligned bilingual exports from any major TMS or CAT tool; preserves inline tags and placeholders.
• TMX for translation memory exchange and batch re-scoring against updated references.
• JSON and CSV for segment IDs, metric scores, and flag columns in pipeline scripts.

Metric runners and integrations:

• sacreBLEU and chrF scripts (pip-installable, reproducible with explicit tokenization flags).
• COMET and BLEURT Python wrappers for supervised metric scoring; containerize for reproducible runs.
• CometKiwi QE 22 via the Unbabel/COMET library for reference-free scoring at scale.
• TMS/CAT export hooks for XLIFF round-trips; most enterprise TMS platforms support XLIFF 2.0.

For scale, prefer API-first metric runners and containerized environments with pinned model versions. Log sacreBLEU parameters (tokenizer, lowercase flag, reference count) in every run so results are reproducible across teams. For non-Latin scripts, verify tokenization settings before running BLEU; character-level chrF is more robust for languages like Chinese or Arabic. Translation quality also affects downstream multilingual SEO content performance, so consistent metric logging across language pairs pays dividends beyond QA.

How do you recruit and train annotators for MQM and S×S tasks?

Annotator quality is the single largest source of variance in human evaluation. A well-designed recruitment and calibration process cuts that variance before it contaminates results.

Recruitment and training steps:

• Recruit bilingual subject-matter experts, not generalist translators, for technical, legal, or medical content.
• Run a structured training module: annotation exercises on pre-labeled segments, followed by a calibration session where annotators discuss disagreements.
• Provide a gold-standard set of 50 to 100 adjudicated segments for ongoing accuracy checks.
• Measure inter-annotator agreement using Cohen’s kappa or pairwise accuracy against the gold set before scaling to the full batch.
• S×S setups typically show higher agreement than point-wise MQM because direct comparison reduces individual scoring drift.

Pro Tip: Run a triply annotated pilot batch of 50 to 100 segments before committing to the full annotation run. If S×S disagreement concentrates on style rather than accuracy or terminology errors, tighten the rubric’s style-severity definitions before scaling. Catching rubric gaps at 100 segments costs far less than re-annotating 2,000.

What belongs in the comparison report, and how do you act on it?

A comparison report is only useful if stakeholders can read it and engineers can act on it.

Minimum report elements:

• Scope: language pairs, content types, file formats, and system versions compared.
• Sample size and sampling strategy, with confidence interval or variance note if under 500 segments.
• Metrics run, with versions and parameter settings (sacreBLEU flags, COMET model version).
• Metric deltas per system pair, with estimated pairwise accuracy for each delta.
• Human-vote tallies (S×S RR) and MQM error counts by category and severity.
• Recommended action: promote System A, re-evaluate, or escalate to full MQM audit.

Export MQM error spans as tagged XLIFF or JSON and route critical-severity errors directly to remediation tickets. Prioritize fixes by severity first, then by volume: a single critical terminology error in a legal clause outranks fifty minor fluency issues in UI strings. Bilingual side-by-side review catches omissions and meaning shifts that automated checks miss, so include at least one human-reviewed sample in every report for high-stakes content.

Pro Tip: Append a one-page delta-to-accuracy mapping table to every report, so a non-technical stakeholder can read a raw metric delta as a confidence percentage without asking.

What are the most common mistakes when comparing translations?

• Over-relying on BLEU for unrelated systems. BLEU penalizes valid paraphrases and correlates poorly across systems with different generation strategies. Use COMET or BLEURT as the primary signal; treat BLEU as a sanity check.
• Small-sample false confidence. A 100-segment comparison with a clean delta still carries wide confidence intervals. Document the interval or increase the sample before reporting a winner.
• Ignoring inline tags and placeholders. A translation that drops {product_name} or <b> tags scores well on fluency metrics but breaks the product. Pre-filter for tag integrity before running any metric.
• Conflating fluency with fidelity. A fluent translation can omit a key clause. MQM’s accuracy category and targeted S×S checks on high-risk fields catch this; aggregate metric scores do not.
• Skipping numeric and legal field checks. Dates, prices, and legal thresholds require a dedicated pass. A cross-language comparison table that enumerates concrete fields spots inconsistencies invisible to single-language review.

The case for S×S human review as the final arbiter

The conventional wisdom in localization QA is to trust the metric with the highest reported correlation. That instinct is understandable but incomplete. Metrics measure what they were trained to measure, and no single metric captures the full range of errors that matter in production: a dropped legal clause, a culturally inappropriate register, a numeric transposition.

The more defensible approach is to use automated metrics for what they do well, which is fast, cheap, large-scale pre-screening, and then reserve human judgment for the decisions that actually carry risk. S×S annotation is the right human step because it forces direct comparison, reduces individual scoring drift, and produces agreement levels that point-wise annotation cannot match.

The metric-delta-to-accuracy mapping is equally important. Converting a raw COMET delta into an estimated pairwise accuracy percentage turns an opaque number into a decision threshold.

How Lara Translate fits into your comparison pipeline

Running a defensible translation comparison requires fast automated scoring, clean bilingual exports, and a coordinated human review step. Lara Translate covers all three within a single platform.

Relevant capabilities for your comparison workflow:

• API for batch translation and scoring across 200+ languages and 61 document formats, including XLIFF, TMX, and JSON exports for direct pipeline integration.
• AI plus human validation that coordinates professional linguist review on the same segments your automated metrics flagged, so S×S human review is a built-in step rather than an external handoff.
• Glossaries and translation memory that carry approved terminology across comparison runs, reducing noise from inconsistent term choices.
• Incognito mode and enterprise-grade security for sensitive legal, medical, or financial content that cannot leave a controlled environment.

Teams that need a quick start for English-centric comparisons can try Lara Translate’s translation workflow immediately. For enterprise integrations with built-in human S×S review, the AI + Human Translation page covers the full service model and available trial options.

FAQ

What is the fastest way to compare translations automatically?

Run sacreBLEU and CometKiwi QE 22 in batch against your segment pairs. CometKiwi QE 22 requires no reference translation and reaches high pairwise accuracy at smaller deltas than BLEU, making it the faster path to a defensible automated signal.

How many segments do you need to compare translations reliably?

Aim for at least 500 segments. Below that threshold, metric variance is high enough that the same delta can flip direction with a different random sample, so treat smaller runs as directional only.

When does side-by-side human review outperform automated metrics?

Side-by-side human review catches omissions, meaning shifts, and cultural register errors that no metric reliably detects. It is the required step for legal, medical, or safety-critical content, and for any comparison where automated metric deltas fall below your target pairwise accuracy threshold.

What is MQM and why does it matter for translation comparison?

MQM (Multidimensional Quality Metrics) is a structured annotation framework that categorizes errors by type (accuracy, fluency, terminology) and severity (minor, major, critical). It produces weighted error counts that support root-cause analysis and direct remediation, unlike aggregate metric scores.

Can Lara Translate support a full comparison workflow?

Lara Translate provides API-based batch translation, XLIFF/TMX/JSON exports for pipeline integration, and an AI plus human validation service that coordinates professional linguist review on flagged segments, covering both the automated and human steps of a comparison workflow.

This article is about

How to compare translations reliably: a two-stage method that pre-screens segment pairs with automated metrics (BLEU, chrF, COMET, CometKiwi QE 22), maps metric deltas to estimated pairwise human accuracy, and validates meaningful differences with side-by-side MQM human review, including sample-size guidance, decision rules, tooling, annotator training, and where Lara Translate’s API and AI plus human validation fit.

Sources

• Enhancing Human Evaluation in Machine Translation with Comparative Judgement – ACL Anthology
• A Survey on Evaluation Metrics for Machine Translation
• Translation Quality Assurance Side-by-Side Expert Review – Algoran

Recommended

• AI + Human Translation | Lara
• Lara Translate AI: Reliable, Fast, Free
• Lara Translate AI: Reliable, Fast, Free
• Translate to and from English with Lara Translate