J2450 Translation Quality Metric
The aim of the metric is to maintain “a consistent standard against which the translation quality of automotive service information can be objectively measured,
- regardless of the source language,
- regardless of the target language,
- regardless of how the translation is performed--i.e., human translation or machine translation.”
This is a points based method. The more points a translation receives, the poorer the quality of the translation.
As a brief overview, the metric divides errors into seven categories, each of which is outlined comprehensively
- Wrong Term
- Wrong Meaning
- Omission
- Structural Error
- Misspelling
- Punctuation Error
- Miscellaneous Error
Each category is weighted and some error categories are considered to impinge on quality more than others. A Spelling Error may for example receive fewer points than errors in the Wrong Term category.
When an error is identified as belonging to a category, the reviewer decides whether the error is serious or minor. A serious error is weighted to constitute more points than a minor error.
Evaluation
The metric is easy to follow, easy to implement and is an excellent step towards creating an objective, linguistic quality measure. In addition it is highly customisable and should you feel that a spelling error is more damaging to a translation than an incorrect term, it is easy to change the weighting.
The results of the metrics can then be used for benchmarking linguistic standards and serve as a basis for discussion with both clients and translation groups.
However, there are the following drawbacks:
-
The metric is aimed specifically at automotive service information and is therefore not the most suitable method for evaluating translation where style / voice are key to the overall piece.
-
Results for the reviews have to be collated manually and comparisons can be difficult if you are working directly from spreadsheets.
-
Considerable guidance is needed for reviewers using the metric, particularly on what constitutes a serious or a minor error. The evaluators need to be trained to ensure clear and common understanding. Also, the results from our testing of the metric have shown that two error categories are not sufficient.
-
Once the points have been allocated, there is no benchmark to determine what constitutes a good or bad mark.We carried out a test of the metric by submitting translations of the same text into five different languages to our in-country reviewers. After evaluation we found that the language which scored the second best in points was given a scathing review by the evaluator whereas reviewers for languages that had scored more points were, overall, happy with the translation.
The J2450 Translation Quality Metric can be purchased through the SAE internet site.