Error Analysis of Rule-based Machine

Translation Outputs

A Case Study on English – Persian MT

System

تجزیه و تحلیل خطا از حکومت مبتنی بر خروجی دستگاه ترجمه؟

مطالعه موردی در زبان انگلیسی - فارسی MTسیستم های

Zahra PourniksefatIslamic Azad University – Science & Research

Branch

Agenda

Introduction Machine Translation Overview Evaluation of MT systems

Methods & Materials Error Categories & DescriptionResults & Discussion

Machine Translation Overview

Definition: The term Machine Translation (MT) is used for translating text or

speech from one natural language to another by using computers and software.

• Systran: MT is much faster than human translators because it is much

cheaper and has a better memory than human translators.

• Shahba 2002 believed that “It’s better to spend our time on the actual act of

translation rather than typing the English text or scanning it for the MT to

translate. Efforts in MT are by themselves valuable as they at least satisfy one of

the needs of human beings: need for innovation and discovery”

• MT is more economic on time and money, but it is less accurate than human

translators (Frederking, 2004).

Why MT matters?

According to Hatim and Munday it’s an important topic - socially, politically,

commercially, scientifically, and intellectually or philosophically (2004)

• The social or political importance of MT arises from the socio- political importance of translation

in communities where more than one language is generally spoken. So translation is necessary for

communication- for ordinary human interaction, and for gathering the information one needs to

play a full part in society.

• The commercial importance of MT is a result of related factors. First, translation itself is

commercially important. Second, translation is expensive.

• Scientifically, MT is interesting, because it is an obvious application and testing ground for many

ideas in Computer Science, Artificial Intelligence, and Linguistics.

• Philosophically , MT is interesting, because it represents an attempt to automate an activity that

can require the full range of human knowledge.

Some Misconceptions about MT

MT is a waste of time because you will never make a machine that can translate

Shakespeare. This criticism that MT systems cannot translate Shakespeare is a bit like

the criticism of industrial robots for not being able to dance.(Hatim and Munday, 2004)

• First, translating literature requires special literary skills – it is not the kind of

thing that the average professional translators normally attempt

• Second, literary translation is a small proportion of the translation that has to

be done.

• Finally, one may wonder who would ever want to translate Shakespeare by

machine – it’s a job that human translators find challenging and rewarding, and

it’s not a job that MT systems have been designed for.

Approaches to MT• Direct Machine Translation Approach

The first developed MT systems where a word–for–word translation from the source language to the target

language is performed.

• Transfer Machine Translation Approach

1. The analysis stage that is the direct strategy which takes benefits of a dictionary in source language to

demonstrate the source language from linguistic point of view.

2. The transfer stage varies the outputs of the analysis stage to produce structural and linguistic equivalents

between the two languages.

3. The generation stage is the third stage in which a target language dictionary is applied to result the target

language document on the basis of linguistic information. (Steiner, 1988)

• Interlingua Machine Translation Approach

First the source text meaning is decoded

Second the resulted meaning is re-encoded in the target language

Approaches to MT cont’d.• Rule-based Machine Translation Approach

It operates on the linguistic data on source and target languages fundamentally

taken from bilingual dictionaries and the basic semantic, morphological, and

syntactic grammar of the individual language (Gelbukh, 2011).

Minimally, to get a Persian translation of English sentence one needs:

1. A dictionary that will map each English word to an appropriate Persian word.

2. Rules representing regular English sentence structure

3. Rules representing regular Persian sentence structure

4. And finally, we need rules according to which one can relate these two

structures together.

Approaches to MT cont’d.

• Statistical Machine Translation Approach

This system uses a corpus or database as a translated example for analyzing and decoding source

language. In comparison with the machine translation of about three decades ago, Google Translate

as an example of more contemporary automated engine for the task of translation has taken a giant

leap. However, it is still too imperfect. (Nierenberg, 1998)

• Hybrid Machine Translation Approach

1. Rules post-processed by statistics in which translation are practiced on the pivot of rule-

based engine. Next statistics are applied to correct the output.

2. Statistics guided by rules in which rules have an important role to pre-process date to

quite the statistical representation to normalize. This approach is powerful, flexible and

under more control when it’s translating.

Evaluation of MT Systems• Human translation assessment (Secară 2005; Williams 2001) has been

moving from microtextual, word- or sentence-level error analysis methods

toward more macrotextual methods focused on the function, purpose and

effect of the text. At the same time, machine translation assessment has

mainly been microtextual and focused on the aspects of accuracy and

fluency.

• Hovy (2002) discussed the complexity of MT evaluation, and stressed the

importance of adjusting evaluation to the purpose and context of the

translation.

Evaluation of MT Systems cont’d.

Mary A. Flangan Believed that Machine translation quality can be difficult to quantify for a

number of reasons:

1) A text can have several different translations, all of which are correct.

2) Defining the boundaries of errors in MT output is often difficult. Errors sometimes involve only single

words, but more often involve phrases, discontinuous expressions, word order or relationships

across sentence boundaries. Therefore, simply counting the number of wrong words in the translation is not

meaningful.

3) One error can lead to another. For example, if the part of speech of a word is identified incorrectly by the

MT software, the entire analysis of the sentence may be affected, creating a chain of errors.

4) The cause of errors in MT output is not always apparent. The evaluator usually does not have access to a

trace of the software's tests and actions. Thus it can be difficult to identify what went wrong in the

translation of a sentence.

Evaluation of MT Systems cont’d.

Types of Evaluation Automatic Evaluation

the Word Error Rate (WER), the Position independent word Error Rate (PER), the

BLEU (Papineni et al., 2002) and the NIST (Doddington, 2002) where the MT

output is compared to one or more human reference translations.

Human Evaluation

Due to the complexity of natural language, manual evaluation seems more reliable

1. Three passages were selected and translated by Rule-based MT Systems

and compared with one Statistical MT System and Human translator

2. Error categories were derived after the analysis of each text

Methods & Materials • Three passages were translated by two different MT systems and also a human

translator. • From each text type a passage of approximately 400 words was taken from story,

user guide and magazine.• The rule-based MT – Arya TM– system was designed based on thousands of lexical

and grammatical rules.• The statistical system, Google Translate by Google Inc., is based on the use of large

monolingual and parallel corpora for translation. • The unit of analysis was set to a sentence level because it’s the largest unit which

can be easily recognized in MT systems and ST sentence can be clearly corresponded to its TT pairs.

Table of Source Text Passages for Analysis  Number of Words Number of Sentences

Short Story The Lottery

 398 

 13

User GuideMicrosoft Access 2012

 390

 16

Magazine Academic article

 415

 15

Errors Category

Syntactic

Word Order

Missing

Words

Punctuation

Parts of

Speech Conjugation

Unknown Words

Semantic

Incorrect Words Polysemy

Idiomatic Expressions

• For English-to-Persian Rule- based MT systems the following categories were

derived

Error Categories & Descriptions

Error Categories & Descriptions cont’d.

Description of Error Categories:

• Syntactic Errors: Those errors that are related to the grammar of the language such as

parts of speech or conjugation

Word order that means sentence elements ordered incorrectly

Example: Commands generally take the form of buttons and lists. (User Guide)

Missing words: incorrect elision of some words

Example: This requires better data collection and analysis tools for studying outcomes and

consistent use of these tools across individual studies. (Magazine)

Arya Translation System و گیرد می شاگرد فرم کلی بطور ها دستور

ها . فهرست

Google Translate . لیست و ها دکمه شکل به کلی طور به دستورات

Arya Translation System ها ابزار و های نیاز بهتر اطالعات مجموعه این

سازگار و ها حاصل کن می مطالعه برای تحلیل

کند . می استفاده

Google Translate ابزار از استفاده با و بهتر ها داده آوری جمع مستلزم امر این

این از مداوم استفاده و نتایج بررسی برای تحلیل و تجزیه

. فردی مطالعات سراسر در ابزار

Error Categories & Descriptions cont’d.

Unknown words: word not in a dictionary

Example: The women, wearing faded house dresses and sweaters, came shortly after their

menfolk.( Story)

Punctuation: incorrect punctuation Example: The children assembled first, of course. (Story)

Arya Translation System محو , ها ژاکت و خانه ها لباس کننده خسته ها زن

از پس زودی به ، آمدند menfolkکردند شان

Google Translate مدت آمد در خانه، پژمرده ژاکت و لباس پوشیدن زنان،

از پس .menfolkکوتاهی را خود

Arya Translations کردند , جمع البته اول ها . بچه

Google Translations البته اول، مونتاژ .کودکان

Error Categories & Descriptions cont’d.

Parts of speech: errors in identifying pars of speech such as noun or verb

Example: If you decrease the width of the ribbon, small button labels disappear. (User Guide)

Conjugation: incorrectly formed verb or wrong tense

Example: Soon the women, standing by their husbands, began to call to their children, and the

children came reluctantly, having to be called four or five times.

Arya Translation System ناپدید کوچک دکمه ، کاهشبیابید نوار پهنا شما اگر

زند می برچسب

Google Translate کوچک دکمه ها برچسب دهد، کاهش را شما نوار عرض اگر

شوند می ناپدید

Arya Translations شروع , , به شان های شوهر کن می حمایت ها زن بزودی

می , ، آمدند اکراه با ها بچه و شان ها بچه به صدا کردن

دوره . پنج یا چهار زده صدا اشد که دارد

Google Translations فرزندان به تماس به شروع ایستاده، خود شوهران زنان، زودی به

. بار پنج یا چهار نام به اکراه، به ها بچه و خود،

Error Categories & Descriptions cont’d.

• Semantic Errors: Those errors that are related to the meaning such as incorrect meaning

of words or expressions which caused the incorrect meaning of the whole sentence.

Incorrect word: completely incorrect meaning

Polysemy: incorrect selection of the meaning of the words with more than one meaning

Example: The people of the village began to gather in the square, between the post office and the bank,

around ten o'clock.

Style and idiomatic expression : incorrect translation of multi-word expression

Example: They greeted one another and exchanged bits of gossip as they went to join their husbands. Arya Translations کردند معاوضه غیبت ذره و همدیگر سالم آنها

شان . های شوهر کنند وصل که رفتند آنها

Google Translations بی شایعات از بیت بدل و رد و یکدیگر استقبال آنها

خود شوهر به پیوستن برای را آنها عنوان به را اساس

رفت.

Arya Translations جمع که کردند شروع روستا مردم

و , پستخانه میان در مربع در شوند

ساعت دsه حدود ، بانکGoogle Translations آوری، جمع به شروع میدان در روستا مردم

ده حدود ساعت بانک، و پست اداره .بین

Results & Discussions

    

RBMT 

 SMT

Human

  Word Order Missing Words Unknown Words

Punctuation Parts of Speech Conjugation

Story  12 6  3 12 8 9

User Guide    17  5 1 10 5 13

Magazine  14   4 10 7 15

Story  11  7 3 12 7 8

User Guide 17 5  1 9 7 11

Magazine  15 2  5 9 6 14

Story  1  2 0  3 1  3 

User Guide 0 0  2 1   0  1

Magazine  2  1 1  1  2  2

Syntactic Category

Word Order Missing Words

Unknown Words Punctuation Parts of

Speech Conjugation

Tabl

e of

Syn

tact

ic E

rror

s

   

RBMT  SMT  Human

  Incorrect Lexicon   Polysemy  Idiomatic Expression  

Story  10 7 12

User Guide    7 9 5

Magazine  7 11 9

Story  8 8 9

User Guide 3 7 3

Magazine  5 13 8

Story   0 0 2 

User Guide  1  0  0

Magazine   0  1  1

Semantic Category

Incorrect Lexicon Polysemy Idiomatic Expression

Tabl

e of

Sem

anti

c E

rror

s

Results & Discussions contd.

Results & Discussions cont’d.

• Both systems made the least errors with the simpler sentences and the most ones with the

compound- complex sentences, as well as lexically or structurally ambiguous texts. This is because

ambiguous source texts with different contents can correspond with more than one representation.

• For the rule-based system, the most typical errors are in conjugation, word order and also in

rendering polysemous words and idiomatic expressions. For the statistical system the most

common error is in conjugating and determining the tense. However, it has also some problems in

translating words with multiple meaning and idiomatic expression.

• To see whether machine translation accuracy is affected by text-type three different genres were

analyzed thoroughly. And for the different text types, the rule- based system had similar amounts

of syntactic and semantic errors in each text.

Future!

• Evaluating MT quality is necessarily a subjective process because it involves

human judgments.

• Determining the best category for an error in MT output is not easy because

we have to place them on how they are realized rather than the cause of errors

and many machine translated sentences contained multiple, linked errors.

• Future work will therefore be focused on the cause of errors and ranking error

categories. The error categories presented here is flexible, allowing for the

deletion or addition of more categories.