Statistical NLP: Lecture 6

Corpus-Based Work

(Ch 4)

Corpus-Based Work

• Text Corpora are usually big.– Corpora 사용의 중요한 한계점으로 작용– 대용량 Computer 의 발전으로 극복

• Corpus-Based word involves collection a large number of counts from corpora that need to be access quickly

• There exists some software for processing corpora

Corpora

• Linguistically mark-up or not • Representative sample of the population of interest

– American English vs. British English– Written vs. Spoken– Areas

• The performance of a system depends heavily on– the entropy – Text categorization

• Balanced corpus vs. all text available

Software

• Software– Text editor : 글자 그대로 보여준다 .– Regular expression : 정확한 patter 을 찾게 한

다 .– Programming language

• C/C++, Perl, awk, Python, Prolog, Java– Programming techniques

Looking at Text

• Text come a row format or marked up.• Markup

– A term is used for putting code of some sort into a computer file

– Commercial word processing : WYSIWYG• Features of text in human languages

– 자연어 처리의 어려운 점

Low-Level Formatting Issues

• Junk formatting/Content.– document headers and separators, typesetter codes, table

and diagrams, garbled data in the computer file.– OCR : If your program is meant to deal with only

connected English text

• Uppercase and Lowercase: – should we keep the case or not? The, the and THE

should all be treated the same but “brown” in “George Brown” and “brown dog” should be treated separately.

Tokenization: What is a Word?(1)

• Tokenization– To divide the input text into unit called token– what is a word?

• graphic word (Kucera and Francis. 1967)

“a string of contiguous alphanumeric characters with space on either side;may include hyphens and apo-strophes, but no other punctuation marks”

Tokenization: What is a Word?(2)

• Period– 문자의 끝을 나타내는 의미가 있다 .– 약어를 나타낸다 . : as in etc. or Wash

• Single apostrophes– isn’t, I’ll 2 words ? 1 words – 영어의 축약 : I’ll or isn’t

• Hyphenation– 일반적으로 인쇄상 다음 줄로 넘어가는 한 단어를 표시– text-based, co-operation, e-mail, A-1-plus paper, “take-it-

or-leave-it”, the 90-cent-an-hour raise, mark up mark-up mark(ed) up

Tokenization: What is a Word?(3)

• Word Segmentation in other languages: no whitespace ==> words segmentation is hard

• whitespace not indicating a word break.– New York, data base– the New York-New Haven railroad

• 명확한 의미의 정보가 다양한 형태로 존재한다 .– +45 43 48 60 60, (202) 522-2230, 33 1 34 43 32 26,

(44.171) 830 1007

Tokenization: What is a Word?(4)

Phone number Country Phone number Country

0171 378 0647 UK +45 43 60 60 Denmark

(44.171) 830 1007 UK 95-51-279648 Pakistan

+44 (0) 1225 753678 UK +411/284 3797 Switzerland

01256 468551 UK (94-1) 866854 Sri Lanka

(202) 522-2330 USA +49 69 136-2 98 05 Germany

1-925-225-3000 USA 33 1 34 43 32 26 France

212.995.5402 USA ++31-20-5200161 The Nerherlands

Table 4.2 Different formats for telephone numbers appearing in an issue of

the Economist

Morphology

• Stemming: Strips off affixes.– sit, sits, sat

• Lemmatization: transforms into base form (lemma, lexeme)– Disambiguation

• Not always helpful in English (from an IR point of view) which has very little morphology.

• IR community has shown that doing stemming does not help the performance

• Mutiple words a morpheme ???• Morphological analysis 를 구현하기 위한 추가비용에 비해

효능이 안 좋다

Stemming

• 동일 의 단어의 다양한 변형을 하나의 색인어로 변환– “computer”, “computing” 등을 “ compute” 로 변환

• 장점– 저장 공간의 사용을 감소 , 검색 속도 개선– 검색 결과의 질 향상 ( 질의가 “ compute” 일 경우

“ computer”, “computing” 등 포함 하는 모든 단어 검색 )

• 단점– Over Stemming: 문자를 과도하게 제거하여 연관성

없는 단어들의 매칭을 발생– Under Stemming : 단어에 포함된 문자를 적게 제거하여

연관성 있는 단어 매칭이 안 되는 현상

Porter Stemming Algorithm

• 가장 널리 사용되며 , 다양한 규칙을 이용• 접두사는 제거하지 않고 접미사만을 제거하거나 , 새로운

String 으로 대치

– Porter Stemming 실행 전

– Porter Stemming 실행 후

Porter Stemming Algorithm

Porter Stemming Algorithm

• Error #1: Words ending with “yed” and “ying” and having different meanings may end up with – Dying -> dy (impregnate with dye) – Dyed -> dy (passes away)

• Error #2: The removal of “ic” or “ical” from words having m=2 and ending with a series of consonant, vowel, consonant, vowel, such as generic, politic…: – Political -> polit – Politic -> polit – Polite -> polit

Sentences

• What is a sentence?– Something ending with a ‘.’, ‘?’ or ‘!’. True in 90% of the cases.– Colon, semicolon, dash 도 문장으로 여겨질 수 있다 .

• Sometimes, however, sentences are split up by other punctuation marks or quotes.

• Often, solutions involve heuristic methods. However, these solutions are hand-coded. Some effort to automate the sentenceboundary process have also been done.

• 우리말은 더욱 어려움 !!!– 마침표가 없기도 하고 종결형 어미 뒤 ?– 연결형 어미이면서 종결형 어미– 따옴표

End-of-Sentence Detection (I)

• Place EOS after all . ? ! (maybe ;:-)

• Move EOS after quotation marks, if any

• Disqualify a period boundary if:

– Preceeded by known abbreviation followed by

upper case letter, not normally sentence-final:

e.g., Prof. vs. Mr.

End-of-Sentence Detection (II)

– Precedeed by a known abbreviation not

followed by upper case: e.g., Jr. etc.

(abbreviation that is sentence-final or medial)

• Disqualify a sentence boundary with ? or !

If followed by a lower case (or a known

name)

• Keep all the rest as EOS

Marked-Up Data I: Mark-up Schemes

• 초기의 markup schemes– 단순히 내용정보만을 위해 header 에 삽입

(giving author, date, title, etc.)

• SGML– 문서의 구조와 문법을 표준화하는 grammer language

• XML– SGML 을 web 에 응용하기 위해 만든 SGML 의 축소판

Marked-Up Data II: Grammaticaltagging

• first step of analysis– 일반적인 문법적 category 로 구별하는 것– 최상급 , 비교급 , 명사의 단수 , 복수 등의 구별

• Tag sets (Table 4.5)– morphological distinction 을 통합한다 .

• The design of a tag set– 분류의 관점

• Word 의 문법정보가 얼마나 유용한 요소인가 하는 관점– 예상의 관점

• 문맥에서 다른 word 에 어떠한 영향을 미치는지 예상하는관점

Examples of Tagset(Korean)

Examples of Tagset(English)

Brown corpus tagsetPennTreebank tagset