Principles of organizing a common morphological tagset and

a search engine for PolUKR (Polish-Ukrainian Parallel Corpus)

Польсько-Український паралельний корпус Polsko-Ukraiński Korpus Równoległy

http://corpus.domeczek.pl

Natalia Kotsyba, Institute of Slavic Studies, Polish Academy of Sciences

Olga Shypnivska, ULIF, Ukrainian Academy of Sciences

Magdalena Turska, Warsaw University

Main objectives and expected applications

• at least 3 mln tokens ; representative• sentence-level alignment• morphological annotation with a common tagset• public access; user-friendly

• linguistic material for – (independent) language learning– bilingual dictionaries– research on grammar and lexis

• translation memory for humans and machines

Statistics (prototype version)

total Polish part Ukrainian part

texts 70 35 35

tokens 359 926 179 087 180 120

characters 3 863 564 1 449 376 2 407 034

Kb 3941 1492 2439

Search (present)

• based on PERL regular expressions• any searched chain has to be “embraced” by “/”.

E.g. /Холодна війна/• special characters:І alternative; ) end of subchain[ i ] beginning and end of a defined character class? 1 or 0 appearances; * 0 or more appearances+ 1 or more appearances\s any empty character\w any letter, digit, underlining sign\b end of word, \ escape

Examples of search formulae

/jako/ „jako”

/jako\s/ „jako, niejako, dwojako”

/\bjako/ „jakość’

/norma\./ „norma” before a dot

Sources of morphological information

• Polish: IPI PAN corpus + …

• Ukrainian:

- grammatical dictionary by ULIF, UAS (Igor Shevchenko) lemma <> wordform

- morphological analyzer (information is slightly different, built for homonymy disambiguation)

- no lemmatization (so far)

Types of tagsets

SYMBOLS: encoding all possible grammatical characteristics of a wordform in one symbol English (BNC), Ukrainian

- takes little machine memory but requires too much of the human one

CHAINS:

contain codes corresponding to particular grammatical categories and/or their values; morphological characteristics of a wordform is represented by a sequence of such codes

can be even more economic than symbols, if a query concerns morphological categories owned by several lexico-grammatical classes

• positional Czech

every category (and its values) have a fixed position in a chain• flexemic Polish, Russian

every category has its own subtagset

Multext-East tagset for En Ro Sl Cz Bg Et Hu Hr Sr Re

• chain-like; criticised• 14 PoS: N10, V15, A12, P(ron)17, Det10, T(he)6,

adveRb6, S(adposition)4, C(onj)7, nuMeral12, Intjn2, X(residual), Yabbr5, Qparticle3

• only Bg and Hu do not have modal verbs and copulas• En Ro have determiners, Ro Hu Re have articles, Bg –

has neither (analitism, segmentation); • Is a Bg noun formally indefinite if the article is attached

to the adj? (cf. agglutinativity of Pl być)• negation as morphological category• Cz transgresivity (adverbial participle)

Treatment of participles• Polish (no aspectual characteristics)

(Here and further cited by: Adam Przepiórkowski i Marcin Woliński A Flexemic Tagset for Polish.)

• Ukrainian (aspect and tense)Дієслово, дієприслівник, доконаний вид, минулий час, активний стан

VW прочитавши

Дієслово, дієприслівник, недоконаний вид, теперішній час, активний стан

UQ читаючи

(Here and further cited by: Широков В.А et al. Корпусна лінгвістика.)

• PolUKRparticiple I (doing/having done) characterised by aspect

Treatment of pronouns

• notorious Slavonic pronoun problem: 296 unique tags for 309 pronouns

• Polish: division into 1-2 p, 3p and siebie (ów, jak?)

• Ukrainian: pro-noun, pro-adjective • Russian: also pro-predicative and pro-adverb• Czech: many subcategories on the level of

SubPoS• PolUKR: Ua approach and Pl division into 1-2

and 3 person

Treatment of predicatives

• Polish: adverbs with modal semantics like można, trzeba (it is) allowed/one can, (it is) necessary, ?to

• Ukrainian (code X0) includes adverbs of state like жарко, шкода, жаль (it is) hot, (it is) a pity

• PolUKR moving the category from the morphological level to the semantic one

http://www.ruscorpora.ru/search-main.html

Search engine for PolUKR• choose the direction of the search (Ua>Pl or Pl<Ua) • search conditions for both languages (RvonW)• 3 levels of search: - exact form - (lemma) with the morphological choice - using Poliqarp-like tag formulas (for advanced users)• idea of subcategories (either a POS or a SUBPOS can be selected,

but not both; similarly, one cannot select all subcategories of a POS), cf. aliases in IPI PAN corpus

• alternative is ensured through tick-off boxes, so that one can choose EITHER „VERB finite past” OR „NOUN dative neutral” OR sth else, etc.)

• restrictions on choice within 1 of 10 POS

VERB infinitive participle I non-finite form finite form

aspect perfective imperfective

mood imperative indicative

person first second third

tense present future past

gender masculine feminine neutral

number singular plural

NOUN general proper name pro-noun 1-2 person pro-noun 3 person

case nominative genetive dative accusative instrumentative locative vocative

gender masculine feminina neutral pluralia tantum

number singular plural

ADJECTIVAL adjective, participle I

and cardinal numeral pro-adjective indeclinable adjective

case nominative genetive dative accusative instrumentative locative

gender masculine feminina neutral

number singular plural

NUMERAL genderic non-genderic

case nominative genetive dative accusative instrumentative locative

gender masculine feminina neutral

ADVERB PARTICLE PROPOSITION CONJUNCTION INTERJECTION

Built-in restrictions on search

category can be selected (active) only if the following category/value(s) have been selected by the

user:

mood OR tense OR person

finite form

gender finite form AND past tense OR adjective AND singular number OR pro-adjective AND singular number OR pro-noun 1-2 person AND singular number OR pro-noun 3 person AND singular number OR ЧИСЛІВНИК родовий AND singular number

gender pluralia tantum NOUN general OR NOUN proper name

case vocative NOUN general OR NOUN proper name

none indeclinable adjective OR ADVERB OR PARTICLE OR PROPOSITION OR CONJUNCTION OR INTERJECTION

Literature

• INTERA unified tagset project www.elda.org/intera• Tomas Erjavec et al. Multext-East specifications for Slavic

languages, Budapest, 2003.• Jan Hajič. Positional Tags: Quick Reference (Czech „HM”

Morphology), 2000.• Adam Przepiórkowski and Marcin Woliński. A Flexemic Tagset for

Polish. In: The Proceedings of the Workshop on Morphological Processing of Slavic Languages, EACL 2003. http://nlp.ipipan.waw.pl/~adamp/Papers/2003-eacl-ws12/ws12.pdf

• Elena Paskaleva. Balcan South-East Corpora Aligned to English. In: The Proceedings of the Workshop on Common Natural Language Processing Paradigm for Balkan Languages, EACL 2003

• Широков В.А et al. Корпусна лінгвістика. Київ: Довіра, 2005.