言語処理学会 第23回年次大会 発表論文集 (2017年3月)

Kotonoha: An Example Sentence BasedSpaced Repetition System

Arseny Tolmachev Sadao KurohashiKyoto University

arseny@nlp.ist.i.kyoto-u.ac.jp, kuro@i.kyoto-u.ac.jp

1 IntroductionWord learning is one of the most time consumingparts of language learning, especially when master-ing languages to advanced level. For example, N1,the most difficult level of Japanese Language Pro-ficiency Test requires knowing about 10000 words.Furthermore, learners need not only know the wordsindependently, they also need how words are used incontext: together with other words.

柿

A flashcard for the word 柿(answer card)

Question for the reading card

User is expected to remember

the writing and meaning.

かき(japanese) persimmon

隣の客はよく柿食う客だ。

かき

A client next table eats

persimmons often.

Figure 1: Flashcards for the word “柿”

Word learning is often done using flashcards – away of organizing information into question-answerpairs. An example of a flashcard for the word “柿” isshown on the Figure 1. An answer card has severalfields: kanji writing, reading, definition and referenceexample sentence. A question card usually has onlyone of those fields. A student is expected to remem-ber remaining information. We denote a flashcardthat has only the specified field for a question as a<field> card. For example, we call the card on theFigure 1 a reading card because the question containsonly reading field.

Flashcard systems frequently use Spaced Repeti-tion technique to optimize learning process. Thetechnique is based on observation that people tendto remember things more effectively if they studyin short periods spread over time (spaced repetitionpractice) opposed to massed practice (i.e. cramming)

[5, 2]. Anki1 is probably a most widely known opensource Space Repetition System (SRS). One of draw-backs is that a learner has to create all flashcards tolearn. There exist decks created by community ofvarious quality, however they are not as effective asmanually created ones. The more serious problem islack of context in flashcards. Cards usually have theform like displayed on Figure 1. Namely, the ques-tion card has no context information. Even if thereis an example sentence present, it is static (it doesnot change from repetition to repetition) and can notshow the full spectrum of word usage.

We have developed a Kotonoha2 Spaced Repeti-tion system which uses automatically extracted ex-ample sentences to form flashcard questions. Itssources are available on github3. Kotonoha showsa new example sentence on each flashcard repeti-tion. Also, it makes adding and keeping flashcarddeck process easy by requiring learners to input onlywriting information of words. Kotonoha adds lexicalinformation like meaning, pronunciations and glossesfrom dictionary automatically. Example sentencesare attached without any learner intervention as well.

Kotonoha is a useful system for learning Japaneselanguage. However, it is also designed to be avery useful source of research data. For example,Kotonoha makes it possible to evaluate effect of ex-ample sentences on learning words of a foreign lan-guage. Usually, methods of evaluating quality of ex-ample sentences are often subjective (researchers asklearners of their opinion) or time consuming (need toperform tests on large number of participants). Incontrast to that, a spaced repetition system can beused to directly measure whether example sentencesare useful for remembering words or not without fre-quent manual intervention from researchers.

1http://ankisrs.net2https://kotonoha.ws3https://github.com/kotonoha/server

Copyright(C) 2017 The Association for Natural Language Processing. All Rights Reserved.　　　 　　 　　 　　　 　　　　　　　　　　― 847 ―

2 Spaced Repetition

隣の客はよくかき食う客だ。

柿かき(japanese) persimmon

隣の客はよく柿食う客だ。A client next table eats

persimmons often.

柿かき(japanese) persimmon

隣の客はよく柿食う客だ。A client next table eats

persimmons often.

1. A card

is shown

Card 1

Question

Card

Answer

Card

Time

2. Learner

asks to show

the answer

3. Learner rates

his/her confidence

with the card

4. Learner asks

to show next card ...

酒にはしりたい気分だな。

Confidence:

5 (Excellent)

Card 2

Figure 2: Learning process with flashcards

Spaced repetition is based on spacing effect. It isan phenomenon first documented by Ebbinghaus in1885. People tend to remember things more effec-tively in long term if they use short study periodswhich are spread over time. Moreover, it was shownthat if spacing between practices increases, peopleremember things even better [3].

Learning process using spaced repetition is donein short sessions. In each session a student reviewsseveral flashcards. A flashcard can be shown to userfrom two sources: either it is a new one, so it wasnever reviewed before, or it is an old one, so it wasreviewed before. Old cards are shown only after acertain period of time passes from the previous repe-tition. The period of time is computed based on cardrepetition history and algorithm-dependent. For abrief review of scheduling algorithms for spaced rep-etition systems please refer to the paper [6].

A repetition process sketch for a single flashcard isshown on the Figure 2. The process consists of threerepeated steps. In each step, a student:

1. Views a question card

2. Tries to remember the answer

3. Views card’s answer

4. Evaluates own confidence in the answer

5. Proceeds to the next card

After the repetition, each card is scheduled basedon the confidence score reported by learner. Cardswith high confidence are scheduled after intervalswhich are exponentially increasing. In contrast,cards with low confidence are scheduled after a shortinterval.

3 Kotonoha SRSKotonoha is a web-based open-source Spaced Repe-tition System. It’s main distinctive feature is that it

uses example sentences for reading and writing ques-tion cards. Kotonoha uses algorithm for Spaced Rep-etition from SuperMemo family [9] similarly to Anki.

Figure 3: Screenshot of reading card for the word“盗む” in Kotonoha SRS

Figure 3 shows question and answer sides of a read-ing card for the word “盗む”. Because it is a readingcard, the question, which is an example sentence, hasthe target word written in hiragana. Hiragana read-ing is estimated using Kytea analyzer [4]. Answercards for verbs has information about polite and -teform verb inflections which can be exceptions fromusual rules. For example,帰る having帰ります politeform while 変える has 変えます.

Figure 4: Fragment of word add form of Kotonoha

Process of adding new words to Kotonoha is de-signed to be batch by default. For example, in Ankiusers have to create cards one by one. In contrast,Kotonoha lets learners to input a list of words oneper line. Figure 4 shows word add form of Kotonohawith three words to add. Screenshot shows that thelearner has not added 耽る or 漆黒 before, but hasalready added 眠る. For the words, which are al-ready added, Kotonoha give the learner an optionto mark the word as forgotten (down arrow in thetop-right of word card). In this case, the word willbe scheduled earlier than spaced repetition algorithmhad computed. Dictionary information for words isfilled from JMDict [1]. It is a Japanese — multi-language dictionary. Examples on screenshots haveglosses in English and Russian.

Copyright(C) 2017 The Association for Natural Language Processing. All Rights Reserved.　　　 　　 　　 　　　 　　　　　　　　　　― 848 ―

Figure 5: Word approval process

After adding words, a learner goes over the listonce more, approving them for learning. Approvalprocess is shown on the Figure 5. In this step, thelearner can edit flashcard, assign tags or assign adifferent dictionary entry for the word if there areseveral ones with the same writing. Kotonoha alsoshows several recommended words containing samekanji characters which were not yet added by thelearner, so those words could be easily added forlearning as well.

Figure 6: Writing practice output

In the learning process there could be words diffi-cult to remember for the first time. Kotonoha sup-ports outputting such words in a form suitable forwriting practice with stroke order. Figure 6 showsseveral words in such form.

All these features makes Kotonoha a useful andinteresting system for learning Japanese words.

4 Example SentencesExample sentences for Kotonoha are added to cardscompletely automatically. They come from a webcorpus which contains around 700M sentences. Ex-traction process is described in the thesis [7]. Theprocess is done in two steps. In the first step, the sys-tem creates a search query based on the target wordand its part of speech and makes search request to

DepFinder [8] selecting up to 10k example sentencecandidates. The second step selects sentences whichare non-similar to each other, representative, and thetarget word should be the most difficult one in thesentence.

Example extraction was evaluated by Japaneselearners. Each learner has reviewed example sen-tences by the system and two baselines for 14 words.For each word there was three lists of sentences pre-sented: one for each method. Learners preferred thelist produced by the system in 50% of cases.

Still, the evaluation was subjective and the processwas very time consuming. Participants were select-ing example sentences they like, not ones that wouldbe helpful for remembering the target word.

In contrast to that, Kotonoha can be used to evalu-ate which example sentences were actually helpful forlanguage learning. In spaced repetition participantsreport confidence of card knowledge for each card.We suppose that reported confidence would dependon multiple parameters of last repetition, includingthe shown example sentence. In the case when flash-cards content was the same, with exception of ex-ample sentences, and students were of similar profi-ciency, we hypothesize that confidence scores couldbe used for evaluating whether the example sentencefrom the previous flashcard was helpful for remem-bering the word. Of course, to get the estimationstatistically significant we need to have many learn-ers using the system. Still, we believe that gettinga large number of students to use a spaced repeti-tion system is easier than performing a large scaleexperiment.

5 Limitations and ProblemsExample sentences extraction uses JUMAN dictio-nary, which has different word segmentation com-

Copyright(C) 2017 The Association for Natural Language Processing. All Rights Reserved.　　　 　　 　　 　　　 　　　　　　　　　　― 849 ―

pared to JMDict. Presently, example sentences areextracted for words which have one-to-one correspon-dence in JUMAN and JMDict.

For the test users of Kotonoha, example sentencescould be found for approximately 70% of addedwords.

Words without examples could largely be dividedinto following groups.

• Words which consist of more than one JUMANmorpheme. This group has two subcategories.The first one contains mostly multi-morphemeexpressions which are usually added into dictio-naries with glosses, but are not useful in dictio-naries of morphological analyzers. This groupincludes entries like “癇にさわる”, “年末年始”,“けじめを付ける” or “粗大ごみ”. Handling multi-morpheme expressions of this kind is a futurework for the example extraction.The second one are compound verbs which aredivided into independent morphemes by JU-MAN. The problem with compound verbs isthat it is very difficult to balance semantic con-sistency and usefulness for search engines in onestandard. Compound words which have finergranularity (are split into multiple morphemes)could be handled similarly to the first subcat-egory. Compound words which have roughergranularity require modification of either dictio-nary or tokenization process, which is not triv-ial.

• Words of unsupported parts of speech. In thepresent, example extraction supports only fourparts of speech: nouns, verbs, adjectives and ad-verbs. Words like counter suffixes, while beingvery useful to learners, are not supported yet.

• Words which do not exist in JUMAN dictionary.Those words need to be added into the dictio-nary to be supported.

6 ConclusionThis paper presents a new spaced repetition sys-tem which uses automatically extracted example sen-tences to create question cards. In our best knowl-edge, this is the first spaced repetition applicationwhich automatically assigns example sentences fromhuge corpora to flashcards and shows a new exam-ple sentence for each repetition. Most of other appsuse manually prepared example sentence corpora orshow only a single example sentence for a word.

We theorize that it would be possible to use thesystem as a source of data for objective evaluation ofexample sentence quality. Also, the learning log fromthe system should be useful for general improvementof example sentence quality.

Lastly, example extraction and spaced repetitionmethods are language independent. The same ap-proach could and should be done for other languagesexcept Japanese.

References[1] Jim Breen. JMdict: a Japanese-Multilingual Dic-

tionary. In COLING 2004 Multilingual Linguis-tic Resources, pages 65–72, Geneva, Switzerland,August 2004.

[2] Nicholas J. Cepeda, Harold Pashler, Edward Vul,John T. Wixted, and Doug Rohrer. Distributedpractice in verbal recall tasks: A review andquantitative synthesis. Psychological Bulletin,132(3):354–380, May 2006.

[3] Arthur W. Melton. The situation with re-spect to the spacing of repetitions and memory.Journal of Verbal Learning and Verbal Behavior,9(5):596–606, October 1970.

[4] Graham Neubig, Yosuke Nakata, and ShinsukeMori. Pointwise prediction for robust, adapt-able japanese morphological analysis. In Pro-ceedings of the 49th Annual Meeting of the As-sociation for Computational Linguistics: HumanLanguage Technologies: Volume 2, Short papers,pages 529–533. Association for ComputationalLinguistics, 2011.

[5] Philip I. Pavlik and John R. Anderson. Using amodel to compute the optimal schedule of prac-tice. Journal of Experimental Psychology. Ap-plied, 14(2):101–117, June 2008.

[6] Burr Settles and Brendan Meeder. A TrainableSpaced Repetition Model for Language Learning.In Proceedings of the 54th Annual Meeting of theAssociation for Computational Linguistics (Vol-ume 1: Long Papers), pages 1848–1858, Berlin,Germany, August 2016. Association for Compu-tational Linguistics.

[7] Arseny Tolmachev. Automatic Extraction of Di-verse and High-quality Example Sentences fromLarge Scale Corpora for Language Learning. Mas-ter’s thesis, Kyoto University, Kyoto, 2016.

[8] Arseny Tolmachev, Morta Hajime, and SadaoKurohashi. A grammar and dependency awaresearch system for japanese sentences. In言語処理学会 第 22回年次大会 発表論文集, pages 593–596,仙台, mar 2016.

[9] Piotr A. Wozniak. Optimization of learning. Mas-ter’s thesis, University of Technology in Poznan,Poznan, 1990.

Copyright(C) 2017 The Association for Natural Language Processing. All Rights Reserved.　　　 　　 　　 　　　 　　　　　　　　　　― 850 ―