The Possibility of the Application of Neural Test Theory to SLA

Research

Kenichi YAMAKAWA*1, Naoki, SUGINO*2, Yuko SHIMIZU*2, Hiromasa OHBA*3, & Michiko NAKANO*4

 

*1: Yasuda Women’s University, *2: Ritsumeikan University, *3: Joetsu University of Education, *4: Waseda University

AAAL 2010 Annual Conference: March 6, 2010, Sheraton Atlanta Hotel, Atlanta, GA*The present study is supported by Grants-in-Aid for Scientific Research 1348064 (2001-2003), 16320078 (2004-2006) and 19320091 (2007-2009) from the Japan Society for Promotion of Science.

 

This poster explores a possibility of using Neural Test Theory in SLA research. A grammaticality judgment task aiming to explore the acquisition of unaccusative and unergative verbs was administered to 369 Japanese EFL learners. It was found that the learners could be categorized into four latent ranks.

Abstract summary

Continuous ability scale

Based on http://antlers.rd.dnc.ac.jp/~shojima/ntt/index.htm

•Tests do not have enough resolution to continuously evaluate human ability. Tests are at best capable of ranking test takers into 5 to 20 ranks.

• It is difficult to clarify the relationship between scores on a continuous scale and ability specification.

87 pts

85pts

Problems ！

30pts

Test scores

Students’ language ability on　 a continuous scale (= test scores)

NTT is a test theory designed for evaluating achievements on an ordinal scale. Each level is called a “latent rank”.

Latent ranks are based on the response patterns of the test takers. 

The number of latent ranks is determined by the data analyst.  Goodness-of-fit indices are used in this determination. (Note 1)

What is NTT (Neural Test Theory) ?

Based on http://antlers.rd.dnc.ac.jp/~shojima/ntt/index.htm

Ordinal scale based on NTT

Based on http://antlers.rd.dnc.ac.jp/~shojima/ntt/index.htm

Latent rank 1

Latent rank 2

Latent rank 5

Test scores

• It is easy to explain the relationship between scores and abilities because the individual abilities also change in stages.

• This increases the test’s accountability, and helps to draw a can-do chart.Ability to do X1

Ability to do X2

Ability to do X3

Ability to do X4

Ability to do X5

To explore a possibility of using NTT in SLA research (Note 2) To divide learners into different latent ranks and to attempt to draw a can- do chart

1) Subjects --- university-level 369 Japanese EFL learners 2) Materials --- a grammaticality judgment task on unaccusative /

unergative verbs (Note 3) (36 items: see below) (Note 4) 3) Procedures --- judgments made on a 5-point scale (1-5) converted

into 1-4 points

if ungrammatical if grammatical                0 point      <---    5 (definitely possible)      ---> 4 points                1 point      <---    4 (probably possible)        ---> 3 points                2 points    <---    3 (not sure)                          --->   2 points                3 points    <---    2 (probably impossible)   --->    1 point                4 points    <---    1 (definitely impossible)   --->    0 point

Method

Purpose

Unaccusative verbs

Unaccusative verbs (appear, arrive, die, exist, fall, happen)

Category A (NP+V: 6 items)        e.g.   Your letter arrived yesterday.

Category C (*NP+be+p.p.: 6 items)        e.g.  *Because of the rain, the train was arrived late.

Category E (*NP+V+NP: 6 items)        e.g.  *Finally the waitress arrived the salad to us.

Unergative verbs (cry, dance, laugh, play, sing, work)

Category B (NP+V: 6 items)        e.g.   Her father cried at her wedding ceremony.

Category D (*NP+be+p.p.: 6 items)        e.g.  *He was cried when he heard of his mother’s death.

Category F (*NP+V+NP: 6 items)        e.g.  *The boy hit his little sister and cried her.

Six categories (A-F) in the judgment task

The learners were best distinguished    on an ordinal scale based on NTT    when they were ranked into four levels.

Characteristics of each latent rank were described with regard to their acquisition of unaccusative / unergative verbs.

Some learners in higher ranks seem to regress on an acquisition path when they judge incorrect passive sentences in Categories C and D.

Conclusion

Rank 1

Rank 2

Rank 3

Rank 4

Rank 4: Ability to correctly judge Category D

Rank 3: Ability to correctly judge Categories E and F

Rank 2: Ability to correctly judge Category A

Rank 1: Ability to correctly judge Category B

Rank 4

Rank 3 D) *They were laughed when …Grammatical ???

C) *…,the accident was happened.

• Developed by Dr. Kojiro SHOJIMA and his colleagues• NTT is a nonparametric test theory based on     the self-organizing map (SOM) and generative     topographic map (GTM).• First presented at the International Meeting of the 

Psychometric Society in 2007 (IMPS2007) in Tokyo • The statistical software ： (Shojima, 2008)                 available at http://www.rd.dnc.ac.jp/~shojima/exmk/index.htm

• Selected publications on NTT: 1) About NTT: http://www.rd.dnc.ac.jp/~shojima/ntt/index.htm 2) Shojima, K. (2009). Neural test theory. In K. Shigemasu et al. (Eds.), New Trends in Psychometrics (pp.407- 416). Universal Academy Press. 3) Shojima, K. (2008). Neural test theory: A latent rank theory for analyzing test data. DNC Research Note, 08-

01.

(Note 1) More about NTT

The present joint research has been continuing for eight years with the purpose of exploring the overall development of grammatical competence of Japanese EFL learners. We developed several grammaticality judgment tasks on unaccusative/ unergative verbs, dative alternation, psych verbs, relative clauses, wh-questions, articles, in addition to three sets of standardized test of the English grammar and a test on cue dependency, all of which were conducted on university-level Japanese learners of English. The research findings on each task/test have been published elsewhere (Yamakawa et al., 2003; Ohba et al., 2006; Shimizu et al., 2006; Sugino et al., 2006; Nakano et al., 2007; Ohba et al., 2008).  Based on the data we obtained, we employed IRT (Item Response Theory), and equated the data of 1,200 learners from different tasks/tests (Yamakawa et al., 2008). We obtained the distribution of the average values of all the grammatical categories on the b-parameter (difficulty parameter) scale. To our knowledge, this was a first attempt to apply the IRT over a range of grammatical features, and revealed many characteristics of the development of their grammatical competence. In addition, we also attempted to compare the results of grammaticality judgment tasks (unaccusative/unergative verbs and relative clauses) conducted on Japanese and Korean EFL learners (Ohba et al., 2009; Yamakawa et al., 2009). Although we have obtained substantial information on the order of the difficulty of grammatical categories, we need to explore the relationship between learners’ acquisition stages and their mastery of grammatical categories, which is the focus of the present study. 

(Note 2) Summary of the background of our research

Unaccusat

1) Unacusative verbs ( e.g., happen; break) The accident happened 15 years ago. [ empty [ VP happen the accident ] ] The accident (THEME)

2) Unergative verbs (e.g., dance; cry ) Mary danced. [ Mary [ VP danced] ] Thematic role: Mary (AGENT)

*Most of people are fallen in love and marry with someone. (Japanese L1) (Zobl, 1989: 204)

1) NP movement account: Analogy to the passive *The accident was happened 15 years ago. [ empty [ VP happen the accident ] ]

2) Lexical causativization account: Creation of a temporary causer *The driver happened the accident 15 years ago.

(Note 3) Unaccusative verbs and unergative verbs

The Unaccusative Hypothesis (Perlmutter, 1978, Burzio, 1986)

L2 literature: overpassivization errors of unaccusatives

Two Major Accounts of Passivized Unaccusative Verbs

Our previous studies utilizing ANOVA and IRT also found that the order of difficulty of the categories were B<A<F≦E<D<C. This means that, if used in the same construction, learners always find unaccusatives more difficult than unergatives, and that they start to be able to judge grammatical sentences (NP+V) as “grammatical” first, and then judge ungrammatical ones (NP+V+NP) as “ungrammatical”. Unaccusatives, if used in NP+be+p.p., are the most difficult for them. All the learners in Ranks 1-4 judged in the same order.

Some learners in higher ranks may seem to regress on an acquisition path, exhibiting a U-shaped curve.  Oshita (2001) argues that a U-shaped development can be observed in the acquisition of unaccusative verbs, especially when learners progress from beginners’ level to intermediate level.  He claims that this phenomenon is caused by learners’ reanalysis of unaccusative verbs analogous to the passive formation rule.

 

(Note 5) The order of difficulty of the categories

(Note 6) U-shaped development: Unaccusative Trap Hypothesis (Oshita, 2001)

Nakano, M., Sugino, N., Yamakawa, K., Ohba, H., & Shimizu, Y. (2007). A Study of grammar development among Japanese university students: Intransitive verbs, transitive verbs, ditransitive verbs and logical subjects in Xcomps - Part (1). Proceedings of the 12th Conference of Pan-Pacific Association of Applied Linguistics (pp. 264-267).

Ohba, H., Sugino, N., Nakano, M., Yamakawa, K., Shimizu, Y., & Kimura, S. (2006). The Development of Grammatical Competence of Japanese EFL Learners: Focusing on Relative Clause Constructions. Journal of Pan-Pacific Association of Applied Linguistics, 10, 73–87.

Ohba, H., Sugino, N., Yamakawa, K., Shimizu, Y., & Nakano, M. (2008). Exploring the acquisition of English articles by Japanese EFL learners using on-line tasks. Paper presented at The Third CLS International Conference (National University of Singapore, Singapore).

Ohba, H., Yamakawa, K., Sugino, N., Shimizu, Y., & Nakano, M. (2009). The acquisition of restrictive relative clauses by Japanese and Korean learners of English. Proceedings of the 14th Conference of Pan-Pacific Association of Applied Linguistics (pp. 467-470).

Shimizu, Y., Yamakawa, K., Sugino, N., Ohba, H., & Nakano, M. (2006). Developing a sample-free grammatical proficiency test for SLA research. Proceedings of the 10th Conference of Pan-Pacific Association of Applied Linguistics (pp. 227–235).

Sugino, N., Yamakawa, K., Ohba, H., Nakano, M., & Shimizu, Y. (2006). The acquisition of psych verbs by Japanese EFL learners: From the perspective of cue dependency. Paper presented at the 11th Conference of Pan-Pacific Association of Applied Linguistics (Kangwon University, Korea).

Yamakawa, K., Sugino, N., Kimura, S., Nakano, M., Ohba, H., & Shimizu, Y. (2003). The development of grammatical competence of Japanese EFL learners: Focusing on unaccusative/unergative verb. Annual Re-view of English Language Education in Japan, 14, 1–10.

Yamakawa, K., Sugino, N., Ohba, H., Nakano, M., & Shimizu, Y. (2008). Acquisition of English grammatical features by adult Japanese EFL learners: The application of Item Response Theory in SLA research. Electronic Journal of Foreign Language Teaching (e-FLT), Vol.5, No.1: pp.13-40.

Yamakawa, K., Sugino, N., Shimizu, Y., Nakano, M., & Ohba, H. (2009). The acquisition of unaccusative verbs by Japanese and Korean learners of English. Proceedings of the 14th Conference of Pan-Pacific Association of Applied Linguistics (pp. 9-14).

*The present study is supported by Grants-in-Aid for Scientific Research 1348064 (2001-2003), 16320078 (2004-2006) and 19320091 (2007-2009) from the Japan Society for Promotion of Science.

References

Acknowledgments

*The present study is supported by Grants-in-Aid for Scientific Research 1348064 (2001-2003), 16320078 (2004-2006) and 19320091 (2007-2009) from the Japan Society for Promotion of Science.

Acknowledgments