Click here to load reader

Long Short-term Memory

  • View
    1.253

  • Download
    2

Embed Size (px)

Text of Long Short-term Memory

  1. 1. Long Short-term Memory 2015-07-31
  2. 2. 2
  3. 3. Grid Long Short-term Memory Grid Long Short-term Memory [Kalchbrenner+ 2015] Long Short-term Memory(LSTM) WikipediaNN phrase-based reference system 3 http://arxiv.org/abs/1507.01526
  4. 4. LONG SHORT-TERM MEMORY LSTM LONG SHORT-TERM MEMORY [Hochreiter&Schmidhuber 1997] Recurrent Neural Network LSTM 4 http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf
  5. 5. Recurrent Neural Network NN NN 5
  6. 6. Backpropagation Through Time NN [Williams&Zipser 1992] 6
  7. 7. or 7
  8. 8. Constant Error Carrousel(CEC) 1 f(x) = x 1.0 8 [Hochreiter&Schmidhuber 1997]
  9. 9. : Input Weight Conflict () 9
  10. 10. : Output Weight Conflict Conflict 10
  11. 11. 11
  12. 12. LSTM CEC 12
  13. 13. Memory Cell Block 111 13
  14. 14. Abuse Problem 14
  15. 15. Abuse Problem (Q: A: ) 15
  16. 16. (1) (2) OK 16
  17. 17. Internal state drift ( ) 0 ( ) 17
  18. 18. Forget Gate [Gers+ 1999] Learning to Forget: Continual Prediction with LSTM http://citeseerx.ist.psu.edu/viewdoc/download?do i=10.1.1.55.5709&rep=rep1&type=pdf 18
  19. 19. Forget Gate LSTM [Gers+ 1999] 19
  20. 20. [Hochreiter&Schmidhuber 1997]p.23 ALGORITHM DETAILS Tomonari MASADA* C966 Chainer** 228 RNNLIBC++ 20 * http://diversity-mining.jp/wp/?p=407 ** https://github.com/pfnet/chainer/blob/master/chainer/functions/lstm.py
  21. 21. LSTMN 21
  22. 22. Wxx nn h- 22
  23. 23. ntDn 1CEC 23
  24. 24. wCEC LSTM W 24
  25. 25. 1 1 () 25

Search related