KE
Uploaded byKirill Eremenko
3,774 views
Deep Learning A-Z™: Recurrent Neural Networks (RNN) - The Vanishing Gradient Problem
This document discusses the problems of exploding and vanishing gradients that can occur when training recurrent neural networks. It provides solutions for each problem, such as weight initialization techniques, echo state networks, and LSTM networks. It also references three seminal papers from 1991, 1994, and 2013 that investigated issues with training recurrent networks and long short-term dependencies.
Related topics:
More Related Content
What's hot
Similar to Deep Learning A-Z™: Recurrent Neural Networks (RNN) - The Vanishing Gradient Problem
More from Kirill Eremenko
Deep Learning A-Z™: Recurrent Neural Networks (RNN) - The Vanishing Gradient Problem
- 1.
- 2. © SuperDataScienceDeep LearningA-Z Image Source: people.idsia.ch
- 3. © SuperDataScienceDeep LearningA-Z Image Source: Université Montréal
- 4. © SuperDataScienceDeep LearningA-Z Image Source: recode.net
- 5. © SuperDataScienceDeep LearningA-Z C ŷ C = ½(ŷ- y)2
- 6. © SuperDataScienceDeep LearningA-Z y X2 X1 W11,1 W11,2 W11,3 W12,1 W12,2 W12,3 W21,1 W22,1 W23,1
- 7. © SuperDataScienceDeep LearningA-Z 517 Time 517 517 517 517 Win Wout Wrec Win Wout Wrec Win Wout Wrec Win Wout Wrec Win Wout yt εt xtxt-1xt-2xt-3 Formula Source: Razvan Pascanu et al. (2013) εt+1εt-1εt-2εt-3 Wrec ~ small Wrec ~ large Vanishing Exploding Wrec Wrec Wrec
- 8. © SuperDataScienceDeep LearningA-Z Solutions: 1. Exploding Gradient • Weight Initialization • Echo State Networks • Long Short-Term Memory Networks (LSTMs) • Truncated Backpropagation • Penalties • Gradient Clipping 2. Vanishing Gradient
- 9. © SuperDataScienceDeep LearningA-Z Untersuchungen zu dynamischen neuronalen Netzen By Sepp (Josef) Hochreiter (1991) Link: http://people.idsia.ch/~juergen/SeppHochreiter1991ThesisAdvisorSchmidh uber.pdf Additional Reading:
- 10. © SuperDataScienceDeep LearningA-Z Learning Long-Term Dependencies with Gradient Descent is Difficult By Yoshua Bengio et al. (1994) Link: http://www-dsi.ing.unifi.it/~paolo/ps/tnn-94-gradient.pdf Additional Reading:
- 11. © SuperDataScienceDeep LearningA-Z On the difficulty of training recurrent neural networks By Razvan Pascanu et al. (2013) Link: http://www.jmlr.org/proceedings/papers/v28/pascanu13.pdf Additional Reading: