https://www.chenshaowen.com/en/tags/deep-learning/Recent content in Deep Learning on Shaowen Chen's WebsiteHugo -- gohugo.ioen©2016 - {year}, All Rights Reserved.Sun, 20 Aug 2023 00:00:00 +0000weeklyhttps://www.chenshaowen.com/en/blog/learning-notes-of-transformer.htmlSun, 20 Aug 2023 00:00:00 +0000Sun, 20 Aug 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/learning-notes-of-transformer.html1. Why Transformer Fully connected self-attention In earlier RNN models, each word could only relate to its neighboring words, whereas with the Attention mechanism in the Transformer model, a word can relate to a word at any position, so it can capture global context information. No vanishing gradient problem An RNN applies repeatedly to the same weight matrix, so when its largest eigenvalue is less than 1 the vanishing gradient problem appears.WeChat Official AccountTransformerAILLMLearningDeep LearningNLP