1. Why Transformer
- Fully connected self-attention
In earlier RNN models, each word could only relate to its neighboring words, whereas with the Attention mechanism in the Transformer model, a word can relate to a word at any position, so it can capture global context information.
- No vanishing gradient problem
An RNN applies repeatedly to the same weight matrix, so when its largest eigenvalue is less than 1 the vanishing gradient problem appears. The Attention computation in Transformer is a fully connected softmax attention, so gradients can flow back smoothly.
- Parallel computation
Earlier models compute sequentially, whereas Transformer supports parallel computation and can fully exploit the compute capability of the GPU.
2. Transformer in Large Models
A large model contains multiple Transformers; a Transformer contains multiple Encoder Layers and Decoder Layers; and each Encoder Layer and Decoder Layer is composed of several attention layers (Attention Layer) and feed forward layers (Feed Forward Layer).
The BERT model contains an Encoder module, built by stacking multiple Transformer encoders. The BERT-Base model contains 12 identical Transformer Encoder structures, and the BERT-Large model contains 24.
The GPT-3 model contains a Decoder module, built by stacking multiple Transformer decoders. GPT-3 model sizes range from Small to XL, and the number of Transformer blocks in the decoder is 12, 24, 36, and 48 respectively.
3. Structure of Transformer
The input of each Encoder is the output of the Encoder below it, and the output of the topmost Encoder is fed into every Decoder layer.
The input of each Decoder is the output of the Decoder below it, and the output of the last Decoder layer is fed into a linear layer. The output of the linear layer is a vector the size of the vocabulary, where the value at each position represents the probability that the word at that position is the word at the current position.
The Encoder is responsible for extracting features from the input sequence, while the Decoder is the module that generates the output sequence. For the full flow, refer to this animated diagram:
3.1 Encoder Layer
Each Encoder Layer consists of two sublayers: an attention layer (self-Attention Layer) and a feed forward layer (Feed Forward Layer).
- self-Attention Layer
The self-attention layer is used in the encoder to capture the relationships between different positions in the input sequence.
It maps the input vector to the Query, Key, and Value matrices, performs a dot-product attention computation, and obtains a word-level attention representation.
- Feed Forward Layer
It introduces nonlinearity to help the model better learn the features in the input sequence.
3.2 Decoder Layer
Each Decoder Layer consists of three sublayers: an attention layer (self-Attention Layer), an encoder-decoder cross-attention layer (Encoder-Decoder Attention Layer), and a feed forward layer (Feed Forward Layer).
- self-Attention Layer
It computes the relatedness between words in the target sequence and captures internal dependencies.
It is similar to the Self-Attention in the Encoder, but computes attention only from its own input.
- Encoder-Decoder Attention Layer
The encoder-decoder attention layer connects the input and the output in a sequence-to-sequence model. By dynamically adjusting where to attend, it improves model performance, handles long-range dependencies, and enhances generation accuracy.
- Feed Forward Layer
It introduces a nonlinear transformation to further enhance expressiveness.
3.3 Self-Attention Computation
Step one: here X denotes the input; compute the query matrix Q, the key matrix K, and the value matrix V
Where
- Wq is the linear transformation matrix used to generate the query (Query)
- Wk is the linear transformation matrix used to generate the key (Key)
- Wv is the linear transformation matrix used to generate the value (Value)
Wq, Wk, and Wv are model parameters, learned through training.
Step two: compute the attention scores
3.4 Word Embedding Matrix
The Word Embedding matrix is the matrix used to convert word symbol representations into dense word vectors.
In Transformer’s Self-Attention mechanism, the sizes of the Query, Key, and Value matrices are related to the model’s Word Embedding size. Generally, a few hundred to a thousand dimensions is a fairly common setting. Too few dimensions cannot fully express semantic information, while too many dimensions bring a computational burden.
During training, the model updates not only the weight matrices, but also the word vector table (the Word Embedding matrix). By pretraining on large-scale corpora, the model learns the word vector table and the weight matrices of each layer, encoding linguistic knowledge.
During inference, the word vector table is fixed and is no longer updated; it is used to convert the input text into a vector representation. The weight matrices are likewise fixed, and are used to perform the inference computation and produce the final output.
In other words, the training phase updates the Embedding matrix and the weight matrices, whereas in the inference phase both are fixed and unchanging; only the forward computation is performed, and the model parameters are no longer updated.
4. Multi-Head Attention Mechanism
Compared with the single-head attention mechanism, the multi-head attention mechanism lets the model attend to semantic information at different positions simultaneously, thereby improving the model’s expressiveness.
- Number of heads
Common multi-head designs use 8 or 12 heads, and some models use 16 heads. The more heads there are, the finer the granularity each head can attend to, but the computation also increases linearly.
- Differences between heads
Each head has different QKV, so the semantic information it focuses on differs as well β for example, one focuses on the main information while another focuses on background information.
Using the multi-head attention mechanism, multimodal large models can be built to handle different types of data, such as text, images, and audio.
