A token is a unit tightly bound to data. It can be used to measure how much corpus is needed to train a model, and also to measure the input and output length at inference time.
1. What a token is
A token can be a whole word, a subword, or even a single character. In a language model, text is split into a number of tokens, and the model processes them one by one to produce predictions or generate new text.
Here are some rules of thumb that help you gauge how long a token is:
1 token ~= 5-6 chars in English
100 tokens ~= 90 words in English ζ 10 sentences in English
You can use https://platform.openai.com/tokenizer to see how OpenAI’s tokenizer behaves. In the example below, you can see that tokens do not map one-to-one onto the words a human would recognize.
2. The embedding of a token
In the example above, switch the bottom tab to [Token IDs] to see the embedding ids of the tokens corresponding to the text.
Text is for humans to read; Token IDs are for the model to process. After text passes through the tokenizer, it becomes one or more tokens, and after embedding it is converted into Token IDs. The vocabulary stores the embedding id corresponding to each token.
3. How the vocabulary is generated
The vocabulary is usually generated before model training, and it does not change throughout the entire training process.
SentencePiece is a tool for tokenization and vocabulary generation, supporting a variety of tokenization algorithms such as BPE, WordPiece, and unigram. It is widely used in building the vocabulary and tokenizer for large models, for example LLaMa, BLOOM, ChatGLM, and Baichuan.
Using SentencePiece you can generate a complete vocabulary from a corpus, or expand an existing vocabulary.
For example, when LLaMa was trained, very little Chinese material was used. Before fine-tuning the model on a Chinese corpus, you first need to expand the vocabulary, which can effectively improve the model’s performance.
4. The relationship between tokens and data size
Below is a comparison table of a few typical models and their pretraining data scale:
| Model | Parameter scale (B) | Pretraining data scale (tokens) |
|---|---|---|
| CodeGen | 16 | 577 B |
| LLaMA | 65 | 1.4 T |
| GPT-3 | 175 | 300 B |
Here, B stands for Billion and T stands for Trillion.
Using tokens directly as the data scale is sometimes not intuitive enough. Below is a comparison table between tokens and file size:
| Text type | Average bytes per token | Token count of 1GB of text data |
|---|---|---|
| English | 6 bytes | ( \frac{1 \times 2^{30}}{6} β 0.167 B ) |
| Chinese | 3 bytes | ( \frac{1 \times 2^{30}}{3} β 0.556 B ) |
| Text type | Average bytes per token | File size of 1B tokens |
|---|---|---|
| English | 6 bytes | ( 1B \times 6 ) bytes β 5.6 GB |
| Chinese | 3 bytes | ( 1B \times 3 ) bytes β 2.8 GB |
