https://www.chenshaowen.com/en/tags/token/Recent content in Token on Shaowen Chen's WebsiteHugo -- gohugo.ioen©2016 - {year}, All Rights Reserved.Tue, 10 Sep 2024 10:00:00 +0000weeklyhttps://www.chenshaowen.com/en/blog/what-is-token.htmlTue, 10 Sep 2024 10:00:00 +0000Tue, 10 Sep 2024 10:00:00 +0000https://www.chenshaowen.com/en/blog/what-is-token.htmlA token is a unit tightly bound to data. It can be used to measure how much corpus is needed to train a model, and also to measure the input and output length at inference time. 1. What a token is A token can be a whole word, a subword,WeChat Official AccountMachine LearningLLMAITokenLearningNLPInference