Token
토큰
Also known as: tokenizer
The smallest chunk of text a model works with, sometimes shorter and sometimes longer than a word.
In English a word is usually one or two tokens; Korean splits more finely because of particles and endings. The same meaning costs a different number of tokens per language.
Pricing and limits are counted in tokens, so shortening text is the same as lowering cost.
The amount of text a model can hold at once is the context window, also measured in tokens.
- Context windowHow many tokens fit in one pass.
- KoreanThe same content can cost more tokens than in English.