Tokenizer
Also known as: tokenization, BPE tokenizer.
In plain English
Models don't operate on raw characters or whole words; they operate on tokens, and the tokenizer defines that vocabulary. Most modern tokenizers use byte-pair encoding (BPE): common sequences like 'the' or 'ing' become single tokens, while rare words split into several pieces. Consequences follow directly. English averages about 0.75 words per token, so 1,000 tokens is roughly 750 words. Code, numbers, emoji, and non-Latin scripts (Thai, Chinese, Arabic) tokenize far less efficiently — the same sentence in Thai can cost several times more tokens than in English, which means it costs more and eats more of the context window. Different model families use different tokenizers, so the same text yields different token counts on GPT versus Claude versus Gemini. Understanding tokenization explains a lot of otherwise-mysterious behavior: why costs vary by language, why models miscount characters, and why trimming a prompt saves money.
Example
The word 'unbelievable' is one word but tokenizes into pieces like 'un', 'believ', 'able' — three tokens. 'hello' is a single token; a single emoji can be several. That's why the same 500-word article costs more tokens in Thai than in English, and why an LLM asked 'how many letters are in strawberry?' can miscount — it never saw individual letters, only tokens. The tokenizer is the invisible layer that shapes both cost and quirks.
Tokenizer in Vincony
Because tokenizers differ across providers, the same prompt costs different token counts on GPT, Claude, and Gemini. Vincony's credit pricing normalizes that into one predictable unit, and the Credits Calculator estimates what your typical prompts will cost.
Estimate your creditsTry it — 750+ distinct models across 80+ providers on one account
Vincony bundles GPT-5, Claude, Gemini, Perplexity Sonar Pro, DeepSeek, Mistral, and 750+ other models on one $0/month account.