Paste or drop
Text, or a JSONL, CSV or text file; files are counted per row and in total.
Models
Tokenizer Playground
Token counts decide context limits, cost and fine-tuning length. Paste text or drop a file and see the count for cl100k (GPT-4, GPT-3.5) and o200k (GPT-4o and later) tokenizers, with each token coloured in place so you can see how a script or a code snippet is split, and a cost estimate at a price per million tokens you set.
Loading the tool…
How it works
Read the split
Tokens are shown as coloured segments; hover for the id. Whitespace and Devanagari or Arabic script expand noticeably.
Estimate
Set input and output prices per million tokens and the tool prices a request, a batch or a dataset epoch.
Formats, limits and the numbers that matter
Which tokenizer for which model
cl100k_base for GPT-4 and GPT-3.5, o200k_base for GPT-4o and o-series. Llama, Mistral and Gemma use SentencePiece tokenizers with different counts; a Llama-3 option is planned.
Rules of thumb
English prose is about 1.3 tokens per word; code about 1 token per 3 characters; Hindi and Urdu 3-5x English per word.
Chat overhead
Each message adds a few tokens of formatting; the chat mode adds them so counts match the API.
Common questions
Is the text sent to OpenAI?
No. The tokenizer tables are bundled and run locally.
Why do counts differ from the API by a few tokens?
Message framing tokens and the special tokens the API adds; use chat mode.
Can I count a whole dataset?
Drop the JSONL; the total and per-row histogram are shown.