Tiktokenizer
Free open-source playground that counts tokens for OpenAI and some open models
Facts checked · 15 sources · Report a correction

What is Tiktokenizer?
Tiktokenizer is a free web playground that shows how a prompt is split into tokens and how many tokens it uses. It runs OpenAI's tiktoken encodings, such as o200k_base for GPT-4o and cl100k_base for GPT-4, plus Hugging Face tokenizers for models like Llama 3, Qwen 2.5 and DeepSeek-R1. You can paste plain text or build a chat from system, user and assistant messages. It was built by developer David Duong (dqbd) and is open source under the MIT license.
Our take
Tiktokenizer is the quickest way to see exactly how an OpenAI-style tokenizer splits a prompt, and the token-ID view makes it useful for debugging, not just counting. It has not been updated since April 2025 and does not cover Claude or Gemini, so use those providers' own token counters for their models.
Who it is for
- Developers estimating prompt size and API cost
- Prompt engineers checking how wording changes token counts
- Students learning how LLM tokenization works
Key features
Live token count
Updates the token total as you type.
Token and ID view
Colors each token in the text and lists the matching token IDs, highlighting pairs on hover.
Chat message builder
Lets you add system, user and assistant messages and counts the special tokens used to wrap them.
OpenAI encodings
Supports gpt2, r50k_base, p50k_base, cl100k_base and o200k_base, plus named models such as GPT-4o and GPT-4.
Open-model tokenizers
Includes Hugging Face tokenizers for Llama 3, CodeLlama, Gemma, Phi-2, Qwen 2.5, DeepSeek-R1, Falcon, Yi and Whisper.
Whitespace view
Shows spaces and line breaks so you can see how they are tokenized.
Runs in the browser
Its code tokenizes locally with tiktoken compiled to WebAssembly and transformers.js.
Tiktokenizer pricing
Tiktokenizer has a free option.
| Plan | Price | What you get |
|---|---|---|
| Free | $0Full app; source code under the MIT license | Full app; source code under the MIT license |
No account or paid tier is offered. The code can be self-hosted from the GitHub repo.
Prices as published by the maker on Nov 21, 2026 · pricing page. Taxes and regional prices may differ.
Pros and cons
What works
- Free, open source and needs no sign-up
- Shows token boundaries and IDs, not just a count
- Covers OpenAI encodings and several open-model tokenizers
- Its source code tokenizes in the browser instead of calling an API
What to watch
- No Claude or Gemini tokenizers
- The model list was last updated in early 2025, so newer models are not listed by name
- Chat counts depend on its own message template, so treat them as an estimate
- No privacy policy and no support beyond GitHub issues
How to use Tiktokenizer
- 1
Open tiktokenizer.vercel.app.
- 2
Pick a model or encoding from the dropdown, for example gpt-4o or cl100k_base.
- 3
Paste text, or for chat models add system, user and assistant messages.
- 4
Read the token count and hover over colored tokens to see their IDs.
- 5
Turn on Show whitespace to see how spaces and line breaks are split.
Is Tiktokenizer safe?
No privacy policy or terms page. The MIT-licensed source code loads tiktoken as WebAssembly and Hugging Face tokenizers through transformers.js and tokenizes text in the browser without calling an encode API.
Tiktokenizer alternatives
Also worth a look
- OpenAI Tokenizer OpenAI's own web page for counting tokens in text.
- tiktoken OpenAI's open-source BPE tokenizer library for counting tokens in code.
- The Tokenizer Playground Hugging Face Space for trying different model tokenizers on your text.
More in Developer & LLM Tools
Tiktokenizer FAQ
What is Tiktokenizer?
A free web playground for OpenAI's tiktoken library. Paste a prompt and it shows the token count, how the text is split and each token's ID.
Is Tiktokenizer free?
Yes. It is open source under the MIT license and has no account or paid plan.
Is Tiktokenizer accurate?
For plain text it uses the same encodings as OpenAI's tiktoken library, so counts match that library. Chat counts add special tokens from its own message template, so treat them as an estimate.
Does Tiktokenizer work for Claude or Gemini?
No. It covers OpenAI models and encodings plus open models such as Llama 3, Qwen 2.5 and DeepSeek-R1, but not Claude or Gemini.
Does Tiktokenizer save my text?
Its public source code tokenizes text in your browser with WebAssembly and transformers.js rather than sending it to an API. The site has no privacy policy.
Who made Tiktokenizer?
Developer David Duong (GitHub: dqbd), who also maintains a JavaScript and WebAssembly port of tiktoken.
Sources
- tiktokenizer.vercel.app/
- github.com/dqbd/tiktokenizer
- api.github.com/repos/dqbd/tiktokenizer
- github.com/dqbd/tiktokenizer/commits/master
- raw.githubusercontent.com/dqbd/tiktokenizer/master/src/models/index.ts
- raw.githubusercontent.com/dqbd/tiktokenizer/master/src/models/tokenizer.ts
- raw.githubusercontent.com/dqbd/tiktokenizer/master/src/pages/index.tsx
- raw.githubusercontent.com/dqbd/tiktokenizer/master/src/sections/TokenViewer.tsx
- raw.githubusercontent.com/dqbd/tiktokenizer/master/src/sections/ChatGPTEditor.tsx
- api.github.com/repos/dqbd/tiktoken
- duong.dev/
- raw.githubusercontent.com/openai/openai-cookbook/main/examples/How_to_count_tokens_with_tiktoken.ipynb
- platform.openai.com/tokenizer
- github.com/openai/tiktoken
- huggingface.co/spaces/Xenova/the-tokenizer-playground