Skip to content
SaaS Lore

Tiktokenizer

Free open-source playground that counts tokens for OpenAI and some open models

Developer & LLM ToolsOpen sourceWeb

Facts checked · 15 sources · Report a correction

Tiktokenizer home page
tiktokenizer.vercel.app as it looked on Nov 21, 2026.

What is Tiktokenizer?

Tiktokenizer is a free web playground that shows how a prompt is split into tokens and how many tokens it uses. It runs OpenAI's tiktoken encodings, such as o200k_base for GPT-4o and cl100k_base for GPT-4, plus Hugging Face tokenizers for models like Llama 3, Qwen 2.5 and DeepSeek-R1. You can paste plain text or build a chat from system, user and assistant messages. It was built by developer David Duong (dqbd) and is open source under the MIT license.

Our take

Tiktokenizer is the quickest way to see exactly how an OpenAI-style tokenizer splits a prompt, and the token-ID view makes it useful for debugging, not just counting. It has not been updated since April 2025 and does not cover Claude or Gemini, so use those providers' own token counters for their models.

Who it is for

  • Developers estimating prompt size and API cost
  • Prompt engineers checking how wording changes token counts
  • Students learning how LLM tokenization works

Key features

  • Live token count

    Updates the token total as you type.

  • Token and ID view

    Colors each token in the text and lists the matching token IDs, highlighting pairs on hover.

  • Chat message builder

    Lets you add system, user and assistant messages and counts the special tokens used to wrap them.

  • OpenAI encodings

    Supports gpt2, r50k_base, p50k_base, cl100k_base and o200k_base, plus named models such as GPT-4o and GPT-4.

  • Open-model tokenizers

    Includes Hugging Face tokenizers for Llama 3, CodeLlama, Gemma, Phi-2, Qwen 2.5, DeepSeek-R1, Falcon, Yi and Whisper.

  • Whitespace view

    Shows spaces and line breaks so you can see how they are tokenized.

  • Runs in the browser

    Its code tokenizes locally with tiktoken compiled to WebAssembly and transformers.js.

Tiktokenizer pricing

Tiktokenizer has a free option.

PlanPrice
Free$0Full app; source code under the MIT license

No account or paid tier is offered. The code can be self-hosted from the GitHub repo.

Prices as published by the maker on Nov 21, 2026 · pricing page. Taxes and regional prices may differ.

Pros and cons

What works

  • Free, open source and needs no sign-up
  • Shows token boundaries and IDs, not just a count
  • Covers OpenAI encodings and several open-model tokenizers
  • Its source code tokenizes in the browser instead of calling an API

What to watch

  • No Claude or Gemini tokenizers
  • The model list was last updated in early 2025, so newer models are not listed by name
  • Chat counts depend on its own message template, so treat them as an estimate
  • No privacy policy and no support beyond GitHub issues

How to use Tiktokenizer

  1. 1

    Open tiktokenizer.vercel.app.

  2. 2

    Pick a model or encoding from the dropdown, for example gpt-4o or cl100k_base.

  3. 3

    Paste text, or for chat models add system, user and assistant messages.

  4. 4

    Read the token count and hover over colored tokens to see their IDs.

  5. 5

    Turn on Show whitespace to see how spaces and line breaks are split.

Is Tiktokenizer safe?

No privacy policy or terms page. The MIT-licensed source code loads tiktoken as WebAssembly and Hugging Face tokenizers through transformers.js and tokenizes text in the browser without calling an encode API.

Tiktokenizer alternatives

Also worth a look

More in Developer & LLM Tools

Tiktokenizer FAQ

What is Tiktokenizer?

A free web playground for OpenAI's tiktoken library. Paste a prompt and it shows the token count, how the text is split and each token's ID.

Is Tiktokenizer free?

Yes. It is open source under the MIT license and has no account or paid plan.

Is Tiktokenizer accurate?

For plain text it uses the same encodings as OpenAI's tiktoken library, so counts match that library. Chat counts add special tokens from its own message template, so treat them as an estimate.

Does Tiktokenizer work for Claude or Gemini?

No. It covers OpenAI models and encodings plus open models such as Llama 3, Qwen 2.5 and DeepSeek-R1, but not Claude or Gemini.

Does Tiktokenizer save my text?

Its public source code tokenizes text in your browser with WebAssembly and transformers.js rather than sending it to an API. The site has no privacy policy.

Who made Tiktokenizer?

Developer David Duong (GitHub: dqbd), who also maintains a JavaScript and WebAssembly port of tiktoken.

Sources

  1. tiktokenizer.vercel.app/
  2. github.com/dqbd/tiktokenizer
  3. api.github.com/repos/dqbd/tiktokenizer
  4. github.com/dqbd/tiktokenizer/commits/master
  5. raw.githubusercontent.com/dqbd/tiktokenizer/master/src/models/index.ts
  6. raw.githubusercontent.com/dqbd/tiktokenizer/master/src/models/tokenizer.ts
  7. raw.githubusercontent.com/dqbd/tiktokenizer/master/src/pages/index.tsx
  8. raw.githubusercontent.com/dqbd/tiktokenizer/master/src/sections/TokenViewer.tsx
  9. raw.githubusercontent.com/dqbd/tiktokenizer/master/src/sections/ChatGPTEditor.tsx
  10. api.github.com/repos/dqbd/tiktoken
  11. duong.dev/
  12. raw.githubusercontent.com/openai/openai-cookbook/main/examples/How_to_count_tokens_with_tiktoken.ipynb
  13. platform.openai.com/tokenizer
  14. github.com/openai/tiktoken
  15. huggingface.co/spaces/Xenova/the-tokenizer-playground