BPE Tokenizer Visualizer & Token Heatmap

Inspect Byte-Pair Encoding (BPE) sub-word token boundaries, token IDs, and character-to-token compression efficiency.

Total Tokens
39
Total Characters
231
Chars / Token Ratio
5.92
Try Sample:
Hover or click any token pill
You␣are␣a␣helpful␣AI␣software␣architect.␣Analyze␣the␣trade-offs␣between␣PostgreSQL␣connection␣pooling␣with␣PgBouncer␣vs␣native␣connection␣management␣in␣AWS␣RDS␣Aurora␣Serverless␣v2.␣Output␣your␣answer␣in␣structured␣markdown␣tables.

How Byte-Pair Encoding (BPE) Tokenizes Text

Large Language Models (like GPT-4o, Claude 3.5, and Llama 3) do not process raw text or full words directly. Instead, they convert strings into sequences of integer tokens using Byte-Pair Encoding (BPE). Common words become single tokens (e.g. " architect"), while rare words, numbers, code syntax, and emojis are decomposed into sub-word byte fragments.

Key Tokenization Characteristics

  • Whitespace Attachment: In modern BPE tokenizers (like cl100k_base and o200k_base), leading whitespace is merged into the subsequent word token (e.g. " hello" is a different token ID from "hello").
  • Numbers & JSON: Numbers and JSON punctuation ({, :, ") often consume individual tokens, making structured JSON prompts denser than plain prose.
  • Character Compression Ratio: For standard English prose, 1 token roughly equals 4 characters (~0.75 words). For code and non-Latin languages, token density is higher.