AI Token Counter

Tokens, tokenizers, and the math of LLM cost — what every AI engineer needs in their head

"Token" is the unit of work for every modern LLM. You pay per token, you fit prompts into a context window measured in tokens, you get rate-limited per token. Yet most developers using these APIs never look under the hood at how text becomes tokens, why the same prompt costs differently across providers, or what actually drives output cost. This guide walks through the byte-pair-encoding mechanics, the per-model tokenizer differences in 2026, the cost-modeling math you should use for budgets, and the specific patterns (prompt caching, batch APIs, smaller models for routing) that cut bills by 50–90%.

What a "token" actually is

An LLM does not see characters or words; it sees integer IDs. Behind that is a fixed vocabulary — typically 50,000 to 200,000 entries — built once during model training, where each entry maps to a sequence of bytes. Tokenization is the algorithm that splits incoming text into the longest matching pieces from that vocabulary.

Almost every modern LLM uses Byte-Pair Encoding (BPE) or a close variant (SentencePiece, WordPiece). BPE works bottom-up:

1. Start with the byte-level alphabet (256 entries: every possible UTF-8 byte).
2. Find the most frequent adjacent pair in the training corpus; merge it into a new token.
3. Repeat 50,000–200,000 times.

The result: common words become single tokens (" the", " and", " of"), rarer words split into 2–4 sub-tokens ("unbelievable" → "un" + "believable"), and unusual sequences fall back to per-byte tokens (a long random hex string can become 30+ tokens).

The 2026 tokenizer landscape — same text, different counts

Practical impact: the same 1,000-word English article runs about 1,250 tokens on GPT-4o (cl100k_base + o200k upgrades) but ~1,400 tokens on Claude. A 10,000-word document can differ by 1,500+ tokens — meaningful for both cost and context-window planning.

What inflates token count beyond the 4-chars rule

• Code: indentation, punctuation, and identifiers chew through tokens. JavaScript and Python both run roughly 30% denser than English prose.
• JSON: structural {/}/"/,/: punctuation each becomes its own token. JSON is typically 50% more tokens-per-character than equivalent prose.
• Long random strings: UUIDs, hashes, base64 — each character can become its own token. A 36-char UUID is often 25+ tokens.
• Non-English text: older tokenizers (cl100k_base) split CJK characters into 2-3 byte tokens each. Newer tokenizers (o200k_base, Gemini's SentencePiece) handle this better.
• Emoji and Unicode: each emoji is typically 2-4 tokens because they're multi-byte.
• Tool-call schemas: the JSON schema for a function definition is sent every turn. A 500-token tool definition × 10 tools × 20-turn conversation = 100K wasted tokens.

Pricing landscape — January 2026 reference

The 3:1 to 5:1 input-to-output price ratio is consistent across providers. This is because generation is more expensive than prefill: the model has to autoregress one token at a time, while reading input runs in parallel.

Prompt caching — the biggest cost lever in 2026

Both OpenAI and Anthropic now support prompt caching — pay full price the first time you send a long system prompt, then up to 90% off for repeat reads in the next 5 minutes. The math:

• Anthropic: 90% discount on cache reads, 25% premium on cache writes. Break-even after 2 reads. Use it when the same system prompt fires 3+ times per session.
• OpenAI: 50% discount on cached tokens, no write premium, automatic for prompts ≥ 1024 tokens. No opt-in required.
• Bedrock / Vertex: provider-specific; check the AWS/GCP docs.

If your system prompt is 5,000 tokens and you fire 100 conversations against it per hour, caching saves about $1.10/hour on Sonnet 4 (5,000 × 0.9 × $3/M × 100 = $1.35 saved on input).

Batch APIs — 50% off for non-urgent work

Both OpenAI and Anthropic offer Batch APIs at 50% discount for jobs that don't need real-time responses. The API accepts a JSONL file of prompts and returns results within 24 hours. Use cases:

• Embedding large document corpora.
• Bulk classification (10K product descriptions, 1M support tickets).
• Synthetic data generation for fine-tuning.
• Backfilling AI summaries on historical data.

Combined with caching, this can cut bills 60–70% on batch workloads.

Counting tokens precisely — language by language

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode("Hello, world!")
print(len(tokens))   # exact count

import anthropic
client = anthropic.Anthropic()
result = client.messages.count_tokens(
  model="claude-sonnet-4-latest",
  messages=[{"role": "user", "content": "Hello"}]
)
print(result.input_tokens)

import { encoding_for_model } from 'tiktoken';
const enc = encoding_for_model('gpt-4o');
const tokens = enc.encode('Hello, world!');
console.log(tokens.length);
enc.free();   // tiktoken uses WASM, free to release memory

import Anthropic from '@anthropic-ai/sdk';
const client = new Anthropic();
const result = await client.messages.countTokens({
  model: 'claude-sonnet-4-latest',
  messages: [{ role: 'user', content: 'Hello' }]
});
console.log(result.input_tokens);

import "github.com/pkoukk/tiktoken-go"
enc, _ := tiktoken.EncodingForModel("gpt-4o")
tokens := enc.Encode("Hello, world!", nil, nil)
fmt.Println(len(tokens))

Cost optimization patterns that actually work

1. Smaller model for routing. Route 80% of requests to Haiku/4o-mini ($0.15-1/M); reserve Opus/GPT-4o for the 20% that need it. Savings: 5-10× on routing-eligible traffic.
2. Cache the system prompt. If 90%+ of your input is identical across requests, caching alone cuts input cost by ~80%.
3. Trim conversation history. Don't re-send the full transcript every turn. Use sliding window (last N turns) or semantic compaction (LLM-summarized history).
4. Cap max_tokens. Output is 3-5× more expensive than input. A 500-token cap saves money even when the model could keep going.
5. Use Batch API for non-interactive workloads. 50% off, ships in < 24 h.
6. Strip tool schemas you don't need. Each tool definition is sent every turn. Conditionally include only the tools relevant to the current step.
7. Move from JSON-mode to structured output schemas. Schema-mode (OpenAI's JSON Schema, Anthropic's tool-call schema) reduces malformed outputs and saves retry tokens.
8. Pre-tokenize and reject overlong prompts. A user prompt that exceeds the context window costs you a failed call and the wasted prompt input bytes.

Common token-counting mistakes

• Trusting the 4-chars rule for non-English text. CJK languages run 1.5-3 chars/token on older tokenizers. Use the actual tokenizer.
• Forgetting tool calls and function schemas. They count as input tokens every turn.
• Counting only the user message and missing the system prompt. Long system prompts (instructions, persona, examples) often dominate.
• Counting input tokens but not estimating output. Output dominates cost on most chat workloads.
• Comparing prices without comparing models. Claude Haiku 4 ($1/$5) is not equivalent quality to GPT-4o ($2.50/$10). Run quality eval before optimizing on price.
• Sending whole files as context. A 10 MB log file dumped into a prompt is > 2 M tokens. Pre-process: chunk + retrieve + summarize.