“128K context window” — but 128K of what? The answer (tokens) raises more questions than it settles: what’s a token, how does it relate to words, why does the same text measure differently across models, and how do you estimate what actually fits? The measurement system underlying context windows is unintuitive, variable, and practically important — because you can’t manage what you can’t measure.
This guide covers the units, the conversions, the variations, and the practical estimation techniques. For the concept itself, see what a context window is.
Table of Contents
- The Unit: Tokens
- Tokens vs Words vs Characters
- How Tokenization Works (Briefly)
- Why Measurement Varies by Model
- Why Measurement Varies by Content
- Practical Estimation: Rules of Thumb
- Tools for Exact Counting
- What Counts Against Your Window
- A Worked Example: Estimating a Real Project’s Token Budget
- Frequently Asked Questions
The Unit: Tokens
Context windows are measured in tokens — the sub-word pieces that models actually process. Not words, not characters, not pages. Tokens.
A token is whatever chunk the model’s tokenizer produces: common words are often single tokens (“the” = 1), uncommon words split into pieces (“antidisestablishmentarianism” = several), and characters, punctuation, and whitespace all consume tokens according to the tokenizer’s rules.
Why tokens instead of words? Because it’s the model’s native unit — the window limit applies at the processing level, and processing happens in tokens. Measuring in words would be like measuring fuel in “trips” — intuitively appealing, technically wrong.
The consequence: to understand “128K context,” you must think in tokens. Everything below is about building that intuition.
Tokens vs Words vs Characters
The conversions everyone needs:
English prose: ~1 token per 0.75 words, or ~4 characters per token. So 1,000 tokens ≈ 750 words ≈ 4,000 characters ≈ 1.5 pages.
The key ratios: – 1,000 tokens ≈ 750 English words – 100,000 tokens ≈ 75,000 words ≈ 150 pages – 1,000,000 tokens ≈ 750,000 words ≈ 1,500 pages
Characters are unreliable as a measure — “a” and “antidisestablishmentarianism” differ enormously in token cost but the character ratio doesn’t capture the sub-word dynamics cleanly. Characters per token varies (roughly 4:1 for English, but it’s an average over huge variation).
Words are better but imperfect — the 0.75 ratio is an English average. Technical text, with its jargon and compounds, tokenizes less efficiently. Simple text tokenizes more efficiently.
Pages are folk units — “pages” depends on formatting, font, density. Useful for intuition (“150 pages”), useless for precision.
For serious work: think in tokens, estimate from words using 0.75, verify with tools for critical cases.
How Tokenization Works (Briefly)
Tokenizers (usually byte-pair encoding or variants) learn a vocabulary of pieces from training data:
- Start with individual characters.
- Merge the most frequent pairs iteratively — “t”+”h” becomes “th”, “th”+”e” becomes “the”.
- Result: a vocabulary where common sequences are single tokens and rare ones compose from pieces.
This is why: – Common words = 1 token (“and”, “the”, “is”) – Uncommon words = multiple tokens (“sesquipedalian” might be 4–5) – Made-up words = many tokens (each piece unfamiliar) – Different languages = wildly different efficiency (the tokenizer trained mostly on English)
The tokenizer is model-specific — learned from that model’s training data. Which means token counts differ between models for identical text. “128K tokens” on one model isn’t the same amount of text as “128K tokens” on another, if their tokenizers differ significantly.
Why Measurement Varies by Model
Same text, different models, different token counts. Reasons:
Different vocabularies. A 50K-token vocabulary chunks differently than a 200K-token vocabulary. Larger vocabularies tend to be more efficient (more whole words as single tokens).
Different training data. Tokenizers optimized for English differ from multilingual ones. A model trained heavily on code tokenizes code more efficiently.
Different algorithms. BPE, SentencePiece, tiktoken variants — the algorithm affects chunking decisions at the margins.
Practical impact: usually 10–30% variation between major tokenizers for English prose. Enough to matter when you’re near limits, not enough to change the big picture. For critical applications (fitting exactly), count with the specific model’s tokenizer — never assume transferability.
The standardization gap: there’s no universal token. “Tokens” are model-relative units, like “steps” varying by stride length. The industry quotes them as if standard; they’re not. Keep this in mind when comparing windows across models — 128K tokens on model A might hold 10% more or less text than 128K on model B. When precision matters, always validate your estimate against the tokenizer the target model actually uses rather than trusting a published figure.
Why Measurement Varies by Content
The same model tokenizes different content at different rates:
English prose: the baseline — ~0.75 words/token. What all estimates assume.
Technical/academic text: worse — jargon, compounds, and specialized terms split into more pieces. Expect 0.5–0.6 words/token for dense technical writing.
Code: significantly worse — symbols, indentation, unusual identifiers. Code often runs 2–3x the tokens of equivalent English prose. A 1,000-line file might be 8,000–15,000 tokens.
Other languages: highly variable. Languages well-represented in training (Spanish, French) tokenize reasonably. Others can be 2–4x worse than English — a real equity issue in AI access, and a practical shock when estimating.
Structured data (JSON, XML): the scaffolding tax — brackets, quotes, keys, and repetition inflate token counts substantially versus the information content.
Whitespace and formatting: newlines, indentation, markdown — all consume tokens. Heavily formatted documents cost more than their words suggest.
The rule: estimate from the content type, not just the word count. Technical, code, non-English, or structured content — adjust upward, sometimes dramatically.
Practical Estimation: Rules of Thumb
Quick estimates for planning:
English prose: words × 1.33 = tokens. (1,000 words ≈ 1,333 tokens.)
Pages: pages × 500 words × 1.33 = tokens. (100 pages ≈ 66,500 tokens.)
Code: lines × 10 tokens (rough). (1,000 lines ≈ 10,000 tokens. Wide variance.)
Conversation: messages × average length × 1.33. (50 messages of 100 words ≈ 6,650 tokens.)
Documents with formatting: add 20–30% to prose estimates for markdown/structure overhead.
Non-English: multiply English estimates by 1.5–3x depending on language. (When in doubt, measure — don’t guess.)
Safety margin: for critical fits, estimate then add 25%. Tokenizer variance, hidden system prompts, and formatting surprises all push upward.
These get you within 20–30% for planning purposes. For exact fits (will this prompt work?), use tools.
Tools for Exact Counting
When estimation isn’t enough:
Official tokenizers. Most major labs publish their tokenizers (tiktoken for OpenAI models, etc.) — programmatic counting with the exact tokenizer is the gold standard.
Online counters. Web-based token counters for common models — paste text, get counts. Fine for one-offs, not for sensitive content (you’re pasting your text into a website).
API endpoints. Some APIs return token counts for requests — the authoritative number, since it’s what the system actually counted.
IDE extensions and libraries. Programmatic counting integrated into development workflows — essential for builders managing context programmatically.
The platform’s own display. Some chat interfaces show token counts. Trust these — they’re measuring with the real tokenizer.
Best practice: estimate for planning, measure for critical decisions. The tools exist; use them when precision matters.
What Counts Against Your Window
Everything. The complete inventory:
- System instructions — invisible to you, consuming tokens. Can be substantial (thousands of tokens on some platforms).
- Conversation history — every message, both sides, in full.
- Your current message — obviously.
- Retrieved content — RAG documents, tool outputs, all of it.
- Formatting overhead — the structural tokens wrapping everything (message delimiters, roles, etc.). Typically 5–15% above raw content.
- The response being generated — output tokens count too. The window is shared.
- Tool definitions — in agentic systems, function schemas consume significant tokens.
The invisible consumers (system prompts, formatting) are why your 100K-token document doesn’t leave 28K free in a 128K window — the real available space is less than the arithmetic suggests. When estimating fits, budget 10–20% for overhead you can’t see.
Understanding how ChatGPT works as a product illuminates where these hidden consumers live — the product wraps your input in substantial scaffolding.
A Worked Example: Estimating a Real Project’s Token Budget
Let’s budget a realistic project: analyzing 40 PDF reports (average 30 pages each) with room for a working conversation and the final output.
Step 1 — the documents. 40 × 30 pages = 1,200 pages. At ~500 words per page: 600,000 words. English prose at 1.33 tokens/word: ~800,000 tokens.
Step 2 — formatting overhead. PDFs with tables, headers, and structure: add 20%. Now ~960,000 tokens.
Step 3 — the conversation. Say 50 exchanges of substantive Q&A at ~400 tokens each (both sides, reprocessed): ~20,000 tokens. Plus system prompt and scaffolding: ~5,000. Running total: ~985,000.
Step 4 — output headroom. The final analysis might run 5,000–10,000 tokens. Reserve 15,000 to be safe. Total budget: ~1,000,000 tokens.
The verdict: this project sits exactly at the edge of a 1M-token window — technically fittable, practically uncomfortable, with zero margin for the overhead you can’t see. My call: either use RAG (retrieve per question, keep per-query costs sane) or split the corpus into two 500K halves with an extract-then-synthesize pass.
Notice what the exercise bought: a decision, not just a number. Estimation isn’t about precision — my figures could be off 20% and the conclusion holds. It’s about knowing before you start whether you’re in “paste it all” territory or “build retrieval” territory. Ten minutes of arithmetic saves weeks of wrong architecture.
Frequently Asked Questions
In tokens — sub-word pieces the model processes. Roughly 1 token ≈ 0.75 English words ≈ 4 characters. So 128K tokens ≈ 96,000 words ≈ 190 pages of English prose. Code, technical text, and non-English content tokenize less efficiently.
The sub-word unit models process — common words are often single tokens, rare words split into pieces. Tokenizers learn these pieces from training data, so token counts vary by model and content type. It’s the native unit of context measurement.
Roughly 96,000 English words (~190 pages), minus overhead for system prompts and formatting (10–20%). Technical text, code, or non-English content fits substantially less. Always budget margin — and remember output shares the window.
Each model has its own tokenizer with different vocabulary and training data. A 50K-vocabulary tokenizer chunks differently than a 200K one. Expect 10–30% variation for English prose. “Tokens” aren’t standard units — always count with the specific model’s tokenizer for precision.
Yes, significantly — often 2–3x more tokens than equivalent English prose, due to symbols, indentation, and unusual identifiers. Budget ~10 tokens per line of code as a rough estimate. This is why code-heavy prompts consume windows faster than expected.
