FundamentalsIntermediate
Context Windows & Token Limits Explained: From 8k to 2 Million Tokens
Direct Answer & Overview
A context window is the maximum number of tokens (words and punctuation) that a large language model can ingest and consider simultaneously in a single prompt and completion conversation cycle.
1.The Evolution of LLM Context Windows
In 2023, frontier models operated with 4,000 to 8,000 token windows. In 2026, context capacity has expanded by orders of magnitude:
• GPT-4o: 128,000 tokens (~96,000 words)
• Claude 3.5 Sonnet: 200,000 tokens (~150,000 words or a 500-page book)
• Gemini 1.5 Pro: 2,000,000 tokens (~1,500,000 words, 1 hour of video, or 60,000 lines of code)
• DeepSeek R1: 64,000 tokens with deep chain-of-thought allocation.
2.Needle In A Haystack (NIAH) Retrieval Fidelity
Having a large context window is only useful if the model can accurately retrieve specific facts placed deep within the context. Modern frontier models (especially Gemini 1.5 Pro and Claude 3.5 Sonnet) achieve 99.8%+ NIAH retrieval across their entire multi-million token spans.
3.Token Economics in Long-Context Applications
Repeatedly submitting a 100,000-token prompt on every turn rapidly depletes API budgets. Implementing Prompt Caching (supported on API100 for Claude and Gemini) discounts cached input tokens by up to 80-90%, making long-context document analysis cost-effective.
Querying 100k+ Tokens on Gemini 1.5 Pro via API100python
from openai import OpenAI
client = OpenAI(base_url="https://api.apihundred.com/v1", api_key="your_api100_key")
# Ingest an entire multi-file codebase or technical manual
with open("massive_codebase_dump.txt", "r") as f:
codebase_content = f.read()
response = client.chat.completions.create(
model="gemini-1-5-pro",
messages=[
{"role": "system", "content": "You are a lead security auditor."},
{"role": "user", "content": f"Audit this codebase for race conditions:\n\n{codebase_content}"}
]
)
print(response.choices[0].message.content)Frequently Asked Questions
What is a context window in AI?
The context window is the total capacity of tokens (input prompt plus output answer) that an AI model can hold in its working memory during a single inference call.
Which model has the largest context window in 2026?
Google's Gemini 1.5 Pro offers the largest commercially available context window at 2,000,000 tokens (approx. 1.5 million words).
A100
API100 Engineering Team
Infrastructure & Latency Research