What Is a Context Window? Limits, Costs, and Why Bigger Is Not Better

A context window is the maximum quantity of text a language model can take into account when producing a response, measured in tokens rather than words. Everything the model needs must fit inside it: the system prompt, the conversation so far, any documents retrieved for the task, tool definitions, tool results, and the response being generated.

When the total exceeds the limit, something has to go. Either the request fails, or older content is dropped or summarized to make room.

How are tokens counted?

Tokens are chunks of text, not words. In English a token averages roughly four characters, so a hundred tokens is about seventy-five words. Published model specifications state each model's window in tokens, and Hugging Face's tokenizer summary explains how the counting works, and tiktoken lets you measure it exactly. Common words are usually a single token, while rare words, names, code and non-English text split into more.

The practical consequence is that you cannot reliably estimate usage by counting words, and the ratio varies by language and content type. Code and structured data consume more tokens per visible character than prose.

Two things people frequently miss when budgeting:

The output counts too. A 200,000-token window shared between input and output means a very long prompt leaves little room to answer.

Tool definitions count. Every tool exposed to the model occupies tokens on every request, whether or not it gets used. Connect fifty tools and you are paying that overhead each time, which is one reason MCP server design and tool-set scoping matter at scale.

Why does quality drop before the limit?

The stated limit is a hard ceiling, not a promise of consistent quality up to it. Two effects degrade performance as the window fills.

Position effects. Models attend unevenly across a long input. Information at the beginning and end is used more reliably than material buried in the middle. This is often called the lost-in-the-middle problem, documented in research on how language models use long contexts, and it means that where you place something in a long prompt affects whether it gets used.

Dilution. Attention is finite. Twenty loosely relevant documents compete with the two that actually matter, and the signal-to-noise ratio of the input drives the quality of the output more than raw volume does.

The practical rule that follows is unintuitive: retrieving fewer, better-targeted passages usually beats retrieving more. Teams that see poor answers often add more context, which makes the problem worse. This is the single most common self-inflicted failure in retrieval systems.

Context window versus memory

These get conflated constantly and they are different things.

A model with a very large window still has no memory. It has a large short-term buffer that empties completely when the session ends. Ask it what you agreed last Tuesday and it has no idea, because nothing persisted.

Memory is the layer that decides what survives. For how that works in practice, including episodic versus semantic storage and the vector-versus-graph decision, see AI agent memory.

What does a bigger context window actually solve?

Large windows are genuinely useful for a narrow set of jobs: reasoning over a single long document, working across a large codebase in one pass, or holding an extended conversation without truncation.

They do not solve knowledge, freshness, or cost.

Knowledge. Your company's information is not in the window unless something put it there. That is a retrieval and connection problem.

Freshness. A window holds what you supply. If the data is stale, a bigger window holds more stale data.

Cost. This is the one that surprises teams. Filling a large window on every request multiplies spend directly, because input tokens are billed per request. An agent that stuffs 150,000 tokens into every call to avoid building retrieval is making a costly architectural choice, and it is a common reason token spend climbs without a corresponding increase in output.

How do you manage a context window in production?

Compaction. Summarize older turns rather than carrying them verbatim. Most long-running agents do this automatically once a threshold is hit.

Scoped retrieval. Filter before you rank. Narrow by user, entity, or time window first, and only then rank what remains by relevance. This is the practical argument for structured stores over pure similarity search.

Tool-set scoping. Expose only the tools relevant to the current task rather than everything available.

Prompt caching. Where a provider supports it, a stable prefix such as a system prompt and tool definitions can be cached across requests, which cuts the cost of the repeated portion substantially. Anthropic's prompt caching documentation covers how the cache prefix is matched.

Measure what you send. Log token counts per request, split into system, retrieved context, tools and conversation. Teams are routinely surprised by which of the four dominates.

Where Coworker AI fits

Most context problems are really connection problems. The window is empty of your company's information until something fills it, and doing that well means retrieving the right passages rather than the most.

Coworker AI connects to 50+ tools, maintains organizational memory across them, and supplies scoped context to agents rather than dumping raw documents into a prompt. It exposes that same context over MCP, so the tools your team already uses get it too. Plans are Pro at $29.99 per user per month, Max at $149.99, and Enterprise on request.

Book a demo to see it against your own stack.

Frequently asked questions

What is a context window in AI?

The maximum amount of text, measured in tokens, that a language model can consider in a single request. It includes the system prompt, conversation history, retrieved documents, tool definitions and the generated response, all counted together against one limit.

How many words is a token?

In English a token averages about four characters, so roughly 750 words per 1,000 tokens. Code, names, and non-English text tokenize less efficiently and consume more tokens per visible character.

Is a bigger context window always better?

No. Larger windows help with genuinely long single inputs, but accuracy degrades as the window fills because attention is uneven and relevant material competes with irrelevant material. Cost also rises directly with input size, since input tokens are billed on every request.

What is the difference between a context window and memory?

The context window is short-term working space that empties when the request ends. Memory persists across sessions in an external store and is written, retrieved, updated and pruned over time. A large window is not memory.

Why do answers get worse in long conversations?

Two reasons. Older turns get compacted or dropped to fit, so detail is lost. And as the window fills, the model attends less reliably to material in the middle of a long input, so relevant context can be present but underused.

How do I reduce context window costs?

Retrieve fewer and better-scoped passages, compact conversation history rather than carrying it verbatim, expose only the tools needed for the task, and use prompt caching for stable prefixes. Measuring token counts by category first usually reveals an obvious target.

Related reading

• What Is AI Agent Memory?
• What Is MCP (Model Context Protocol)?
• What Is an LLM Gateway?
• Mem0 vs Zep vs Letta Compared
• LLM Agent Architecture
• AI Agent Orchestration Platforms
• Agent Workflows Explained
• Enterprise AI Agents

To find out where your tokens actually go, see LLM observability.

Quality drops before the limit does, and that has a name. See what context rot is.

Deciding what fills the window is its own discipline. See context engineering.