Picture a refund agent: it reads a ticket, checks order status, verifies policy, produces a recommendation, and asks the user to confirm before submitting. The prompt spells out the task, authorized actions, and confirmation boundaries — on its own, that step looks fine.
The problem shows up starting from the second turn. Once the agent has checked order status and then checked policy, the tool’s raw JSON, the retry log from a previous error, and order fields that have nothing to do with this refund all stay in the conversation. By the fifth turn, half of what’s in the window has nothing to do with the current judgment, and the model starts treating an old tool response as something it should still weigh in. The prompt itself hasn’t changed — but it’s no longer the only thing the model sees.
That’s the problem context engineering deals with: the next time the model samples, the window holds more than that one prompt. It also holds system conventions, retrieved documents, tool results, and everything recorded up to this point, all packed into the same stream of tokens. In the summer of 2025, this kind of work settled into a fairly fixed name. At its core, it means assembling the right context for a task so the model has a chance of getting it right. In production, it means filling the window with exactly the right amount: too little and the model can’t do the job, too much or in the wrong shape and cost goes up while performance goes down.
How it differs from prompt engineering
Prompt engineering is about how to write and organize instructions, especially the system prompt: task, basis for judgment, constraints, examples, output format — these fields answer “how should this particular call be phrased.” Context engineering manages the entire state during inference: beyond the instruction itself, there are tools, the Model Context Protocol (MCP, which drops a batch of tool descriptions into the window all at once), external data, message history, and the decision, at every turn, of what to keep and what to drop.
The refund agent’s prompt only has to be written once; but which turn it’s on, which tool results are still sitting in the window, and whether to carry an error from three turns ago all have to be decided fresh at every step. Many early tasks were one-shot classification or generation, where writing a good system prompt was enough. An agent runs many turns in a row, piling new material into the window at every one, and every step has to re-select what belongs. The prompt is still there — it’s just become one block inside the window. When the window gets dirty, first check whether the instruction was unclear, or whether something that shouldn’t have entered this turn’s window did.
It isn’t the whole of building an LLM application: how to break down a problem, which model to call for what, validation and UI, safety, and evaluation are not covered by this term — those still belong to prompt engineering and product design.
What goes into a single inference call
A window is usually assembled from these pieces:
- Instructions: system prompt, rules, few-shot examples, tool descriptions.
- The user’s current message.
- Short-term state: the conversation and tool trace up to this point.
- Long-term memory: preferences, project summaries, and facts stored across sessions.
- Retrieved external material.
- Tool definitions, and the previous turn’s tool results.
- Output format: JSON Schema, enums, required fields.
By the third turn, the refund agent’s window already has: the system prompt (task, authorized actions, confirmation boundaries), the original ticket, the order lookup’s full response, a policy document excerpt, and the error message from the first failed tool call. It helps to think of the model as a CPU and the window as memory: you don’t load the whole disk into memory, only what this step actually needs.
What a bigger window doesn’t fix
There was a clean experiment back in 2023: in multi-document QA, the answer was always in one document, and only that document’s position changed. Models performed best when the answer sat at the start or end of the context, and noticeably worse in the middle — a U-shaped curve. In some setups, having the answer in the middle performed no better than giving the model no documents at all (closed-book). In a simpler key-value retrieval task, some models even struggled to “pull a matched value out of the middle.” That’s Lost in the Middle.
This lines up with how next-token prediction works. Every token in a Transformer computes a pairwise relevance score against every other token, so relationships scale as with sequence length . As the sequence gets longer, those relationships get sparser, and training data also skews toward shorter sequences. So it’s not that performance suddenly breaks at some length — it degrades gradually as length grows.
The Context Rot report from July 2025 measured this again, across 18 models available at the time. The tasks were kept deliberately simple, varying only input length. The conclusion: models don’t use context evenly, and reliability drops as input gets longer. Needle in a Haystack — finding one verbatim sentence in a long document — is nearly saturated for many models, which creates the illusion that “long context is a solved problem.” Change the needle to something semantically related and add distractors, and the drop-off shows up immediately.
Back in the refund agent example: dumping the full order lookup response and the entire policy document into the window looks like “giving it enough information,” but it’s actually adding noise. Waiting for a bigger window doesn’t fix contamination or relevance — long-running tasks still have to manage what stays in the window.
The principle: small and useful
Aim for a small set of high-signal tokens that make the target outcome more likely. Small doesn’t mean short — necessary behavior, constraints, and examples still need to be given in full. A reasonable approach is to start with the minimal prompt and the strongest model, then add material as failure modes show up, rather than writing everything up front.
The system prompt needs the right altitude. One extreme is writing if-else logic straight into the prompt — brittle and hard to maintain. The other extreme is being too vague, pretending the model shares your background context. The workable range sits in between: specific enough to guide behavior, while leaving room for heuristics instead of hard-coding every path.
Tools are the contract between the model and the outside world — return values should be token-efficient, and functionality shouldn’t overlap. If a refund agent has both a “look up order” tool and a “look up order details” tool, that’s overlap, and the model’s odds of picking the wrong one go up. If a person can’t tell which tool to use at this step, the model won’t be able to either.
Few-shot examples still work, but don’t turn them into a laundry list of edge cases. Pick a diverse, representative set. Too many examples that look alike can pull the model into repeating the same action instead of reasoning about the task.
Write: store outside the window
Writing means storing something outside the window first, then reading it back in when needed.
Within a single task, a scratchpad helps: write the plan into memory or a NOTES.md file, and the plan survives even if the window gets truncated past its limit. Claude Code’s to-do list, or a continuously rewritten todo.md, are the same idea. A refund agent could keep “confirmed ticket details, policy still to check, recommendation still to produce” as a structured state file instead of relying on the model to re-derive progress from conversation history. Repeating the overall goal near the end of the context is effectively pushing the plan back to the most recently seen position, easing the lost-in-the-middle problem.
Cross-session memory is a different layer. Products extract long-term memory from interactions. Getting the selection wrong is annoying — drawing a diagram that shouldn’t include a location, but pulling a home address out of memory and dropping it in anyway. Once memory selection goes wrong even once, users feel like the window no longer belongs to them.
Select: load it in when it’s needed
Selecting means deciding, at this step, what to read in.
One approach is retrieval before inference: embedding search, rule files, a CLAUDE.md that always loads into the window. This is still common in coding agents. Building an index isn’t the same as retrieving well — once a codebase gets large, embeddings alone aren’t reliable, and you end up layering in grep, a knowledge graph, and reranking.
The other approach is just-in-time retrieval: keep only lightweight identifiers (paths, queries, links) around, and load the real content at runtime through a tool. A refund agent doesn’t need the full order history up front — grab the order number first, and query again only when it actually needs to verify refund count. For a large database, write a query and check it with head / tail instead of stuffing the whole table into the window. The trade-off is speed: just-in-time is slower than prefetching, and when tools and heuristics aren’t designed well, the agent can spin its wheels or go down a dead end.
A common hybrid: CLAUDE.md loads into the window from the start, and specific files get pulled in on demand via glob and grep.
Agent Skills turned “show the table of contents first, then flip to the relevant chapter” into a product mechanism. At startup, only each skill’s name and description go into the system prompt; once one becomes relevant, the full SKILL.md gets read; more detailed instructions and scripts open only when actually used. Scripts can run outside the window and send back only their output. That means a skill can bundle almost any amount of material without the window becoming the limiting factor.
Too many tools is also a selection problem. Overlapping descriptions make the model pick wrong. Dynamically loading and unloading tools mid-run usually isn’t worth it either — tool definitions sit early in the window, and changing them invalidates the KV-cache for everything after; history can also still reference a tool that’s since been unloaded. It’s more stable to keep definitions fixed and just mask certain tool names off at decode time.
Compress: shrink it without losing what you’d need back
Compressing means, when the window is nearly full or already noisy, keeping the part that lets work continue.
Compaction summarizes a conversation as it nears its limit, then opens a fresh window with that summary. Keep architectural decisions, unresolved bugs, and implementation details; drop repeated tool output, and carry forward the last few files that were opened. Make sure the essentials survive first, then trim the rest. The lightest version of this is just clearing out very old tool results. By the tenth turn, a refund agent could compress several rounds of policy checking into a single line — “verified against refund policy section 3.2, eligible” — but fields it still needs, like order number and ticket ID, shouldn’t get compressed away.
Go a step stricter, and compression has to be reversible. A web page’s body text can leave the window, but the URL should stay; a document doesn’t need to sit in the window, but the sandbox path should. The filesystem is nearly unlimited context. Irreversible compression is risky, because you don’t know which observation ten steps from now will turn out to matter.
Tied closely to compaction is the KV-cache. An agent’s input is usually far longer than its output — some products see roughly 100
. When the prefix stays stable, caching cuts latency and cost; a cached input can cost roughly a tenth of an uncached one. To get high hit rates:- Keep the prefix stable. A second-precision timestamp at the start of the system prompt invalidates the cache from that token onward.
- Only append; don’t modify actions and observations that already happened. Unstable JSON key ordering during serialization can quietly break the cache too.
- Set explicit cache breakpoints where needed, covering at least the end of the system prompt.
Isolate: don’t cram everything into one window
Isolating means splitting context apart so no single piece of work fills the entire window.
Sub-agents are the most common way to split. A sub-agent runs a long retrieval on its own and hands back only a 1,000–2,000 token summary to the main agent. For complex research tasks, this often ends up cleaner than keeping every search trace in the main window on a single thread. If a refund agent needs to check three different versions of a policy document at once, it can hand each one to a sub-agent and get back only “eligible or not, plus the supporting clause.”
Parallel writes, on the other hand, go wrong easily. It’s not that “multiple models” is inherently broken — it’s that context doesn’t transfer cleanly. What needs to be shared is the full trace, not just a one-line task handed off. Actions carry decisions that were never written down, and when those decisions conflict, the result breaks.
Split “build a Flappy Bird clone” into two subtasks — one for the background, one for the bird — and the background comes back looking like Mario pipes, and the bird comes back not looking like a game asset at all. Copying the original task to each sub-agent isn’t enough either, because a real system runs over many turns with tool calls in between. Even if both sides see the same original task, they can still land on different visual styles, because neither side can see the implicit choices the other has already made.
A more reliable middle ground: parallelize reads, and keep writes on a thread that can see the decisions already made. Investigation can be handed off to a subtask; changing code should stay on the thread with visibility into existing decisions. Early Claude Code, even when it spawned subtasks, mostly used them to answer questions rather than write in parallel.
Heavy objects can also just stay in the environment. Run code in a sandbox; assign images and audio to variables and pass the return value back to the model only when needed. Runtime state objects work the same way — a schema might expose only a messages field to the model each turn, with other fields surfaced on demand.
Every action carries an unstated decision
A “task description” isn’t the only thing that counts as context. Editing a file, picking a library, defining an interface — each of those makes a decision that was never written into the prompt. If a later step can’t see that decision, it keeps working from a different set of assumptions.
In 2024, a fair number of coding agents used a pattern where a large model writes an edit description and a small model rewrites the entire file according to it. A slightly ambiguous description, and the small model gets the edit wrong. Splitting decision from execution, combined with an incomplete context handoff, produces exactly that failure. Later on, editing decisions and writing to disk more often happen in the same pass, by the same model.
When designing a split, it helps to ask up front: once this step is done, what decisions got written into the world, and will the next step actually see them?
Errors and few-shot
At the moment something fails, the model generally does best when it can see the failed action and its observation. Erasing a failure erases the evidence it needs to update its priors. Error recovery is a real signal of whether an agent can keep working, and public benchmarks rarely test for it.
Once something is fixed, the raw stack trace doesn’t need to keep occupying space, but “what was tried, and why it didn’t work” is worth keeping in a denser form.
Few-shot cuts both ways too. Examples should be diverse and typical. For a repetitive task like reviewing 20 résumés, if the window already shows the same action-observation pattern over and over, the model tends to follow along — sometimes into hallucination. A bit of variation in serialization and wording can break the pattern. Don’t dig your own hole with few-shot.
Three examples
Getting a refund agent the window it should see
In a first implementation, the refund agent carried its full conversation history every turn: the original ticket, the entire order-lookup JSON, the full policy document, and the stack trace from the last failure. By the fifth turn, the model started treating “the last lookup failed” as “this one should fail too.”
The fix maps onto write, select, compress, and isolate, one at a time:
- Write: turn “ticket conclusion, policy clauses already checked, pending action” into a structured state, instead of making the model re-derive it from conversation history.
- Select: don’t load the whole policy document — retrieve the matching clause first, and bring in only that clause’s text and link.
- Compress: use the order lookup’s raw response only within that turn, then compress it next turn into a line like “order status: shipped, tracking SF123.”
- Isolate: when several policy documents need checking at once, hand them to sub-agents and bring back only “conclusion + basis” for each.
What stays in the window is the current conclusion the model needs to see, not a raw response it has to re-read every single turn.
Replying to a scheduling email
The email is one line: “Free for a quick sync tomorrow?”
With only that line in the window, the model can only reply with something like “sure, tomorrow works, what time?” — reads like a form response.
Load in the calendar (tomorrow is fully booked), the email history with this person (their tone), the contact record (this is an important partner), and the tools to send an invite or reply, before making the call. Now the model can write: “I’m booked all day tomorrow — if Thursday morning works, I’ll send an invite.”
The difference isn’t that the model suddenly got smarter. It’s whether what this step should see actually made it into the window.
A deployment
The window format can be whatever you define — it doesn’t have to be the standard role: tool message array. Structuring it around events is denser and gives more control over what stays:
The next step is still the same question: here’s what’s happened so far, what happens next. The tool failure stays in the window, so the model shifts to checking the workflow’s status instead of just deploying again.
Coding agents can be organized around the same checklist: keep system conventions (CLAUDE.md / rules) present from the start; pull relevant files in via search rather than embedding the whole repo; write large tool output to a file and keep only the path in the window; summarize once the window nears its threshold, but preserve unfinished decisions and the most recently opened files; hand investigative work to a sub-agent, and keep code changes on the thread that can see existing decisions.
Common mistakes
- Treating a long window as a free lunch. A perfect Needle in a Haystack score doesn’t mean the middle content is being used well. Assume longer means noisier by default, then decide what actually belongs.
- Putting a value that changes — especially a precise timestamp — in the system prompt. The cache invalidates from the first differing token onward.
- Adding or removing tool definitions mid-run. The prefix changes, and history can still reference a tool that no longer exists.
- Writing every edge case as a rule instead of choosing a handful of representative examples.
- Multiple agents writing in parallel without sharing the full trace and the decisions already made.
- Irreversible compression. An observation you’ll need ten steps from now, kept only as a rough summary, may not come back. Keep a pointer if compression can be reversed.
- Too many tools that overlap in function. If people can’t tell which one to use, cut down until they can.
- Context that’s too uniform in a repetitive task, leading the model to echo itself.
- Erasing a failure from the trace the moment it’s fixed, leaving the model no evidence to change its behavior.
- Injecting long-term memory indiscriminately. Get the selection wrong once, and the user feels like the window has been taken from them.
Practical checklist
- Every step can state clearly what it should see, instead of defaulting to full history.
- Large blocks like tool results and retrieved passages are compressed or turned into pointers once used, not left occupying the window indefinitely.
- The system prompt prefix is stable; variable content (timestamps, etc.) doesn’t sit inside the cached prefix.
- Compression is reversible: URLs, paths, and IDs are kept as pointers instead of being summarized away irreversibly.
- Failed actions and their observations stay in the trace instead of being erased right after a fix.
- Parallel subtasks are used only for work that can read independently; writes stay on the main thread that can see existing decisions.
- Tool definitions don’t overlap noticeably, and aren’t added or removed dynamically mid-run.
- Long-term memory writes and reads both have explicit triggers, not indiscriminate injection.
References
- Tobi Lütke, post on context engineering, 2025-06-19.
- Andrej Karpathy, post on context engineering, 2025-06-25.
- Simon Willison, Context engineering, 2025-06-27.
- Phil Schmid, The New Skill in AI is Not Prompting, It’s Context Engineering, 2025-06-30.
- Lance Martin, Context Engineering for Agents, 2025-06-23.
- Walden Yan / Cognition, Don’t Build Multi-Agents, 2025-06-12.
- Peak Ji / Manus, Context Engineering for AI Agents: Lessons from Building Manus, 2025-07-18.
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts, 2023.
- Hong, Troynikov, Huber / Chroma, Context Rot, 2025-07.
- Anthropic, Effective context engineering for AI agents, 2025-09-29.
- Anthropic, Equipping agents for the real world with Agent Skills, 2025-10-16.
- Anthropic, How we built our multi-agent research system, 2025-06-13.
- Dex Horthy / 12-Factor Agents, Factor 3: Own your context window.