Prompt Engineering Is Dead. Here Is What Replaced It.

I want to share something that took me an embarrassingly long time to figure out, even though the evidence was sitting right in front of me the whole time.

We spent months debating model choices. GPT-4o versus Claude 3.5 Sonnet versus Gemini. Every switch was a 2 to 3 week engineering project. Re-running evaluations. Recalibrating prompts. Fixing regressions that appeared out of nowhere. Each switch felt significant because it was expensive.

And then we made a change that had nothing to do with the model at all. We reorganised how user profile information was structured before it entered the context window. Same model. Same task. The intent classification accuracy moved more than the last two model upgrades combined.

That was the moment I realised we had been optimising the wrong thing.

The thing that actually separates AI products that work from AI products that disappoint is not the model underneath them. It is what those products put into the model’s context window at each call, in what form, in what order, retrieved through what logic. That discipline has a name now: context engineering. And I think it is the most underrated and most important concept in AI product development right now.

Let me show you exactly what I mean, and then, because understanding the concept is only half the value, I want to give you a practical guide to finding and landing the roles that are being created around this discipline.

Why Everyone Is Still Arguing About the Wrong Thing

Here is the conversation happening in almost every AI team right now. Should we use GPT-5 or Claude Sonnet? Is Gemini better for reasoning tasks? What about the new open-source models from DeepSeek? The debate is about models, because models are visible. They have names, version numbers, benchmark scores, and press releases.

The problem with this conversation is that it is increasingly a debate about a commodity layer.

Andrej Karpathy, one of the most respected voices in AI engineering, has been arguing for several years that the model itself is the least differentiated part of a production AI system. His framing, what you call a prompt is only one slice of what you are actually handing the model. The rest of it, the retrieved documents, the conversation history, the user profile, the tool definitions, the few-shot examples, the structured constraints, is what actually determines whether the system behaves intelligently or not.

Gartner formalised this in mid-2025 with a declaration that made a lot of AI teams uncomfortable context engineering is in, and prompt engineering is out. Their recommendation to enterprise AI leaders was to stop focusing on prompt phrasing and start building context-aware architectures with dynamic data pipelines.

The numbers from actual deployments back this up. DataHub’s 2026 State of Context Management Report surveyed IT and data leaders across the industry and found that 82% agreed that prompt engineering alone is no longer sufficient to power AI at scale. The same report found that 95% of data teams plan to invest in context engineering capabilities during 2026. That is not a niche opinion. That is the whole industry arriving at the same conclusion from different directions.

So what exactly is context engineering? Let me give you the clearest definition I can manage.

CONTEXT ENGINEERING: A WORKING DEFINITION

Context engineering is the systematic design of what information enters the model at each call: what is included, what is excluded, in what form, in what order, retrieved through what mechanism, and at what computational cost. Prompt engineering is craft at the level of words. Context engineering is craft at the level of architecture.

The prompt is one component. Context engineering covers prompt design, retrieval design, memory design, tool design, structured output design, and cache design. It is where all the consequential technical decisions in AI product development converge.

What Actually Goes Into a Single Model Call

To understand why context engineering matters so much, you need to see what is actually being sent to the model when a user types something. Most people assume it is just the prompt. It is not even close.

Take a customer support AI handling a message like ‘my payment did not go through.’ That sounds simple. But a production system handling that message might be assembling something like this before it touches the model:

• The system prompt: agent persona, behavioural constraints, formatting rules, escalation guidelines
• Tool definitions: schemas and descriptions for every callable function, such as checking payment status, issuing refunds, escalating to a human
• Few-shot examples: two or three carefully chosen prior conversations that teach the model how to handle edge cases in payment disputes
• User profile: pulled from a database at runtime, account age, subscription tier, past complaint history, preferred language
• Conversation history: the current session’s prior turns, compressed and formatted
• Retrieved knowledge: relevant sections of payment policy documents pulled from a vector store based on the semantic similarity to the user’s message
• The actual user message: the thing the user typed

7 components. Assembled in a specific order. Formatted in a specific way. All of that is what the model actually processes. The prompt is one of those seven. The other six are where the product lives.

Change the order of those components and your outputs change. Change which few-shot examples you include and your outputs change. Change how you compress the conversation history and your outputs change. None of those changes require switching models. All of them require context engineering.

The teams that have figured this out are the ones whose products feel qualitatively different even when running on the same underlying model as a competitor. Enterprise AI implementations at Salesforce, ServiceNow, and Microsoft that actually work are almost entirely context systems, CRM history, customer data, policy documents, past tickets, all injected at runtime. The model is the reasoning engine. The context system is the product.

The Five Disciplines That Make Up Context Engineering

When I try to make context engineering practical for the teams I work with, I break it into five specific disciplines. None of these are brand new ideas in isolation. But thinking of them as a coherent set, each one building on the others, is what makes the difference between tinkering and engineering.

Discipline One: Selection

The first question in context engineering, and the most counterintuitive one, is not what to include. It is what to leave out.

The instinct when you have access to a large context window is to fill it. You have a knowledge base, so why not retrieve everything potentially relevant? You have a long conversation history, so why not include all of it? You have a detailed user profile, so why not pass the whole schema?

The reason not to is grounded in research that every AI team should know.

A landmark study from Stanford and UC Berkeley, commonly called ‘Lost in the Middle,’ found that LLMs do not attend uniformly across their context window. They attend strongly to the beginning and end and systematically underweight the middle. The performance drop when critical information sits in the middle rather than at the edges exceeds 30% on multi-document question answering tasks.

Chroma Research extended this in2025 by testing 18 production LLMs, including Claude 4 Sonnet, GPT-4.1, and Gemini 2.5 Pro, on tasks across contexts ranging from 10,000 to 500,000 tokens. Every single model showed monotonically decreasing accuracy as context grew. Not some models. All of them. The steepest degradation occurred in the 100,000 to 500,000 token range, which is exactly the range where teams often assume they can safely dump everything.

A separate analysis found that current LLMs effectively use only 10 to 20% of their context window for retrieval tasks, with accuracy varying significantly based on where in the window the information appears. Stuffing a 200,000 token context window is not just wasteful. It actively dilutes the attention the model pays to the things that matter.

Selection is an active design problem. For a payment support use case, when a user asks about a failed transaction, you do not retrieve their entire account history. You retrieve the last three transactions and the specific policy section covering payment failures. You pass the top two or three retrieved documents ranked by query specificity, not the top ten.

Useful information is the smallest set that lets the model do the task correctly. Anything beyond that is noise that costs you tokens, attention, and latency.

Before adding anything to your context, ask: does removing this change the model output on the task I care about? If the answer is no, remove it. The goal is the minimum sufficient context, not the maximum available context.

Discipline Two: Compression

Once you accept that you cannot put everything in context, the second discipline is what to do with the things you do not put in. Especially conversation history.

A fifty-turn conversation cannot live in the prompt verbatim without becoming expensive and cognitively overwhelming for the model. But the information in those turns matters. So how do you preserve it without dumping it?

3 patterns that actually work in production:

• When a conversation has been running long, replace older turns with a structured summary that preserves the actionable state: ‘Customer is on the Pro plan, has reported two failed payments this week, previous agent offered a courtesy credit that was not applied.’ The model loses verbatim detail but keeps the information it needs to act correctly.
• Do not keep state in the prompt at all when you can avoid it. Persist it to a database and retrieve specific fields on demand via tool calls. This is both cheaper and more reliable than asking the model to extract meaning from a long conversation history it is already underattending to.
• This is a technique highlighted in theManus context engineering post: periodically rewriting the current task state into the end of the context so the most recent attention window sees the most important information. For long agent tasks, this is what separates an agent that stays on plan from one that slowly drifts.

The key insight is that raw history and necessary history are not the same thing. Most of what was said in a conversation is no longer relevant by the time you are three turns in. The work of compression is identifying the small fraction that is.

Discipline Three: Structure

Same information. Different organisation. Noticeably different model behaviour. This is the discipline that took me longest to take seriously and that turned out to deliver some of the largest gains.

A few specific things that are documented and reproducible:

• Structured tags outperform flowing paragraphs. Claude in particular performs measurably better when context is wrapped in XML-style tags than when the same content is written as natural prose. The same user profile, restructured with clear semantic boundaries, produces more accurate references to that profile in the model output. Anthropic documents this in their own engineering guidance.
• Models attend more strongly to instructions at the very start and the very end of context, because of the architectural biases we covered in the selection discipline. Constraints you want the model to consistently respect: safety boundaries, output format requirements, persona definitions, belong at the edges. Not buried in the middle where they will be underweighted.
• Every major model now has a structured tool-calling API. Using it is more reliable than describing tools in the system prompt and parsing string outputs. Anthropic has published data showing that rewriting tool descriptions alone can lift task success 5 to 20% on the same model without any other changes. That improvement is entirely at the context layer, not the model layer.

Treat context like any other designed interface. It should have explicit structure, predictable ordering, and clear semantic boundaries between components. The model is not a human reader who can infer context from narrative flow. It responds to architecture.

Discipline Four: Cache Discipline

This is the most technical of the five disciplines, and the one that product people are most tempted to leave entirely to engineers. That is a mistake, because product decisions made without understanding caching can silently multiply serving costs by three to five times.

Here is the mechanism in plain language. Modern LLM serving systems reuse computation across calls when the beginning of the prompt matches between calls. This is called KV-cache. A high cache hit rate dramatically reduces both latency and cost. A low one dramatically increases both.

What destroys cache hit rates? Changes to the beginning of your prompt. And the things that cause those changes often look completely innocent from a product perspective:

• Adding the current date to the system prompt causes a cache miss every day
• Personalising the system prompt with the user’s name causes cache misses across users
• Dynamically reordering tool lists based on user context causes cache misses on every reorder
• Making any change to your prompt structure to fix an edge case causes cache misses across the entire deployment

Each of these can look like a small, sensible product improvement and quietly make your serving costs three to five times higher. The fix is not to never personalise. It is to push personalisation to the end of context rather than the beginning, preserving the stable prefix that the cache can reuse.

Treat your prompt structure like a database schema. It can be changed, but changes need to be deliberate, tested, and deployed with awareness of the downstream cost impact. Monitoring cache hit rate as a first-class metric, not an afterthought, is what makes this discipline real rather than theoretical.

Discipline Five: Evaluation

The 5th discipline is the one without which the other four are guesswork.

Every change you make to your context, adding a few-shot example, restructuring the prompt, swapping retrieval logic, modifying tool descriptions, should pass through an evaluation gate before it ships. Without that gate, you are operating on vibes. You might be improving things. You might be regressing them. You have no way to know.

What makes context engineering evaluation harder than general model evaluation is that the changes are often subtle. A new few-shot example might improve one failure mode and quietly degrade another. A restructured user profile might increase accuracy on profile-dependent queries while slowing down every call because the prompt is now longer. The gains and regressions live in single-digit percentages, and you cannot see them without measurement.

This is the compounding mechanism that makes context engineering a genuine moat. If you have been running an evaluation pipeline for a year, you have accumulated evidence about which changes help and which hurt across hundreds of edge cases. A competitor starting today cannot replicate that accumulated knowledge. Stanford, SambaNova, and UC Berkeley published research in October 2025 on Agentic Context Engineering, describing systems where agents update their own context playbooks based on what works across tasks. The direction the whole field is moving is toward context that learns and improves over time. Measurement is the precondition for that.

Why This Is the Moat and Why Most Teams Have Not Found It Yet

I want to make a specific argument here and back it up, because ‘this is the moat’ is easy to say and meaningless without the reasoning.

The Model Layer Is Commoditising Fast

The gap between the top closed-source models and the best open-weight alternatives has narrowed every release cycle for the past two years. DeepSeek, Mistral, and Llama variants are landing within months of frontier closed-source quality on most benchmarks. The marginal advantage of using the best available model is shrinking, and the rate of shrinkage is itself accelerating.

Any product whose differentiation reduces to ‘we use the latest model’ has, by definition, a temporary and rapidly expiring advantage. Six months at most before a competitor is on the same model or one that performs equivalently.

The Context Layer Does Not Commoditise

Your context-layer assets are not in any model’s training data. They cannot be replicated by a competitor switching to the same API. They include:

• Your accumulated correction database: Every time a user edited an AI output in your product, that is a labelled example of what right looks like for that specific failure mode in your specific domain. No competitor has access to it.
• Your calibrated few-shot pool: Examples handpicked from your evaluation set, tested against your specific quality bar, refined over months of iteration on real user behaviour. Each one took judgment to select. The whole pool took time to build.
• Your tool descriptions: Rewriting these alone can lift task success by 5 to 20% on the same model. That improvement is now embedded in your product and is not in anyone else’s.
• Your prompt structure: The specific layout that maximises your cache hit rate, your retrieval relevance, your structured output reliability. The result of dozens of experiments you ran and your competitor has not.

Each of these is individually small. Cumulatively they are enormous. And unlike model upgrades, which are a one-time step function, these assets compound. The correction database grows every day. The few-shot pool gets better with every evaluation cycle. The prompt structure improves with every A/B test. A competitor starting today cannot catch up because they are not starting at zero, they are starting behind a moving target.

Why Most Teams Have Not Found It Yet

If context engineering is this important, why is not everyone doing it? Four honest reasons.

First, the work is invisible. A team can spend a quarter improving context engineering and have nothing to demo. No new features. No visible UI changes. Just better outputs in a hundred small ways. That is genuinely hard to justify in a product review against teams shipping flashier work.

Second, it sits in an organisational gap. Context engineering is partly product, what information matters, in what form, and partly engineering, how to retrieve it and cache it efficiently. Product managers assume it is an engineering detail. Engineers assume it is a product decision. Without an explicit owner, it gets done badly or not at all.

Third, the discipline is not taught anywhere. Neo4j’s analysis notes that context engineering only emerged as a formalised concept in mid-2025. There is no university course. No PM training programme. Anyone learning this is learning it on the job from first-party engineering blogs that move faster than any curriculum.

Fourth, the compounding takes time to become visible. The first month of context engineering investment looks unimpressive. The sixth month starts to feel meaningful. The twelfth month is when the gap becomes obvious. Teams that need quarterly wins struggle to justify it. The teams willing to invest anyway are the ones building the actual moat.

What This Means for Your Career and Where the Roles Are

Here is the part most articles on context engineering leave out entirely. Understanding this concept is not just useful for building better AI products. It is one of the most valuable career differentiators in the technical job market right now.

The roles being created around context engineering are real, they are well-compensated, and they are being filled through channels that most people are not watching. Let me get specific.

The Roles That Are Actually Being Created

Based on the patterns visible across AI teams that are doing this well, the roles with genuine demand in 2026 look like this:

• Context architects: people who own the design of what enters the model’s context at each call, how it is structured, and how it is evaluated. Neither a traditional PM nor a traditional engineer. A new kind of role that sits at the intersection.
• Retrieval engineers: people who build and maintain the pipelines that decide what gets pulled from knowledge bases at runtime, with what ranking logic, formatted how. The quality of retrieval is often the single largest driver of output quality in production systems.
• Evaluation engineers: people who build and maintain the eval pipelines that let context changes be measured before they ship. Without this role, context engineering cannot compound. With it, every week of iteration is permanently additive.
• AI product managers with context fluency: PMs who understand KV-cache well enough to know when a product decision will triple serving costs, who own the few-shot pool, who treat the prompt structure as a product asset not an implementation detail.
• Technical writers for AI systems: people who write the tool descriptions, the system prompt language, the few-shot examples. The quality of these artefacts directly determines model behaviour. Most AI teams are genuinely bad at this and know it.

What all of these roles share? They require understanding the problem at a level most people do not have yet. That gap is exactly where the opportunity is.

The Thing That Will Look Obvious in Hindsight

Two years from now I think there will be a role on most serious AI product org charts that does not have a clear name yet. It sits between product management and machine learning engineering. It owns the context layer explicitly. It treats the prompt structure as a product asset, the few-shot pool as a competitive advantage, the evaluation pipeline as an infrastructure investment.

The teams that figure this out first will be the ones whose products feel qualitatively different from everything else in their category, even when they are running on the same underlying models as everyone else. The difference will be invisible from the outside and enormous from the inside.

Right now, there is a window where understanding context engineering clearly is a genuine differentiator because most people and most teams do not understand it yet. That window will close as the concept becomes mainstream and the textbooks catch up.

The model is the visible part of the AI product. It is the part that gets the press releases and the benchmark announcements and the heated debates on social media. Context is the part that actually decides whether the product works, whether it compounds, and whether it is possible to build something a competitor cannot replicate.

Start treating it that way. The compounding starts from the first week you do.

If this reframed how you think about AI products, share it with someone who is still arguing about which model to use.

Disclaimer: This article reflects personal analysis and commentary based on publicly available research and engineering documentation. All statistics are sourced from cited research.