Context Engineering for AI Agents: Keep Context, Memory and Knowledge Apart
Most AI agent failures are context failures, not model failures. How to design what an agent knows and forgets, with a hospital example and code you can adapt.
Written byMuhammad Abdullah Chaudhary
When an AI assistant gives a confident wrong answer, people usually blame the AI model or try rewording the question. In our experience the real cause is usually simpler: the assistant was given the wrong information, too much of it, or information that was out of date.
Think of a doctor starting a night shift. They don't read every record in the hospital. A good handover gives them the few things that matter right now: who is on the ward, what changed today, and which rules apply. AI agents (AI assistants that can look things up and take actions) need the same kind of handover. Designing it is called context engineering.
A recent InfoQ article by Adi Polak, Systemic Approach to Memory, Knowledge, and Context in Agentic AI Architectures, makes this case for engineers. This post explains it in plain terms, with an example from a hospital, and a short code sample for developers who want to try it.
The short version
- AI gets things wrong most often because of what it was given, not because the model is bad.
- Keep three kinds of information apart: what is happening now, what the AI remembers, and the fixed rules.
- Give the AI less, but better: only what matters for this task, with dates and sources.
- Decide what the AI is allowed to see before it sees it, not afterwards.
Three kinds of information
It is tempting to put everything an AI might need into one big pile and let it search. That works in a demo and breaks in real use, because these three kinds of information behave very differently.
| In plain words | How long it stays useful | Example on a hospital ward | |
|---|---|---|---|
| Context | What is on the desk right now | One question | The shift being planned tonight |
| Memory | What the AI remembers from earlier conversations | Minutes to months | This coordinator's usual preferences |
| Knowledge | The rulebook | Until someone changes the rules | Staffing ratios, required qualifications |
Keep them apart and each can be stored, updated and protected in its own way. Mix them and an old note about someone's preferences can end up competing with a legal staffing rule.
Better wording is not enough
Most advice about AI focuses on the prompt: the instructions you type. Good wording helps, but it cannot fix missing or outdated information. Context engineering decides which information the AI receives in the first place.
In practice this is one step, which we will call the context engine, that runs before every question reaches the AI. It works like the handover: it picks what matters, removes what this person is not allowed to see, trims it down, and lays it out clearly.
Because everything passes through one step, there is also one place to look when the AI gets something wrong.
One question, two very different answers
A ward coordinator asks a scheduling assistant: "Who can cover the Ward 3 night shift tomorrow?"
Without a context engine, a typical system finds the 20 documents that sound most similar and sends them all:
You are a helpful scheduling assistant. Here is some relevant information:
<20 documents that look similar to the question>
- Ward 3 night roster, November 2025
- Staffing policy v2 (draft, later replaced)
- Full staff list, including people on leave tomorrow
- "R. Khan prefers night shifts" (note from March)
- Ward 3 day-shift handover notes
- ...
Who can cover the Ward 3 night shift tomorrow?
None of it is false, but most of it is old, off-topic or contradictory. The AI has to guess which policy applies, and it cannot tell who is on leave. The likely result is an answer that sounds right and is wrong, and nobody can see why.
With a context engine, the same question arrives like a proper handover:
[RULES: knowledge, revalidated 2026-11-01]
- Every night shift needs at least one RN with Ward 3 competency.
- Minimum 11 hours rest between shifts.
[CURRENT STATE: live, as of 14:05 today]
- Ward 3 night, 2026-11-13: 1 of 2 RN slots filled.
- On leave tomorrow: S. Tanaka, A. Malik.
[CANDIDATES: filtered by competency, rest and leave]
- R. Khan (RN, Ward 3, last shift ended 07:00 today)
- M. Sato (RN, Ward 3, off today)
[MEMORY: this coordinator]
- Avoid scheduling anyone for more than two nights in a row.
[TASK]
Suggest one RN for the open slot. Use only the facts above and say which
rule each suggestion satisfies. If no one qualifies, say so.
It is shorter, and every fact has a date and a source. The strict checks (who is on leave, who has rested enough, who is qualified) were done by ordinary software before the AI saw anything. The AI only has to choose and explain, which is what it is good at. And the coordinator still makes the final decision, as in our care scheduling work.
Five rules of thumb
1. Pick by purpose, not by resemblance
Many AI systems fetch whatever sounds similar to the question. Similar is not the same as useful. Also filter by who is asking, what they are trying to do, and how recent the information is. Combining a few search methods with direct links (for example, "this patient's current care plan") works far better than similarity alone.
2. Prepare the rulebook in advance
Handing the AI raw chunks of long documents is like handing a new employee a box of files. Do the preparation beforehand: turn documents into short summaries and clear rules. A school's assessment policy should reach the AI as the five rules that matter, not as a forty-page PDF cut into pieces.
3. Let memory expire
An AI that remembers everything forever slowly fills up with outdated facts. Give each kind of memory a lifetime:
- What is happening right now (live readings, results from other systems): minutes to hours.
- The current conversation or task: until it ends.
- Long-term memory and rules: kept, but checked again on a regular schedule.
When something changes (a bed is reassigned, a student moves class), update the AI's copy straight away rather than waiting for it to go stale.
4. Less is more
Longer inputs make AI slower, more expensive and less accurate. Research has found that AI models are noticeably worse at using information buried in the middle of a long input. Set a limit on how much goes in, leave out anything only loosely related, put the rules first and the question last, and label each section as in the example above.
5. Lock the door before, not after
If someone is not allowed to see a record, the AI should never receive it. That is far safer than giving it the record and asking it to keep a secret. It is the same idea behind our on-premise LLMs for regulated data: the AI works with the user's own permissions.
For developers: a minimal context engine
Not technical? Skip to the next section.
The core of the engine is smaller than it sounds. This TypeScript sketch covers rules 3 to 5: permission checks, expiry, a relevance cut-off, a size limit (a "token budget") and labelled sections.
type Item = {
kind: "knowledge" | "live" | "memory";
text: string;
source: string;
roles: string[]; // who may see this item
updatedAt: Date;
expiresAt?: Date;
score: number; // relevance from your search step, 0 to 1
};
const LABELS = { knowledge: "RULES", live: "CURRENT STATE", memory: "MEMORY" };
const tokens = (s: string) => Math.ceil(s.length / 4); // rough estimate
export function buildContext(items: Item[], role: string, task: string, budget = 2000) {
const now = new Date();
const candidates = items
.filter((i) => i.roles.includes(role)) // safeguard first
.filter((i) => !i.expiresAt || i.expiresAt > now) // drop stale memory and signals
.filter((i) => i.score >= 0.5) // relevance threshold
.sort((a, b) => b.score - a.score);
const kept: Item[] = [];
let used = tokens(task);
for (const item of candidates) {
if (used + tokens(item.text) > budget) continue;
kept.push(item);
used += tokens(item.text);
}
const blocks = (Object.keys(LABELS) as Item["kind"][]).map((kind) => {
const lines = kept
.filter((i) => i.kind === kind)
.map((i) => `- ${i.text} (${i.source}, ${i.updatedAt.toISOString().slice(0, 10)})`);
return lines.length ? `[${LABELS[kind]}]\n${lines.join("\n")}` : "";
});
return [...blocks, `[TASK]\n${task}\nUse only the facts above. If they are not enough, say so.`]
.filter(Boolean)
.join("\n\n");
}
It leaves out the search step that produces score (ideally a mix of semantic and keyword search over prepared knowledge), summarising when the limit is tight, and saving results back to memory with an expiry. Those can grow over time. What matters is that every call to the model goes through one function you can read, test and log.
What this looks like in practice
| Where | The rulebook | What it remembers | What is happening now | What goes wrong without it |
|---|---|---|---|---|
| Hospital and care scheduling | Staffing rules, qualifications | Coordinator preferences, past rosters | Today's absences | A shift that breaks a staffing rule |
| School and workplace learning | Curriculum, assessment policy | Each learner's progress | Today's lesson or question | Material at the wrong level |
| Business CRM assistants | Product catalogue, pricing rules | Customer history | The open deal and latest emails | A quote from an expired price list |
Why AI makes things up
When AI states something false with confidence, it is often called a "hallucination". The InfoQ article argues this is almost always caused by badly managed input. We would say it slightly more carefully: AI models do still make mistakes. But in the systems we have reviewed, the most common cause of a wrong answer is missing, outdated or contradictory information, exactly like the first example above. Give the AI a small set of reliable facts, tell it to use only those, and its answers improve without changing the model.
Even then, good input does not remove the need for a person to check. In healthcare and other regulated settings we always pair it with someone who confirms the result, as described in Human-in-the-Loop AI: Draft, Don't Send.
Where to start
- Write down everything your AI reads today, and sort each source into "happening now", "remembered" or "rulebook".
- Put a date and an expiry on everything it remembers.
- Send every question through one handover step, and keep a record of exactly what it passed to the AI.
- Test with the same set of questions before and after each change, the approach we used in our agentic AI CRM research programme.
AI becomes dependable when the information it works from is managed as carefully as any other part of the system. The model matters, but what you give it matters more.
Sources and further reading
- Systemic Approach to Memory, Knowledge, and Context in Agentic AI Architectures, Adi Polak, InfoQ
- Lost in the Middle: How Language Models Use Long Contexts, Liu et al., arXiv
- Model Context Protocol, Anthropic and contributors
- AI Risk Management Framework, NIST
Related case study
Agentic AI on a Live CRM: An Evidence-Based Research Programme
