Your agent nailed the task on Monday and forgot the whole conversation by Tuesday. Frustrating, but not mysterious. Most agents have no memory. They have a prompt, and a prompt is not a memory system.
That single confusion causes more agent failures than model quality does. Let us untangle it.
The Rule That Fixes Most of It
Do not make the prompt your memory. Store memory outside the model, then give the model only the parts it needs for the request in front of it.
Think of the context window as a desk, not a filing cabinet. A desk holds what you are working on right now. A filing cabinet holds everything else. Teams that treat the desk as the cabinet end up with an agent buried in paper, reasoning worse as the pile grows.
The Three Kinds of Memory Worth Separating
| Type | What it holds | Lives where | Typical mistake |
|---|---|---|---|
| Working | The current task and recent turns | Context window | Letting it grow until quality drops |
| Episodic | What happened in past sessions | Database or vector store | Storing raw transcripts forever |
| Semantic | Durable facts about the user or domain | Structured store | Confusing a one-off with a standing fact |
Most production systems need all three, and most prototypes have none. The gap between the two is where reliability lives.
Compaction Is a Design Decision, Not a Fallback
When a session runs long, something has to be summarised. Teams usually bolt this on in a panic and lose the thing that mattered.
Good compaction prompts enumerate what must survive: the decisions made, the constraints stated, the identifiers in play, the open questions. Everything else is compressible. Write that list before you need it, not during an incident.
The Handle Pattern
When a tool produces a large result, do not paste it into context. Store it and hand the model a reference it can pass back later. Visible handles beat invisible state, because the model can reason about what it is holding.
Six Failure Modes to Watch
- Everything is semantic. The agent treats a passing remark as a permanent truth about the user.
- Nothing is consolidated. Episodic memory grows forever and retrieval quality quietly collapses.
- Retrieval with no recency weighting. The agent confidently cites a decision you reversed in March.
- Memory without deletion. If a user cannot make the agent forget something, you have a compliance problem, not a feature.
- Shared memory across agents with no scoping. One agent poisons the well for the rest.
- No trace analysis. You cannot fix memory you never inspect.
Where to Actually Store It
The boring answer is usually right. Postgres handles semantic facts and session records comfortably. Redis suits fast working state. A vector store earns its place when you need similarity search over a genuinely large body of past interactions, and not before.
For multi-agent systems, shared memory becomes the coordination layer. An agent re-invoked hours later retrieves its own prior context rather than starting cold. Design the scoping rules before you turn that on.
Conclusion
Reliable long-running agents run on memory that was designed, not memory that accumulated. Separate working, episodic and semantic. Decide in advance what survives compaction. Store outside the model and retrieve narrowly. Do that and your agent stops forgetting Monday, without a bigger context window or a bigger bill.
Frequently Asked Questions
Does a bigger context window solve this?
It delays it. Quality degrades as context fills, and cost rises linearly. Bigger windows buy room, not memory.
Do I need a vector database for agent memory?
Not at the start. Postgres plus a sensible schema handles more than people expect. Add vectors when similarity search is the actual bottleneck.
How do I stop an agent acting on stale facts?
Timestamp everything, weight retrieval toward recency, and give facts an explicit supersede path so a new decision overwrites the old one.