Agent Memory and Context: What to Store, What to Retrieve, What to Forget
Memory is one of the most misunderstood parts of agentic AI.
Teams often say “we need memory” when they really need one of these:
- Better retrieval from documents
- Consistent storage of customer preferences
- A way to keep a multi-step task coherent
This article breaks memory into practical categories.
Short-term context: the working set
Short-term context is what the agent needs to complete the current task.
Examples:
- The current ticket thread
- The customer’s plan level
- The last two tool outputs
Short-term context should be:
- Minimal
- Fresh
- Task-specific
Too much context increases confusion.
Long-term memory: store facts, not stories
Long-term memory should be structured.
Good examples:
- Preferred contact channel: email
- Approved discount policy: 10%
- Integration environment: staging
Bad examples:
- A long narrative of every conversation
Narratives are hard to validate and hard to keep consistent.
Retrieval: make it boring and reliable
Most “memory” problems are retrieval problems.
Good retrieval systems:
- Use clear document chunking
- Store source references
- Filter by customer, project, or product
- Prefer authoritative documents
If you retrieve irrelevant context, the agent will make confident mistakes.
What to forget
For safety and performance, decide what should not persist.
Examples:
- Temporary tokens
- One-off mistakes
- Sensitive fields that are not required
Forgetfulness is a feature when it reduces risk.
Closing thought
Treat memory like product data.
Define schemas, ownership, and validation. Your agents will become more accurate and easier to trust.