Selective truncation and summarization can improve LLM performance by up to 50% and cut API costs by as much as 93%. That means better AI context management often comes from removing information, not buying a larger context window.
The counterintuitive part is that capacity creates a new failure mode. Models can accept more material than ever, but an agent that receives every document, tool response, conversation turn, and memory entry still has to decide what matters. Irrelevant or stale context competes with authoritative evidence, increases retrieval ambiguity, and gives state drift more room to develop.
The modern context-window era is often traced to the Transformer paper published in June 2017, which established that attention cost scales quadratically with sequence length. Public windows then expanded from GPT-1's 512 tokens in 2018 to GPT-2's 1,024 in 2019, GPT-3's 2,048 in 2020, and ChatGPT-era GPT-3.5's 4,096 in 2022. GPT-4 variants reached 8,192 and 32,768 tokens in 2023, while GPT-4.1 reached 1,047,576 tokens by April 2025 and Meta advertised Llama 4 Scout with 10,000,000 tokens, as documented in this history of context windows.
A large window is useful infrastructure. It isn't a memory policy, an authorization boundary, or a guarantee that the model will use evidence correctly. Production systems need active curation, structured state, versioning, and compaction.
The Capacity Illusion in AI Context Management
Teams usually reach for a larger context window after an agent misses something. That response is understandable, but it confuses storage capacity with usable context. A model may accept a million tokens while still giving weak attention to the one policy clause, account fact, or tool result that determines the correct action.
The distinction became especially important as windows accelerated. Epoch AI reported that the longest LLM context windows were growing by about 30x per year since mid-2023, while Google moved Gemini from an earlier 32,000-token limit to a 1 million-token window for Gemini 1.5 Pro in February 2024 and said it had tested 10 million tokens in research. By 2026, roughly million-token systems had become a commercial baseline, according to Epoch AI's context-window analysis.
Why bigger prompts create weaker decisions
Long context can dilute attention, bury contradictory instructions, and preserve obsolete assumptions alongside current facts. A customer support agent might receive the customer's latest request, an old refund policy, several irrelevant conversation turns, and a tool response containing account data. Nothing is technically missing, yet the model can still select the wrong authority.
The practical unit of optimization isn't the maximum prompt size. It's the minimum relevant context needed for a specific decision. Progressive disclosure works better than indiscriminate loading: expose general policy first, then reveal account-specific tools or jurisdiction-specific rules only after the relevant condition is established.
Practical rule: Treat every token as a competitor for attention. Retrieve and deliver only the evidence the next action requires.
The COLING research cited in the paper on selective truncation and summarization gives this approach a measurable basis. Its reported results reach up to 50% higher performance, as much as 93% lower API costs, and lower latency under the tested conditions. Those results don't prove that every production workload will achieve the same outcome, but they do challenge the assumption that retaining all available context is safer.
Teams designing agents can also use practical guidance on how to build AI features with context engineering, especially when deciding which information belongs in the prompt, a tool, or an external memory store. The right architecture usually treats the model window as a working surface, not the system of record.
Architecting Persistent State for Multi-Session Agents
An agent that spans sessions needs more than a transcript. It needs a state model that distinguishes what is temporarily relevant, what has been confirmed, and what can be retrieved later as evidence.
A useful architecture separates three layers:
| State layer | What it contains | Retention behavior |
|---|---|---|
| Ephemeral working memory | Current request, active tool results, intermediate reasoning inputs, unresolved questions | Cleared or compacted after the task |
| Compact persistent state | Confirmed intent, constraints, preferences, decisions, and open work | Versioned across the session |
| External long-term memory | Source documents, prior episodes, account history, policies, and durable facts | Retrieved only when authorized and relevant |
Ephemeral working memory
Working memory should be disposable. It can contain a tool response awaiting validation, a temporary hypothesis, or a short-lived interpretation of an ambiguous request. Writing all of it into durable memory creates state drift, because provisional assumptions begin to look like user preferences or organizational truth.
A practical state update should distinguish facts from interpretations:
facts: verified account status
constraints: refund requires authenticated account
preferences: customer prefers email
unresolved: reason for cancellation
provenance: source and timestamp for each field
The format matters less than the separation. A generated summary shouldn't silently become policy.
Compact persistent state
Persistent session state should preserve decisions that affect future turns without retaining the entire transcript. For a support workflow, that might include the authenticated customer, the issue category, the product involved, the approved remedy, and the next required step.
Structured intent earns its place here. An intent layer can capture facts, constraints, preferences, unresolved dimensions, and provenance so downstream components don't have to infer the same state repeatedly from conversational prose. The AI knowledge base architecture guide is useful background for separating authoritative knowledge from conversational memory.
External memory
Long-term memory should remain outside the prompt and behind retrieval controls. Store source identity, tenant identity, timestamps, validity status, version, and permitted scope alongside each item. Retrieval should then assemble a task-specific evidence set rather than dump a complete history into the model.
A user prompt, attached file, retrieved passage, tool response, or memory entry is data, not a trusted instruction. Microsoft's retrieval hygiene guidance for AI defense recommends source provenance, permission-aware indexing, and validation at every read and write. That distinction prevents untrusted content from becoming durable authority.
Mitigating State Drift Through Structured Protocols
State drift appears when a system carries forward an assumption after its original basis has changed. The agent may start with a correct interpretation, summarize it loosely, retrieve a stale policy, and then treat the combination as one coherent truth. More turns don't repair the problem. They often make the incorrect state harder to identify.
A durable fix requires protocols that make state transitions explicit.
Five controls that stabilize state
Re-anchor intent at each turn. Extract the current objective, relevant entities, constraints, and unresolved questions before selecting tools or memories. Don't assume the previous turn still defines the task.
Validate every state update against a schema. Reject updates that lack required provenance, authority, tenant scope, or validity status. Free-form “memory writing” is convenient during prototyping and difficult to audit later.
Freeze approved snapshots. Once a policy decision, customer authorization, or workflow checkpoint is approved, store an immutable version. Later model output can propose a change, but it shouldn't overwrite the approved state without approval.
Timestamp and version memory entries. A memory without temporal metadata can't reliably compete with a newer source. Retrieval should expose freshness and validity so the generation layer can prefer current authority.
Run drift audits. Compare the current structured state with the original request, approved policy, and supporting evidence. Flag contradictions instead of asking the model to reconcile them invisibly.
A useful controller loop is:
acquire → classify → authorize → validate → compact → deliver → observe → retire
The controller decides what enters working memory, what becomes persistent state, and what must be discarded. This matches the emerging definition of agentic context management as deciding what an agent should hold, when, for how long, and at what cost. It also supports the contrarian conclusion from recent work: some memory should be actively pruned because stale context can degrade long-horizon performance.
Authority before autonomy
Models can propose actions. They shouldn't decide which policy, identity, or tenant boundary grants them authority. Authenticated policy and canonical state must control those decisions.
For example, an agent can suggest a refund after reading a customer's request. A policy service should determine whether the account is authenticated, whether the transaction qualifies, and whether the acting principal can perform the operation. The model then receives the authorized result and explains it.
This separation also makes resets safer. When a session accumulates contradictory or unvalidated assumptions, discard the working layer and reconstruct it from compact state plus authoritative retrieval. Preserve approved facts, not every intermediate thought.
Frozen evaluations make these controls testable. Define the task, evidence set, state schema, version, denominator, and limitations before changing the controller. Otherwise, an apparent improvement may come from a changed test set or a narrower claim rather than a better context policy.
Comparing Episodic Versus Semantic Memory Strategies
Episodic and semantic memory solve different problems. Choosing between them isn't a matter of picking the more refined approach. It depends on whether the agent must reconstruct what happened or answer from what is generally true.
| Dimension | Episodic memory | Semantic memory |
|---|---|---|
| Primary purpose | Preserve specific events, turns, and actions | Preserve distilled facts, concepts, and rules |
| Strength | High-fidelity continuity | Stable, compact grounding |
| Risk | Staleness, duplication, retrieval noise | Loss of temporal detail and causality |
| Best fit | Support history, incident timelines, user commitments | Product knowledge, policies, definitions, account facts |
Episodic memory preserves continuity
A support agent needs episodic memory when the customer says, “I already completed that verification,” or asks for the status of a prior escalation. The exact sequence can matter. A summarized fact such as “customer had a billing issue” may omit the failed remedy, promised follow-up, or unresolved dispute.
The cost is noise. Episodes contain conversational phrasing, temporary assumptions, irrelevant turns, and facts that were true only at the time. Store them with event time, source, participants, tenant, and validity. Retrieve a narrow episode window when the task depends on chronology, rather than embedding every exchange into a single undifferentiated memory collection.
Semantic memory reduces token load
Semantic memory distills recurring knowledge into retrievable facts and concepts. It works well for product capabilities, current policies, terminology, and validated customer preferences. Because it stores abstractions rather than full conversations, it usually creates a smaller evidence package for generation.
But distillation can erase qualifiers. “Customer prefers email” may be useful, while “customer prefers email for billing notices but requested phone contact for the active fraud case” requires context and scope. Semantic writes therefore need provenance and a rule for when a newer episode supersedes an older fact.
Use episodic memory for continuity, semantic memory for grounding, and never let either one bypass authorization.
A hybrid design usually performs better operationally. Retrieve semantic facts first to establish the current baseline, then fetch selected episodes when the request involves history, commitments, or disputes. If the two conflict, expose the conflict to a controller or human reviewer. Don't ask the generator to quietly choose.
The retention policy should be explicit. Keep an episode when it supports an unresolved obligation, legal or operational audit, or active workflow. Compact it when only a durable fact remains. Retire it when its validity expires or when the source authority invalidates it.
Building a Governance Layer for Safe Retrieval
Memory quality doesn't compensate for weak governance. A system can retrieve the most relevant document in its database and still produce an unsafe answer if the document belongs to another tenant, contains obsolete policy, or was never authorized for the current action.
Safe retrieval needs a fail-closed path:
request → establish identity → apply tenant scope → authorize query
→ retrieve candidates → validate provenance and status → deliver evidence
→ generate bounded answer or abstain
Put authorization before retrieval
Tenant isolation must be independent of authentication and authorization. A user may be correctly authenticated and authorized for their own resources while still receiving another tenant's data if the retrieval layer doesn't enforce tenant scope.
Apply the tenant identifier before retrieval, not after generation. Store it on every episode, memory item, and query scope. The same boundary must extend to vector indexes, caches, logs, evaluation sets, tool calls, and usage caps. Multi-tenant isolation guidance for AI memory describes this as a boundary condition, not an optional filter.
Trace evidence through the answer
A provenance-aware retrieval trace should connect:
- Trigger: the query, intent, or context that initiated retrieval
- Evidence: retrieved memory items and original sources
- Signals: relevance, validity, authority, tenant, and version
- Downstream use: claims, tool calls, actions, final answers, and later memory updates
This trace lets an engineer answer a difficult production question: Which evidence caused the agent to make this claim? The provenance-aware retrieval trace specification provides a useful model for linking retrieval to downstream behavior.
GQueries and IntentParse represent complementary infrastructure patterns here. IntentParse can convert raw language into structured intent and unresolved dimensions, while GQueries can provide persistent memory, authorized retrieval, and fail-closed grounding. AletheionAGI positions these as a grounding and evidence-control layer that works with existing LLMs, RAG pipelines, vector databases, and memory systems, rather than replacing them.
A team building an internal knowledge system may also find it useful to deploy an internal knowledge base with explicit ownership and source controls. The important design choice isn't the vendor. It's whether the system preserves authority, scope, provenance, and lifecycle state.
For business teams, the meaning of guardrails in business is best expressed operationally: controls should constrain what the model can access and do, not merely ask it to behave better in prose.
Failure Modes and Production Testing
The most dangerous context bug can look like a successful answer. A customer asks about an account-specific entitlement, retrieval returns a plausible document, and the agent responds confidently. Only later does an engineer discover that the document came from another tenant or an obsolete policy namespace.
A simple isolation test catches one class of this failure. Store a nonce fact under tenant A, issue a semantically similar query under tenant B, and verify that tenant B cannot recall it. Run the test through the actual production path, including caches, memory stores, retrieval filters, tools, logs, and fallback routes. Testing only the vector query is insufficient if a later cache lookup ignores tenant scope.
What to test before launch
- Cross-tenant recall: Tenant B must not retrieve, summarize, cite, or infer tenant A's nonce fact.
- Stale authority: An invalidated policy must not outrank a current authorized source.
- Prompt injection: Retrieved content must remain data unless a trusted policy explicitly promotes it.
- Provenance continuity: Every material claim should map to evidence, or the system should mark it unsupported.
- Abstention: Missing or conflicting evidence should produce a bounded uncertainty response, not a fabricated conclusion.
- Reset behavior: A poisoned working session should be discardable without writing its assumptions into durable memory.
The multi-tenant AI agent testing guidance recommends establishing tenant identity once per request from a non-forgeable runtime signal, such as a verified token or trusted backend key, and enforcing it across the full request lifecycle. That is stronger than trusting a tenant ID supplied in the prompt.
A retrieval miss is recoverable. A plausible answer built from unauthorized evidence is an incident.
Measure more than answer quality. Record whether the agent used authorized evidence, whether it cited a valid source, whether it called the correct tool, whether it attempted an action outside policy, and whether it updated memory appropriately. The testing patterns in this guide to AI agent testing help turn those concerns into repeatable checks.
Teams evaluating a broader production-ready agentic AI architecture should require the same controls at orchestration boundaries. A secure retriever paired with an unscoped tool router still leaves a gap.
The operational standard is simple: authorized evidence in, bounded action out. If the system can't establish identity, validate evidence, or preserve provenance, it should stop and ask for clarification or human review.
AletheionAGI provides infrastructure for persistent memory, structured intent, authorized retrieval, provenance, and fail-closed grounding across existing AI applications. Visit AletheionAGI to evaluate how its composable products can help your team preserve useful state while limiting drift, unsupported claims, and cross-tenant leakage.
