2026-06-20 · Updated 2026-07-22

AI Agent Memory: Schema-Guided State and Provenance

Long-running agents often retrieve stale facts because ordinary semantic memory has no rule for deciding which value is current.

When a user changes a passport deadline from July 15 to June 30, vector search can retrieve both statements. A stateful memory layer must record that the June 30 value supersedes the earlier one.

TL;DR: Treat durable agent memory as typed application state. Extract memory candidates through a structured-output boundary. Store records with tenant scope, validity windows, supersession, provenance, and schema versioning, then retrieve the smallest current slice on the read path. Reserve vector search for fuzzy recall. Read mutable facts from scoped, current records.

The context-window trap

The context window is the input available to one model call. An application may carry messages into later calls, but application policy must decide which old facts remain true and who may see them.

Long-running agents need to remember preferences, task status, customer facts, tool decisions, compliance notes, and prior mistakes. The easy version is to append summaries or dump old notes into a vector store. That works until one of the remembered facts changes.

Now the agent has two passport deadlines, two preferred formats, or two project decisions. Semantic search may retrieve both. A summary may overwrite one. A long context may include the stale one next to the active one. These designs recall text while leaving the current value unenforced.

The memory contract has to answer concrete questions:

• What is true now?
• What was true on June 2?
• Who said it?
• Which tenant does it belong to?
• Which older fact did this fact replace?
• Can I delete or expire it?

Schema-Guided Agent Memory (SGAM) stores those answers as fields and relations instead of leaving them implicit in prose.

What SGAM means

Three similarly named ideas establish the scope of SGAM.

Schema-Guided Dialogue (SGD) is the 2019 Google task-oriented dialogue dataset. Its schema describes service APIs, intents, and slots so a dialogue model can track state for services it has not seen before. It is useful precedent for schema-based tracking, with a scope limited to dialogue services.

Schema-Guided Memory (SGM) is the research term used by Mei et al. in According to Me: Long-Term Personalized Referential Memory QA. The paper compares free-text Descriptive Memory (DM) with fixed-schema key-value memory items. Both representations contain the same source information in different structures.

In this article, I use Schema-Guided Agent Memory (SGAM) to mean an engineering pattern in which schemas govern writes, updates, retrieval, and deletion. The schema defines application state and its lifecycle.

ATM-Bench shows why the representation matters. It uses roughly four years of personal data from emails, images, and videos. The questions require personal references, location, multiple pieces of evidence, and updates over time. Performance drops on the hard split, while SGM improves over DM because the retriever can address fields such as time, source, location, entities, and tags directly.

SGM versus DM answers a storage question: should memory remain free text, or should it use named fields? A production agent has another problem before storage. It must reason from an unstructured conversation to a proposed memory update. Schema-Guided Reasoning (SGR) defines that decision path: inspect the evidence, identify the subject and attribute, check whether the fact changes existing state, then produce a candidate write. SGAM applies the storage and lifecycle rules after that model call.

Separate model extraction from memory ownership

The memory write crosses three layers. Structured Output (SO) enforces the candidate object’s shape. Schema-Guided Reasoning (SGR) encodes the steps and order the model must follow to reach that candidate. Schema-Guided Agent Memory (SGAM) manages the candidate as durable state after the model call.

A verdict, route, or plan usually expires with the current request. Another run may read a memory candidate days later or use it to choose a tool call. That longer lifetime demands storage rules that SGR doesn’t provide.

SGR constrains one model call by defining its reasoning topology. For a memory write, the schema might require source evidence, a normalized subject and attribute, a comparison with current state, and only then the proposed update. Pydantic or JSON Schema describes that path. Provider-native Structured Output or a guided decoding runtime such as XGrammar prevents the model from skipping fields or returning a different shape.

The schema cannot guarantee a correct conclusion. It makes the required decision path explicit and inspectable, including the evidence and comparison that produced the candidate.

SGAM decides what happens after that object exists. Should it be stored? Does it supersede an older fact? Which tenant can see it? Is it current or historical? Which source episode backs it?

The table sets out the ownership and failure mode of each layer:

On the write path, the sequence is:

SGR reasoning schema + Structured Output -> candidate write -> SGAM policy and persistence

The following illustrative excerpt from memory_models.py defines the object passed from extraction to the SGAM write service:

from datetime import datetime
from pydantic import BaseModel, Field


class MemoryDelta(BaseModel):
    tenant_id: str = Field(description="Isolation boundary, e.g. acme")
    subject: str = Field(description="Normalized entity ID, e.g. mira")
    attribute: str = Field(description="Property being updated")
    value: str = Field(description="New value")
    valid_from: datetime
    source_episode_id: str

MemoryDelta captures what the model extracted. The SGAM write service still decides whether to reject, merge, or store it.

Write and read paths have different jobs

Only the write path mutates stored state. The read path selects records for the current request.

The ingestion flow is the write path:

1. Capture a raw episode from messages, tool results, or business events.
2. Extract typed candidates through structured output.
3. Validate the schema and reject malformed writes.
4. Reconcile conflicts, close stale facts, and keep provenance.
5. Commit the record to the SGAM store.

The request flow is the read path:

1. Start with the user question.
2. Decide whether the question needs current state or point-in-time state.
3. Filter by tenant, memory type, subject, attribute, and validity window.
4. Add vector or graph expansion only if the exact state lookup is not enough.
5. Assemble the smallest cited context for the model.

Read that diagram from left to right in two lanes. The top lane writes memory, and the bottom lane reads it. Both use the same store.

What belongs in a memory schema

A minimal SGAM record needs more than text.

tenant_id
memory_id
subject
attribute
value
memory_type
schema_version
valid_from
valid_to
supersedes_memory_id
source_episode_id
confidence
retention_policy

With these fields, a new passport deadline can close the previous deadline without erasing history. The same table can answer current and point-in-time queries, then trace the result to its source episode. schema_version supports migrations, while retention_policy tells deletion jobs what else they must remove.

Use RAG to retrieve documents and SGAM to maintain mutable state. Vector search still belongs in the system for fuzzy recall, clustering, and expansion. The current value of mira.passport_deadline should come from a scoped memory record rather than whichever chunk happened to rank first.

A stale-fact example

Consider a synthetic two-episode trace represented as a DM baseline and an SGAM ledger:

e1: Mira prefers concise answers. Her passport deadline is 2026-07-15.
e2: Mira corrected the deadline. It is now 2026-06-30.

A DM baseline keeps both episodes as free text, so text search can return e1 because it contains the right words. SGAM extracts a typed fact from each episode, keyed by tenant, subject, and attribute. It should return e2 as current state and keep e1 for a historical query.

The following function is SGAM write-path code. It would live in a store module such as sgam_store.py. DM has no counterpart because it doesn’t maintain one current row per attribute. Before the caller inserts a replacement, this function closes the current row:

def close_previous_fact(db: sqlite3.Connection, fact: MemoryFact) -> int | None:
    row = db.execute(
        """
        select fact_id
        from memory_facts
        where tenant_id = ?
          and subject = ?
          and attribute = ?
          and valid_to is null
        order by valid_from desc
        limit 1
        """,
        (fact.tenant_id, fact.subject, fact.attribute),
    ).fetchone()
    if not row:
        return None

    db.execute(
        "update memory_facts set valid_to = ? where fact_id = ?",
        (fact.valid_from, row["fact_id"]),
    )
    return int(row["fact_id"])

When e2 arrives, the function sets valid_to on the fact extracted from e1 to e2’s timestamp. The caller then inserts the new fact with an open valid_to. DM has no equivalent update step, so the old text can still outrank the correction.

Running current and historical deadline queries against both representations produces different results:

Naive text memory:
  returned episode: e1 -> passport deadline is 2026-07-15

SGAM current state:
  mira.passport_deadline = 2026-06-30
  valid_from=2026-06-03T10:00:00Z, source=e2

SGAM point-in-time state:
  on 2026-06-02, mira.passport_deadline = 2026-07-15

In production, pair this transaction with structured extraction on the write path. The database transaction updates temporal validity. The model extracts a candidate fact but doesn’t decide which stored row remains current.

Storage choices follow the retrieval pattern

Projects use several names for parts of this pattern: memory stores, context graphs, profiles, long-term stores, graph RAG, and stateful agents.

Graphiti is a concrete open-source implementation of relational, temporal memory. It tracks facts as they change, keeps pointers to source episodes, and supports hybrid retrieval. LangGraph separates thread checkpoints from cross-thread stores. Mem0 packages memory operations as a managed service. Letta uses editable context blocks rather than field-level SGAM, but it still treats agent state as persistent data.

Start with the data model. If exact fact lookup is the main operation, a relational table with JSON payloads, validity columns, tenant indexes, and a vector sidecar is usually enough. Add a graph when relationship traversal is part of the product, not because the graph demo looks impressive.

Build the write path before the graph

First decide what the product is allowed to remember. The graph-versus-vector choice comes later.

A support agent might remember account tier, open cases, and durable contact preferences. It shouldn’t promote every frustrated aside into profile state. A coding agent might remember repo conventions and unresolved tasks. It shouldn’t keep a private note forever because that note happened to be retrieved once.

Start with the write path and treat memory as a small state mutation:

1. Name the memory type, subject, tenant scope, and retention class.
2. Extract candidate records with structured output.
3. Validate the payload with Pydantic or the schema layer your stack already uses.
4. Resolve conflicts before insert, including whether the new record supersedes an old one.
5. Keep a source pointer to the raw episode, tool result, file, ticket, or user confirmation that produced the record.
6. Write the schema version with each record instead of leaving it only in application code.

The first SGAM store can be a relational table with a JSON column and a few indexes. A graph becomes useful when the product needs to traverse relationships such as customer-to-account, account-to-policy, task-to-artifact, or project-to-decision.

Hot path and background writes

Immediate extraction is worth it when the next turn depends on the new memory. If the user says “remember that I prefer short answers,” the system should not need a nightly job before it behaves differently.

Most turns don’t need an immediate write. Save the raw episode with tenant, session, and tool metadata, then let a background worker extract candidates later. With recurrence-based consolidation, the worker buffers weak signals and promotes a fact only after similar evidence repeats or the user confirms it. This adds freshness lag. That is acceptable for “user often asks for CSV exports” and risky for “customer changed the delivery address.”

Keep the read path deterministic. Enforce tenant scope and validity first, then use fuzzy retrieval only when it can add useful context.

1. Filter by tenant, memory type, and validity window.
2. Retrieve exact structured state before semantic neighbors.
3. Use vector or graph expansion for supporting evidence, related entities, and examples, not as the authority for current facts.
4. Assemble the smallest cited context that can answer the question.

Treat schema migration as a product change because it alters what the agent can recall, cite, or delete. It can also change which historical facts count as current. Plan migration scripts, backfills, dual-read windows, and deletion behavior in the same release.

When SGAM is worth the complexity

Use SGAM when facts can change over time:

• user preferences that can be updated or revoked
• customer or account facts with audit requirements
• task state for long-running assistants
• coding-agent project memory
• multi-agent shared state
• compliance notes where provenance matters
• temporal questions such as “what did we believe before the migration?”

SGAM is overkill when memory is short-lived, exploratory, or cheap to recompute. If the agent only needs a few turns of continuity, a checkpoint and trimmed message history are enough. Static document QA may need only RAG. And if the domain is so unsettled that the schema changes every day, typed memory will slow the team down.

Evaluation checklist

Evaluate the memory lifecycle as well as the final answer. A system can produce a plausible response after it wrote the wrong fact, retrieved a stale one, or crossed a tenant boundary.

I use the same stage-by-stage split as my RAG evaluation article. Measure the stage where a failure can happen rather than limiting evaluation to the generated text. The trace discipline from the agent evaluation article also applies because a memory bug often appears in the run history before it reaches the answer.

I would test SGAM with replay. Feed a fixed sequence of episodes into the memory writer and inspect the ledger after each meaningful turn. Then ask current-state and point-in-time questions against the resulting store.

Benchmarks such as LoCoMo, LongMemEval, and ATM-Bench provide public test cases. They don’t replace a domain test suite. A coding assistant, customer support bot, and compliance copilot need different schemas, filters, retention rules, and failure tests.

Caveats

SGAM is my label for a pattern, not a standard. Existing projects divide the problem differently. LangGraph memory and LangMem describe short-term and long-term stores, profiles, collections, hot-path writes, and background memory managers. Zep Graphiti uses the term temporal Context Graph. Letta persists editable memory blocks, while Mem0 offers a managed memory layer. Microsoft GraphRAG, LlamaIndex property graphs, and Cognee frame related parts of the problem as knowledge graphs.

A user profile, an episode log, a document graph, and an agent-editable memory block solve different retrieval and update problems. I reserve SGAM for durable memory that represents current application state and therefore needs schema, validity, provenance, conflict handling, retention, and migration.

Typed memory can still be wrong. A schema makes bad writes easier to inspect; it does not make them trustworthy. You still need source trust, user confirmation for sensitive facts, conflict policy, deletion, and monitoring.

Schema migration is work. Once memory becomes state, you own versioning, backfills, old records, and deletion behavior. Skip that work and old records will outlive the semantics or retention policy that created them.

References

• According to Me: Long-Term Personalized Referential Memory QA - Mei et al. paper introducing ATM-Bench and Schema-Guided Memory.
• Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue Dataset - Rastogi et al. paper on the SGD dataset.
• LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory - Wu et al. benchmark for long-term memory abilities.
• Graphiti: Build Temporal Context Graphs for AI Agents
• Zep: A Temporal Knowledge Graph Architecture for Agent Memory
• Mem0 Platform Overview
• Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
• LangGraph Memory Overview
• LangGraph Persistence
• LangMem documentation
• Letta: Introduction to Stateful Agents
• Microsoft GraphRAG documentation
• LlamaIndex: Using a Property Graph Index
• Cognee Documentation
• Pydantic model validation docs
• Python sqlite3 documentation