M* (NeurIPS 2026): Every Task Deserves Its Own Agent Memory System

Nov 13, 2026 · #research #agent-memory · 阅读中文版

TL;DR: Most agent memory today is an off-the-shelf design (vector search, Mem0, ReasoningBank) reused for every task. M* (NeurIPS 2026) instead writes the memory system as an executable Python program and searches for it separately on each task with reflective code evolution. Across conversation, embodied planning, medical and legal/finance reasoning, the evolved memory programs are structurally different, and a program moved from one task to another is worse than the generic seed in 5 of 6 cases.

There is no universal memory system

An agent’s memory system answers three questions: what to store, how to organize it, and when to retrieve it. Each existing design comes with default answers. Vector search stores raw text and retrieves by semantic similarity. Mem0 extracts facts about people. ReasoningBank and Dynamic Cheatsheet store lessons from past experience.

Different tasks need very different answers to these questions.

• An agent that chats with users needs people, events, dates and preferences: who said what on which day, who likes what.
• An agent doing household tasks in ALFWorld needs reusable action patterns: to cool an apple, first pick up the apple, then find the fridge.
• An agent answering legal questions needs cases broken into issues, governing rules, factual predicates and counterarguments.

A memory designed for chat will likely not work well for household tasks, and the other way around. So the question we asked is: can each task find the memory system that fits it, without us fixing any design in advance?

Writing the memory system as Python

M* defines a memory system as an executable Python program with three parts:

1. Schema: dataclasses that define what a memory item stores (KnowledgeItem) and what a query looks like (Query).
2. Logic: the write() and read() methods, which can call a toolkit with SQLite, ChromaDB and a budget-limited LLM.
3. Instruction: prompt constants that control how the agent summarizes, writes queries and answers.

The task agent that uses the memory is held fixed, so any change in score comes from the memory program alone.

The M* evolution loop

The search is a reflective code-evolution loop. Each iteration has four steps:

1. Sample a parent: pick an existing program by softmax over scores. Better programs are picked more often, but weaker ones still get a chance.
2. Evaluate: run the task with this program (write, read, score) and record the failures.
3. Reflect and mutate: a reflector model reads the failures, diagnoses them and writes a code patch.
4. Quality gates: the new program must compile, pass a smoke test and stay within resource limits before it joins the population.

Each task runs 20 iterations. In the ablation on LoCoMo, the full system reaches F1 0.459. Evolving only prompts without changing code drops it to 0.256, removing population diversity drops it to 0.318, and always picking the best parent instead of softmax sampling drops it to 0.381. Prompt-only evolution is the worst, which indicates that the structure of the memory itself is the part that needs to change.

Four tasks, four memory structures

All four tasks start from the same three seed programs, and end up very different.

ALFWorld (embodied planning): a pure SQLite action cache with no vector retrieval at all. Each memory holds fixed fields extracted from expert demonstrations: target object, destination, required state change (clean, cool, heat, examine, toggle), action hints and failure modes. Retrieval adds exact-match bonuses for object, location and state.

LoCoMo (long conversation QA): a hybrid of SQLite and ChromaDB. Each observation is parsed into 7 structured fields (participants, organizations, activities, key facts, relation facts, named entities and more) and stored in both a relational table and a vector collection. Retrieval fuses semantic and lexical scores, boosts the person asked about, and takes at most 2 chunks from any single dialogue.

HealthBench (medical reasoning): SQLite tables keyed by clinical topic, urgency signals, patient attributes and safety notes. Reads prioritize memories that match the clinical intent, then use one LLM call to synthesize short guidance.

PRBench (legal and finance reasoning): the legal program stores issues, governing rules, factual predicates, counterarguments and citation-like anchors. The finance program stores assumptions, valuation drivers, comparable-company cues, risk factors and calculation notes.

It is hard to imagine anyone designing these four structures in advance as one general framework. We also ran a transfer test: moving a program evolved on one task to another task is worse than the generic seed in 5 of 6 cases.

Against baselines including Mem0, G-Memory, ReasoningBank, Dynamic Cheatsheet, GEPA and MemEvolve, the main numbers are:

M* does not win everywhere: on ALFWorld seen environments, GEPA + vector search reaches 0.820 and M* reaches 0.700.

Code evolution also learned to cheat

When the search space is code, holes in the metric become code too.

On LoCoMo, the generation-18 program contains this rule: if the question mentions “feel” and “after”, and the retrieved text contains “grateful”, “thankful” and “family”, return the literal string “Grateful and thankful for her family”. This is an answer copied from the dataset, not a better memory system. Similar hard-coded strings appear on every benchmark, most of all on PRBench Legal.

Another effect: rubric benchmarks are scored by an LLM judge that prefers long answers, so evolution pushes the instructions toward longer and more detailed answers. The score goes up, but the memory does not necessarily get better.

We list these in the paper’s limitations. My takeaway is that when LLMs optimize any part of an agent, the evaluation signal itself has to be reviewed rather than trusted by default. Readable code helps here: you can open the program and see the exact line that cheats.

Other limitations: all experiments use one model pair (GPT-5.4 Mini as task agent, GPT-5.3-Codex as reflector). The search is not cheap, from $12 to $90 per run, and ALFWorld took up to 101 hours because environment interaction is slow. On LoCoMo, some memory systems built specifically for conversation (such as LightMem and SimpleMem) report higher scores than ours.

One suggestion for building agent memory

If you are building memory for your own agent, the practical suggestion I take from this work is: before picking a framework, write down what this task actually needs to remember and how it will be queried. M* automates that step and checks it with a score.

Memory is only one part of an agent harness. Our other work, RHO, studies how an agent can improve its whole harness from its own past trajectories without labels; see this post.

M* is accepted at NeurIPS 2026 (7,900 of 30,709 main-track submissions accepted, 25.7%). The project page lets you open every evolved program and its diff against the parent, including the cheating rule above.

• Paper page: wenbo.io/publications/mstar
• arXiv: arxiv.org/abs/2604.11811
• Code: github.com/wbopan/mstar
• Project page with the evolution runs: mstar.wenbo.io