🧠 MemHarness: Memory Is Reconstructed, Not Replayed

MemHarness is a framework that equips LLM agents to actively harness and reconstruct past experiences based on the present context — instead of replaying retrieved memories verbatim.

Most memory-augmented agents treat retrieved experiences as static records and inject them into the context regardless of whether they align with the agent's current situation. This "replay" paradigm ignores the gap between the abstract, general nature of stored experience and the concrete, ever-changing states at decision time, frequently causing negative transfer.

Inspired by the reconstructive nature of human memory, MemHarness decomposes memory-guided decision-making into five stages — observation → retrieval → critique → reconstruction → action. At each decision step, a unified policy model critiques the retrieved experience against its original context, reconstructs it into state-aligned guidance, and only then acts. This reconstructive ability requires no extra human annotation — it emerges naturally through end-to-end training with GRPO.

Figure 1: Memory utilization paradigms. Top: prior methods directly replay retrieved memories, risking state misalignment. Middle: human memory reconstructs past experience according to the current context. Bottom: MemHarness reconstructs retrieved memories into state-aligned guidance.

• 2026-08-07: 🤗 Cold-start SFT data released on Hugging Face Datasets.
• 2026-07-31: 🤗 Model weights (cold-start & GRPO-trained) are available on Hugging Face.
• 2026-07-30: 📄 Paper is publicly available on arXiv.
• 2026-07-30: 🎉 Codebase is publicly available.

• 🔄 Reconstruct, not replay — explicit critique and reconstruction are inserted between retrieval and action, turning memory from a static prompt fragment into context-sensitive guidance.
• 🚀 Substantial gains — 85.2% success rate on ALFWorld and 75.6% on WebShop, outperforming pure GRPO by +8.8% / +9.5%, and surpassing Gemini-2.5-Pro by +23.1% / +39.7% despite the 7B scale.
• 🛡️ OOD robustness — 85.9% average success rate on unseen ALFWorld layouts, vs. 76.3% for naive memory replay.
• 🧩 Latent guidance — the reconstruction objective improves the policy even when memory is disabled at test time (83.0% vs. 76.4% for pure GRPO on ALFWorld), fundamentally enhancing the agent's intrinsic reasoning capabilities.

Figure 2: Overview of MemHarness. The execution pipeline consists of three stages: (1) Memory Retrieval, where the policy generates a query to fetch relevant past experiences; (2) Contextual Memory Reconstruction, where the policy compares the memory's source state with the current state to reconstruct adapted guidance (or falls back to self-reasoning if the memory is deemed unhelpful); and (3) Action Generation guided by the reconstructed memory.

MemHarness maintains an explicit, inspectable memory bank while parameterizing state-conditioned reconstruction within a single policy:

1. Retrieve — the agent generates a query and pulls the top-k relevant experiences (each paired with its historical source observation) from a Milvus vector memory bank.
2. Critique & Reconstruct — conditioned on the current state, the policy compares the memory's source state with the present situation, then rewrites it into state-specific guidance — or rejects it and falls back to self-reasoning.
3. Act — the agent takes environment actions informed by the reconstructed guidance.
4. Write back & prune — after each episode, experiences are summarized from the trajectory, written back to the memory bank with semantic deduplication, and periodically pruned by empirical utility.

All of this is trained end-to-end with GRPO on top of the verl-agent infrastructure.

We recommend using Conda for environment management. The setup follows verl-agent, with an additional Milvus vector database for the memory bank.

# 1. Clone the repository
git clone https://github.com/KnowledgeXLab/MemHarness.git
cd MemHarness

# 2. Create and activate conda environment
conda create -n memharness python==3.12 -y
conda activate memharness

# 3. Install vLLM
pip3 install vllm==0.8.4

# 4. Install Flash Attention 2
pip3 install flash-attn --no-build-isolation --no-cache-dir

# 5. Install MemHarness (verl-based trainer + agent system)
#    This also installs pymilvus — Milvus Lite runs locally as the memory vector database.
pip install -e .

# 6. Experiment tracking
pip install wandb

Clone the memharness environment into a dedicated ALFWorld environment, then install ALFWorld on top:

conda create -n memharness-alfworld --clone memharness
conda activate memharness-alfworld

pip3 install gymnasium==0.29.1
pip3 install stable-baselines3==2.6.0
pip install alfworld

# Download PDDL & game files and the pre-trained MaskRCNN detector (stored in ~/.cache/alfworld/)
alfworld-download -f

Similarly, clone the memharness environment for WebShop:

conda create -n memharness-webshop --clone memharness
conda activate memharness-webshop

cd ./agent_system/environments/env_package/webshop/webshop
./setup.sh -d all

WebShop upstream recommends Python ≤ 3.10. If ./setup.sh fails in the cloned environment, create a fresh python==3.10 environment and reinstall MemHarness instead (see verl-agent).

If you encounter issues with gdown, visit https://drive.google.com/, get your Google Drive cookie, and paste it into .cache/gdown/cookies.txt, or download the files manually.

The memory bank embeds experiences with BGE-M3 through an OpenAI-compatible API. The simplest option is to serve it with vLLM:

vllm serve BAAI/bge-m3 --port 8001

Then point the training scripts to it via EMBEDDING_API_URL (default: http://localhost:8001/v1).

Following verl-agent, the training data for ALFWorld / WebShop only acts as a modality & size indicator — the actual agent input comes from the environment through env.step(). The training scripts run the preparation step automatically; you can also run it manually:

# ALFWorld
python3 -m examples.data_preprocess.prepare \
  --mode text --local_dir data/MemHarness/verl-agent/alfworld --infer_alfworld_sizes \
  --alfworld_eval_split eval_in_distribution

# WebShop
python3 -m examples.data_preprocess.prepare \
  --mode text --local_dir data/MemHarness/verl-agent/webshop --infer_webshop_sizes

By default the training scripts expect data under data/MemHarness/verl-agent/{alfworld,webshop}/text/ (train.parquet / test.parquet). You can adjust setup_verl_agent_text_data_paths in run_scripts/memory_eval_helpers.sh to point to your own location.

MemHarness training consists of two stages:

A brief cold-start stage aligns the base model with the MemHarness interaction format before GRPO training. For convenience, we also release cold-start model checkpoints on Hugging Face — you can skip this stage and use them directly as MODEL_PATH in Stage 2.

Cold-start data (200 train + 20 val samples per benchmark, plus GPT-5.1 teacher memory records) is available on KnowledgeXLab/MemHarness. Download and place under data/MemHarness/:

data/MemHarness/cold_start/
├── alfworld/
│   ├── train.parquet
│   ├── val.parquet
│   └── memory_records-gpt-5.1.jsonl
└── webshop/
    ├── train.parquet
    ├── val.parquet
    └── memory_records-gpt-5.1.jsonl

huggingface-cli download KnowledgeXLab/MemHarness --repo-type dataset --local-dir data/MemHarness

Then run SFT (set TASK=alfworld or TASK=webshop):

TASK=alfworld bash scripts/cold_start_sft.sh

# ALFWorld
bash run_scripts/train_alfworld.sh

# WebShop
bash run_scripts/train_webshop.sh

The scripts handle everything end-to-end: launching the memory vector database (locally, or on a Slurm cluster via MEMORY_REMOTE_SLURM=True), agentic memory retrieval, experience summarization / write-back / utility pruning, and GRPO training with format rewards.

Key configuration knobs in the training scripts (all overridable via environment variables):

Figure 3: Main results on ALFWorld and WebShop. MemHarness achieves the best performance (85.2% / 75.6% average success rate), outperforming pure RL and static memory-augmented baselines. Naively injecting raw memory on top of RL degrades performance, while state-conditioned reconstruction safely leverages experience.

Figure 4: Out-of-distribution generalization on ALFWorld with unseen layouts and object placements. Verbatim replay introduces state-mismatch noise, whereas MemHarness dynamically filters and rewrites mismatched guidance, achieving the highest average success rate (85.9%).

Please refer to our paper for ablations, mechanism analyses, and training dynamics.

MemHarness/
├── agent_system/           # Agent environments & memory system
│   ├── environments/       # ALFWorld, WebShop, Search, etc.
│   └── memory/             # Memory bank (Milvus), summarizer, retriever, utility pruning
├── verl/                   # RL training infrastructure (GRPO, based on verl & verl-agent)
├── run_scripts/            # GRPO training & evaluation scripts
├── scripts/                # Cold-start data building & SFT scripts
├── examples/               # Data preprocessing
└── tests/                  # Unit & e2e tests

All model weights are hosted on the Hugging Face Hub at KnowledgeXLab/MemHarness.

These checkpoints align the base model with the MemHarness interaction format before GRPO training. You can use them directly as MODEL_PATH in run_scripts/train_alfworld.sh / train_webshop.sh without running cold-start SFT.

Download the desired checkpoint from the Files tab and set MODEL_PATH in the training scripts accordingly.

• EvolveR — our prior work on self-evolving LLM agents through an experience-driven lifecycle (ICML 2026).

MemHarness is built upon verl-agent and verl, with vLLM for efficient rollout and model serving. The supported environments are adapted from ALFWorld and WebShop. The base trajectories (without memory) used to build our cold-start data are sourced from the open-source AgentGym project. We thank the authors and contributors of these projects for their valuable open-source work.

For any questions or feedback, please:

• Open an issue in this GitHub repository
• Reach out to us at wurong1159@zju.edu.cn

If you find our paper and code useful, please kindly cite us:

@misc{wu2026memharnessmemoryreconstructedreplayed,
      title={MemHarness: Memory Is Reconstructed, Not Replayed}, 
      author={Rong Wu and Daocheng Fu and Licheng Wen and Xuemeng Yang and Shu Zou and Jianbiao Mei and Yuxin Wang and Hairong Zhang and Yu Yang and Tao Hu and Cong Zhang and Botian Shi and Pinlong Cai},
      year={2026},
      eprint={2607.28272},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.28272}, 
}