MemRiskBench: Trace-Aware Risk-Preserving Evaluation for Long-Horizon LLM Agents
J. Jiang, D. Yuan, W. Li · 2026
Ce que cela signifie pour vous
Un assistant qui se souvient de vous doit aussi savoir quand une information est périmée, et l'oublier quand vous le demandez, ce qu'une bonne note moyenne ne garantit pas.
Résumé
Argues that average scores hide the rare but serious failures of agents with long-term memory. Defines five risk types (stale facts after an update, unresolved conflicting facts, leakage across users, reuse of revoked memories, and gradual decay of standing constraints) and checks each one deterministically against the agent's execution trace instead of relying on an LLM judge. Across 120 scripted episodes and five small open models, overall pass rates conceal large per-risk gaps, with cross-user leakage the weakest category for every model tested. A subset that keeps only the episodes where models disagree reproduces the full ranking at a fifth of the cost.