AI Agents

This paper argues that standard semantic search for agent memory is a security vulnerability. Retrieving semantically similar but contextually inappropriate memories can cause data leaks or tool-use errors. The authors demonstrate these flaws and propose MemGate, a lightweight (9M parameter) plugin that sits between the vector store and the LLM. It acts as a query-conditioned neural gate, learning to block or allow retrieved memories based on task appropriateness, not just similarity, without requiring LLM modification.

The paper introduces Retrospective Harness Optimization (RHO), a self-supervised method for improving an agent's skills and workflows using only its past execution history. RHO identifies challenging past tasks, re-runs them with variations, and then uses self-validation and pairwise self-preference to select harness improvements. Critically, it requires no external labeled data. On SWE-Bench Pro, a single round of RHO improved the pass rate from 59% to 78%, demonstrating a concrete path to automated, continuous agent improvement in production.

The LatentSkill framework converts textual agent skills from in-context prompts into modular LoRA adapters using a hypernetwork. This moves skill knowledge from the context window into the model's weights, drastically reducing per-step token overhead. Experiments show this not only cuts costs but also improves performance: ALFWorld success rose by up to 21.4 points with 64.1% fewer prefill tokens. The resulting skill adapters can be composed and scaled, offering a more efficient and less exposed substrate for building agent capabilities.

AdaMEM is a framework for test-time adaptation that challenges the static, one-shot retrieval common in memory-augmented agents. It uses a hybrid architecture: a long-term memory of past raw trajectories and a dynamic, short-term strategy memory generated on-the-fly to guide each step. This allows the agent to continuously adapt as a task unfolds, without online parameter updates. The method yields significant gains on ALFWorld (+13%) and WebShop (+11%) over static memory baselines by enabling more flexible, continuous reasoning.

This work challenges the rigid "retrieve-then-reason" memory pipeline. The proposed MRAgent framework instead treats memory as something to be actively reconstructed. It uses a Cue-Tag-Content graph structure and integrates LLM reasoning directly into memory access. The agent can iteratively explore and prune paths through the memory graph based on intermediate findings, making retrieval dynamic and context-aware. This active reconstruction method showed a 23% improvement on the LoCoMo benchmark over strong baselines, with lower token costs.

lowfat is an open-source CLI filter designed to slash token costs for agents using shell commands. It acts as a middleware layer, intercepting verbose output from tools like kubectl or docker and stripping it to the essentials before it reaches the LLM. The author shares real-world benchmarks from two months of use, showing token savings over 90% for common commands. The tool is a single binary with a plugin system for creating custom filters, directly addressing a major operational cost.

A detailed statistical analysis investigates whether the adoption of AI coding assistants correlated with an increase in bugs in the rsync project. The analysis examines commit data, bug reports, and code complexity metrics over time to untangle correlation from causation. This piece serves as a rare, data-driven case study on the real-world safety and reliability impact of using LLMs in critical open-source software, providing a methodological template for evaluating this new development paradigm at a project level.

A developer shares a specific workflow for integrating an AI agent into a test-driven development (TDD) cycle. The process constrains the agent by having it first generate a failing test, then the minimal code to make it pass, and finally refactoring. This structured interaction uses the TDD process itself as a scaffold, breaking down a large code generation task into a series of smaller, verifiable steps. It's a practical pattern for guiding an agent towards a more reliable output.

An observation on developer psychology: programmers who neglect writing documentation for their human colleagues are often motivated to write detailed documentation for an AI. The likely reason is the immediate, tangible reward—the AI uses that documentation to provide better answers and assistance. This suggests the feedback loop for documenting for an AI is much tighter and more gratifying than documenting for a hypothetical future developer, changing the incentive structure around a notoriously difficult task in software engineering.

Simon Willison details his latest approach to sandboxing agent code execution, combining MicroPython and a WASM runtime. The post explains how this stack meets his long-sought requirements for a sandbox: fast startup, low resource usage, and strong security guarantees. He has released the implementation as micropython-wasm and is using it in his datasette-agent project. It's a practical guide to solving a core problem for any agent that needs to run untrusted code from users or other models.

OpenAI has rolled out "Lockdown Mode" for ChatGPT, a feature designed to mitigate data exfiltration from prompt injection attacks. As Simon Willison analyzes, this feature directly targets one leg of the "Lethal Trifecta" of agent vulnerabilities by blocking or restricting outbound network requests. This makes it much harder for an attacker to steal sensitive data, even if they successfully inject a malicious prompt into a GPT that has access to private information. It's a critical defense-in-depth control.

This post is a practitioner's critique of common failure modes in reinforcement learning environments used for agent training. The author argues that subtle bugs and design flaws in the evaluation "harness" often degrade model performance rather than improve it. Drawing on years of analyzing agent trajectories, the piece details common anti-patterns in reward functions and state transitions, providing concrete advice on how to build more robust and realistic environments. A necessary read for anyone building custom agent evaluations.

This is the alpha release announcement for micropython-wasm 0.1a2, the sandboxing tool from Simon Willison's companion blog post. The main feature in this update is the addition of a command-line interface. This allows developers to easily experiment with the MicroPython-in-WASM sandbox directly from their terminal, providing a quick way to test its capabilities and limitations without first integrating it into a larger application. The full technical details are in the main blog post.