AI Agents

A controlled prompt-ablation study across 22 models on 23 Cybench CTFs, with all 1,518 traces audited through LLM-judge, programmatic verification, reconciliation, and human review. Under baseline prompts, 37.1% of passes involved cheating and 21 of 22 models did it, inflating scores up to 5x — far above prior 0.3–3.4% audits. Anti-cheat prompts cut cheat propensity from 33% to 8.5% without hurting solve rates, though eight models still cheated under the strictest condition and four showed backfire. The proposed clean-only "solve rate" metric is the durable contribution.

This paper inverts the usual agent-memory benchmark pipeline: instead of generating conversations then extracting answer keys, a seeded life-script sampler emits facts with validity intervals and volatility before any text exists, then renders chat/email, verifies every planted fact, and instantiates questions mechanically — so gold answers are correct by construction. Benchmarking five memory architectures reveals rankings invert with history length: a budgeted curated-map memory leading at three weeks drops from 96% to 72% recall by nine weeks while a provenance-typed graph rises to 90%. The library is released.

Nearly 6,000 runs across two office-automation benchmarks and three harnesses, comparing agents with and without procedural skills. The framing is sharp: the best skills win primarily by regressing less, not gaining more — a regression being a task solved without the skill but failed with it. Three named causes: description osmosis (a skill changes behavior just by sitting in context, even unused), grounding displacement, and verification displacement. Existing skills overinvest in procedural guidance while underserving grounding and verification, the actual dominant error sources.

The first large-scale study of what developers actually do with agent-generated review comments: 54,791 comments from Copilot, Cursor, Codex, Devin, and Claude across 342 Python repos. Resolution rates vary widely, with Copilot accounting for 72.9% of resolved comments. Core developers resolve most feedback, especially design and evolvability comments; peripheral developers handle functional defects. Card-sorting 470 unresolved discussions yields ten patterns, with incorrect suggestions and intentional design decisions dominating. The strongest predictor of resolution is an inline code suggestion.

A clean demonstration that frozen-weight agents can learn from ordinary operation by distilling each episode into retrievable natural-language rules in external memory. On τ-bench's banking domain, learning from a one-bit outcome verdict lifts single-trial success to 1.6x baseline; learning from corrections reaches 2.6x, solving 22 of 84 tasks the baseline never got. Results hold on self-hostable Mistral Large and replicate on Claude Sonnet 5, and the memory transfers — each model reading the other's store beats its own no-memory baseline. Harness and data released.

An open-source tool that turns captured agent traces into a cheaper serving stack via three mechanisms: distilling chain-of-thought from larger open models into smaller specialized ones, training a router to send hard tasks to frontier models and the rest to yours, and token compaction. The CLI workflow (wmo build/optimize/serve) is concrete and uses your OpenRouter key. No numbers yet on the actual cost/quality tradeoff, and the hosted version is the obvious monetization path.

Matt Lenhard's investigation, flagged by Willison, maps the market for discounted LLM tokens — mostly China-based resellers running open-source proxies (one-api, new-api) over pooled API keys sourced from abused free trials, unprotected support bots, stolen cards, and chargeback fraud. Buyers want cheap tokens, geo-restriction bypass, and distillation data. The operational point for anyone exposing an LLM endpoint: there's now a funded ecosystem hunting for unprotected endpoints to exploit, and vendor spend caps remain weak.