AI Agents

SWE-NFI benchmarks something correctness suites ignore: behavior-preserving non-functional improvements — refactors, structural cleanups, quality changes that don't alter observable behavior. It's 188 tasks from real merged Python PRs, with 92 executable rules combining functional-correctness tests and rule-based NFI scoring. The best agent hits 70% functional correctness but every agent trails human developers on NFI overall, and the gap is stark on structural improvements (agent scores 0.0–1.3 versus 1.5 human). A reproducible answer to why agent PRs feel technically-passing-but-off.

This audits how computer-use agent benchmarks actually score. Reviewing 150 public failure-scored trajectories across five web, enterprise, and desktop benchmarks, the authors find 15.3% of FAIL verdicts are wrong — 10.7% evaluator false negatives, 4.7% broken tasks. For genuine failures, a three-tier taxonomy shows verification/feedback and planning errors dominate over execution/grounding, which a single scalar success rate can't surface. They organize the failure modes into a reliability framework and derive stage-specific design rules for task construction, trajectory capture, scoring, and reporting.

The first systematic study of the now-default agent memory pattern: a directory tree of markdown files the agent reads, writes, and reorganizes with generic file tools. The authors formalize three roles — management, search, and execution agents — around one memory filesystem, then vary memory shape (organized hierarchy, verbatim dump, chunk retrieval), scale, tool harness, and agent strength while tracking answer quality, cost, and store health. The reliable payoff is search economy: organized stores roughly halve retrieval cost when material is large. But under sustained growth, organization erodes — today's agents fall short of the pattern's promise.

RoboBRIDGE is an orchestration layer that turns a pretrained Vision-Language-Action model into a deployable robotic agent without retraining the backbone. Five modules — Monitor, Perceptor, Planner, Controller, Robot Interface — address the gaps VLAs expose in deployment: no failure recovery, drift over long horizons, brittleness to observation and embodiment shifts. The Monitor does rapid detection plus hierarchical recovery before errors cascade; the Planner replans while the Perceptor updates scene understanding asynchronously to avoid stalls; the Controller factors skills into LoRA-adapted primitives. Evaluated on LIBERO, RoboCasa, and real hardware across multiple platforms.

TAPO addresses a waste in agent RL: methods lean on sparse task rewards while ignoring the dense signal already flowing back as environmental feedback after each action. It alternates policy optimization with transition supervision, repurposing existing rollout data to train action-conditioned next-observation prediction on a shared backbone — no extra expert data, sampling, or inference overhead. Tested on WebShop and ALFWorld across model scales and multiple policy-optimization algorithms, it consistently beats pure policy optimization as a plug-in module. A cheap, well-motivated add-on rather than a new training regime.

A technique for bursty LLM inference: predictively replicate KV cache to absorb load spikes, with a GitHub repo (bite-the-bullet) attached. The framing targets a real operational pain — bursty request patterns where cold KV state stalls throughput. The item itself is thin (41 points, 4 comments, mostly a link), so the actual replication strategy, eviction policy, and any latency numbers live in the repo and post rather than here. Worth a look if you're running agents with spiky, correlated traffic.

datasette-apps 0.2a0 adds two agent tools worth noting. app_debug() renders an app in a hidden iframe (opacity: 0, pointer-events: none) and runs agent-supplied JavaScript inside that sandbox, so the agent can smoke-test that its own generated app works and even measure element dimensions. app_list() lets the agent enumerate apps the user can edit. Both ride the new context.browser_task() mechanism from datasette-agent 0.4a0. A clean, concrete pattern for giving an agent closed-loop verification of UI it just built.