AI Agents

This varies reviewer capability in execute-review-revise pipelines while holding the problem set fixed at 100 olympiad math problems. A mid-tier cross-family reviewer lifts accuracy from 52% to 64% with zero damaged answers. Same-model self-review has the highest error-detection recall (0.85) yet no net gain: it rejects 2.1x as often for a third the repair rate and falsely rejects 35% of its own correct answers. Its apparent safety is revision inertia — ignored rejections survive, complied-with ones turn wrong.

The claim is that agents plan blind to the substrate they run on — memory, wall-clock, compute limits — and that handing them the constraints as an explicit contract changes what they generate. Across Claude Opus 5, GPT-5.6-Sol, and Gemini 3.7 Flash on a Euclidean-distance task, disclosing a 128MB/10s contract cut peak memory in 13 of 14 comparisons and ran up to 3.1x faster. At a tighter 96MB budget, correct-and-within-budget rates jumped from 0/5 to 4/5 for Opus. Narrow task, but clean.

For monitoring web agents when you can't see token logits — the black-box production case — this predicts failure from observable trajectory signals alone. Macro features summarize agent-environment behavior; micro features probe consistency of intention, action, and expected state via repeated queries. The labeling trick matters: they mark the first uncorrected critical error as a key-step boundary instead of inheriting the final result label, so early prefixes of doomed runs stay on-track. On WebArena-Lite and Online Mind2Web across five backbones, it matches internal-signal baselines and supports early intervention.

Prospective memory — firing a deferred intention at the right future cue — is reframed here as schema-constrained state tracking rather than open-ended reasoning, so it can be pushed into code. The Prospective Intention Store keeps lifecycle logic in code and leaves scoped language work to the model, training-free with no fine-tuning. DeepSeek-Chat with PIS hits 82.9% Set-F1 on PM-Bench versus the prior best scaffold's 65.1%; Gemma-E2B goes from 4.2% to 66.2%. A small model beating a large-model scaffold by moving the loop into deterministic code.

AGENTSCOPE diagnoses agent failures by abstracting long trajectories into structured representations and checking them against neural invariants — behavior properties specified over the abstraction. It's a neuro-symbolic middle path: pure LLM-as-judge is unreliable, traditional software debugging doesn't fit, so they combine LLM-guided reasoning over a symbolic structure to pinpoint both failure step and type. Evaluated on the Who&When dataset and a new AgentErrata set, it beats prior state of the art on fault localization and attribution. The invariant-specification approach is the interesting part for anyone building their own failure taxonomy.

Dan Luu examines how coding agents actually use tests and verification rather than whether they can pass a benchmark. It's the practitioner-observation genre, not a controlled study, but Luu's track record on measuring what tools really do makes it worth reading for anyone whose agents write and run tests. The HN discussion (92 points, 26 comments) is where much of the nuance lives; treat the piece as calibration for your own eval harness, not a citable result.

Trail of Bits shipped a tool for running Claude Code and Codex inside isolated VM environments rather than on your host with shell and commit access. It's a concrete answer to the obvious question — what stops a coding agent from touching things it shouldn't — from a security firm rather than a vendor. Early days (61 points, 16 comments) and the README carries the details, but VM-level isolation is the right default for agents that execute arbitrary code.