AI Agents

MENTOR tackles compliance for tool-using agents governed by natural-language procedural manuals (travel booking, procurement). Rather than just blocking bad tool calls at runtime — which leaves the agent stuck and tends toward over-conservatism — it autoformalizes the manual into a history-dependent runtime shield and uses a game-based technique to stay robust under uncertainty while also guiding the agent toward compliant progress. The headline result: it lifts shielded lightweight open models above proprietary baselines on compliance.

This paper questions the foundation of agent debugging: the human- or judge-labeled 'first mistake' step. It makes the distinction executable by restarting failed trajectories from reconstructed states and estimating each step's prefix value — probability of success from there. On SWE-bench Verified with frozen open-weight agents, judges agreed on the exact step in only 21/57 cases, and for 66% of labels success probability didn't drop at the labeled step. Restarting at the measured point raised success 0.44→0.57; restarting at judge labels didn't help.

VulnBump evaluates agents on a task that matters in practice: upgrading vulnerable dependencies while keeping builds and tests green. 307 tasks, 2,066 fixable vulnerabilities, 43 repos across six package managers, verified in fresh containers independent of the agent. Results split by failure mode: Snyk's purpose-built agent passes 90.2% but mostly fails by breaking builds; unaided Claude Code hits 59.6% and mostly leaves vulnerabilities in place. A scan tool raises Claude Code's pass rate 17.9 points — but knowing the target version isn't enough.

A behavioral diagnosis of 'System One' models that pick an agent's next action in a single scoring pass. By pairing each next-action question with independent factual probes about the same state across 17 fast systems plus a reasoning model, the authors show most wrong actions follow correct reading — and the more accurate the system, the larger that share. Even when probed facts provably determine the action, models read every fact but fail to combine them; errors cluster on the same states.

A new evaluation dimension — epistemic humility — for how agents handle knowledge conflict when retrieved evidence contradicts their priors or sources disagree. The authors operationalize it as Identify/Solve/Escalate across controlled and naturally occurring conflicts with matched controls. Across four agents, higher task accuracy didn't track greater humility: some high-accuracy configs detected conflicts mid-execution but never communicated unresolved uncertainty in wrong final answers. Agents often spot conflicts early then lose them downstream, and interventions that improve humility tend to cost accuracy.

An open-source benchmark that measures something existing coding-agent evals ignore: how much net code a model adds to solve a task. It reuses published patches from DeepSWE, SWE-bench Verified, and SWE-bench Live, counting syntax 'coding units' — statements, calls, names, literals — rather than lines, so cramming isn't rewarded, and subtracting deletions. The method is clean, but newer models were self-run on 3–10 tasks on personal subscriptions, so those numbers are directional at best.

A YC launch pitching an LLM gateway for ITAR-restricted and CUI workloads, routing requests to FedRAMP High–equivalent infrastructure — the founder cites a FedRAMP equivalency audit against 410 controls. For teams in aerospace, defense, or government supply chains, the compliance posture is the whole product; most such orgs ban AI tools outright today. It's a vendor pitch, so verify the audit scope and routing architecture yourself, but the problem it addresses is real and underserved.

Anthropic replaced Git with a custom VCS after agent-driven commits broke their monorepo's scaling assumptions: 2,500 merges per day in May became over 30,000 by September, with a new 'change request' primitive overtaking pull requests between June and July. Simon Willison annotated their chart with model release dates to show the correlation. The honest number here is the point — this is what code velocity looks like when agents, not humans, author most changes.

A sharp little pattern from Geoffrey Huntley: instead of relying on AGENTS.md to keep a coding agent using your diagnostics library — which drifts — enforce it with a pre-commit hook that scans the diff for direct observability calls, fails the commit, and injects a prompt telling the agent to fix it. The insight is treating deterministic checks as the steering mechanism and the prompt as the feedback channel, rather than hoping documentation holds.