AI Agents

A reliability audit of agent evaluation harnesses — the software that runs tasks and decides pass/fail. Using five state-integrity invariants and known-answer controls, it detects seven injected oracle defect classes with no false positives, then applies the method to a live benchmark and finds a reward function that mutates environment state before scoring. Replay-based correction across 4,320 episodes drops measured success from 94.3% to 65.5%. The honest limitation: the method couldn't validate a second benchmark because its API logs lacked reconstructable terminal state.

A clean negative result for the retrieve-and-compress orthodoxy in coding agents. At equal token budget on 300 SWE-rebench issues, a simple selector that picks whole files to show verbatim resolves 43.7% (GPT-5.6) and 46.7% (Claude Opus 5) — 6.3 points over the best compressor and up to 12.7 over the best retriever, beating even two methods trained for issue resolution. The mechanism is the point: retrievers miss files that must change, compressors keep them but alter the text, so patches fail to apply.

A sharp diagnosis of context compaction failure: because compaction prompts ask summarizers for decisions and next steps, each summary blends environment facts with the compressor's own reasoning, and the agent can't tell them apart later. In one widely used compressor, 83% of failed runs trace to an action executing an unsupported summary entry. The fix, Source-Judged Compaction, scores each entry for whether an environment output or prior summary supports it and drops the rest — training-free, one judge call per compaction, 2.1–8.9% reliability gains on AppWorld and OfficeBench with no extra agent tokens.

When agents are handed a full training stack they converge on SFT and spend most effort revising data, but a single agent only manages one crude lever — adjusting category proportions — before its context fills with execution output and gets discarded mid-search. AutoSFT is a multi-agent system that separates execution from analysis, gathers proposals from independent angles, and calls a synthesis subagent for gaps the data pool can't cover. Component ablations are individually attributable: the executor halves context loss, analysis subagents add +6.5 IFEval, synthesis adds +3.4.

An open recipe for the under-documented midtraining stage between pretraining and post-training, aimed at priming base models for agentic work. It trains at 256K context on hundreds of billions of tokens of natural software-engineering data — packed repos, PRs, commit histories, build logs — plus a small share of agentic traces. On Nemotron 3.5 Lightning, 500B midtraining tokens lift SWE-bench Verified 60.7→64.5 and Multilingual 55.8→62.8 after identical SFT; gains hold on the larger Super model and across harnesses. Notably, base-model proxy metrics didn't predict post-training results, forcing a short-SFT evaluation protocol.

Anthropic's own account of how the Claude Code team runs review internally, drawing a sizable HN thread (354 points, 332 comments). The core shift is from blocking PR review to continuous automated checking that handles routine issues before human eyes arrive. Treat it as a primary source on a working production workflow rather than a vendor abstract — the comments are worth skimming for where practitioners push back on the "review as spellcheck" framing.

A refreshed benchmark where LLMs play StarCraft: Brood War by writing code, now with flagship models from OpenAI, Anthropic, Google, and Mistral. The interesting part is the agentic framing — models express strategy as code rather than per-frame actions, which stresses planning and long-horizon reasoning under a real-time opponent. As an eval it's novel and watchable, though as always with game benchmarks, how much it transfers to the work you care about is the open question.

Claude was used to reconstruct the origins of the unexplained seed values in standard ECDSA curves, a long-standing cryptographic curiosity, generating a big HN response (883 points). It's a genuinely impressive AI-assisted research result and a good data point on where models help with hard investigative work. The direct relevance to building agents is thin, though — this is a model aiding a human researcher, not an autonomous agent system you'd deploy.

The Claude Code team reframes review away from "did a human read the code?" toward "what could go wrong, and which check catches it?" The practical consequence is timing: review runs continuously while code is written rather than as a gate at PR time, with Claude reviewing and fixing most changes before a human sees them. Human attention gets reserved for higher-level scrutiny. This is a concrete production pattern, not a thought experiment, worth reading alongside the full writeup.