AI Agents

ORCA-bench puts coding agents in a production-fidelity oncall setting: a live OpenTelemetry microservice system exposing six days of metrics, logs, and traces through Prometheus, Jaeger, and OpenSearch, plus full source access, with 1,079 RCA tasks that vary report specificity, detection lag, and co-occurring faults. Ground truth is SRE-signed and the LLM judge re-scored by humans (κ_w=0.90). Best RCA accuracy is 25.3% on Medium and 10.0% on Hard; the weakest model hallucinates a root cause in 40% of reports. Removing source access degrades every metric.

SIGIL attacks a real problem with prose skills: because the procedure lives in context and never in the runtime, the model re-derives control flow every run and skips mandated steps — across 30 skills and two model generations, a prose agent performs only 56% of the steps its own skill requires while still passing output checks. SIGIL compiles the prose into an executable harness via AG-IR, a typed IR separating model cognition from code mechanism. Compiled harnesses hit 86% step compliance, complete full procedures 2.3x as often, at 0.58x the tokens.

Echoverse tackles the bottleneck in synthetic training environments for computer-use agents: not how many exist, but what's inside each. It compiles specs into stateful apps graded against their own databases, with a co-evolution loop that reads each rollout twice — as repairs to the environment and verifier, and as training signal. A 9B model trained on twelve environments jumps from 36.5% to 67.1%, within 14 points of the frontier teacher. Ablations are the strength: shallow environments push accuracy below baseline, deep ones raise it, and repairing one environment lifts its model from 16.2% to 38.5%.

This paper separates tool ranking from tool acquisition — a ranking tells you the order, not how many to take. CAM-DF frames stopping over ranked tool prefixes as a cost-aware decision, training on the offline gap between stopping now and the best continuation, with a proof that score-only thresholds are suboptimal under heterogeneous costs. Across 1,343 tasks in five domains it beats predict-then-threshold baselines, and in live execution exposes the agent to 37% fewer tools while holding task success. A clean, deployable answer to a genuine production tradeoff.

A systematic study of inference-time scaling for local computer-use agents, evaluating Qwen3-VL-8B/30B, UI-TARS-1.5-7B, and OpenCUA-7B on OSWorld across contextual, temporal, structural, and parallel scaling. The finding worth internalizing: more compute mostly changes failure modes rather than fixing them. Contextual scaling stabilizes trajectories but saturates and shifts failures toward premature false successes; temporal scaling cuts stalls but extends erroneous trajectories instead of correcting them; structural decomposition adds planning overhead. Parallel scaling helps but at heavy cost. A useful corrective to assuming test-time compute is a free reliability knob for constrained models.

A deprecation postmortem: the team built an LLM router, then killed it. The value is the negative result — a piece of infrastructure everyone is currently building, retired with the tradeoff reasoning that justified it. HN traction is decent (121 points, 74 comments) but the provided text is just the link, so the specific failure modes and cost math live behind the post itself. Worth reading if you're mid-build on your own routing layer and want a counterweight to the default assumption that you need one.

qm is a multiplayer agent harness aimed at real work, and it landed hard on HN (581 points, 119 comments). The premise — multiple agents (and presumably humans) coordinating in a shared harness rather than isolated single-agent loops — is squarely on-beat for anyone wiring up team-scale agent workflows. The provided data is thin: no abstract, just the repo link and traction numbers, so the coordination model, isolation guarantees, and failure handling aren't verifiable from here. The signal is community interest; the depth check requires the repo.

A SWE-agent benchmark spanning 13 models and 4 agents across Go, Java, Python, Rust, and TypeScript. The multi-language breadth is the point — most code-agent evals overfit to Python, and cross-language results are where the interesting variance usually hides. Traction is modest (48 points) and the excerpt gives no methodology detail, so task construction, contamination controls, and grading rigor can't be judged from the provided text. If you build code agents and ship in more than one language, the per-language breakdown is the reason to click through.

Simon Willison and Jesse Vincent's Prime Radiant lab shipped smevals, a CLI for running small eval suites across model and harness configurations. The design separates concerns cleanly: an eval is a directory of YAML files, runs are decoupled from grading, and a localhost server lets you explore results. The intended workflow is agent-native — point your coding agent at uvx smevals docs, have it build the suite, then run against multiple models in one command. Lightweight and concrete rather than another sprawling framework.

Willison walks through the 2026-07-28 MCP spec — stateless MCP, the biggest change since launch — with before/after HTTP examples and honest tradeoffs. His framing is useful: MCP got eclipsed in 2025 by Skills once people realized an agent with a terminal and curl could do most of it. The renewed case is safety — giving an agent an open shell with internet access is risky, and constrained MCP tools are a tighter contract. He built mcp-explorer and datasette-mcp against the new spec.