AI Agents

A systematic look at whether SWE-bench-like benchmarks actually pair PRs with the issues they resolve. Studying SWE-bench Verified, the authors find 13.6% of instances are misaligned across five patterns and eleven scenarios — meaning the problem statement and the test oracle don't match, quietly corrupting scores. Their detection system, a three-phase multi-agent pipeline combining pattern identification, cross-agent label synthesis, and code-level validation, reaches up to 92% binary accuracy on SWE-Gym and SWE-bench Multilingual. A direct hit on the reliability of the field's default coding benchmark.

A multi-agent coding benchmark built on ten real full-stack projects, testing ten collaboration topologies with agents syncing periodically and deploying through real CI/CD. The headline result: organizational topology rivals model capability. Holding task and model fixed, varying the topology shifts scores by over 30 points and doubles wall-clock time — structured pipelines converge fastest and cleanest, while heavy managerial oversight actively degrades output. The grading probes implementations dynamically and measures functional success, latency, and prefix-cached token cost jointly. Benchmark released.

A clever evaluation method for the "can agents do AI research" question: give a frontier agent the central open research question from a high-quality unpublished paper, then have the paper's own authors grade the output. On two NeurIPS 2026 submissions, with six days and thousands of dollars of compute each, agents completed all the engineering unaided but made no substantial progress on the research questions — both were unambiguously rejected by the authors. Five recurring failure modes named, reproduced with a second model and scaffold. Reviews, repos, and logs released.

A benchmark that asks whether VLM judges of computer-use agent trajectories are actually trustworthy — a question that matters because these judges increasingly stand in for humans in CUA evaluation, data curation, and RL reward. Built from diverse agent backbones on human-verified instructions with multi-stage ground-truth labeling, it finds even SOTA models share a systematic leniency bias, mislabeling failed runs as successes. The reliable judges are too expensive to run at scale; affordable open ones trail badly. They release a 100K-trajectory reasoning-annotated corpus to close the gap.

An RTL-optimization agent with two findings worth separating. First, a methodology fix: report normalized dollar cost per LLM call alongside quality, enabling fair comparison across reasoning-effort levels. Using that accounting, the negative result — an engineered cross-design long-term memory brings no dependable gain over plain concatenation of the same experience. The positive contribution is adaptive effort: escalate to deeper reasoning only when progress stalls, via a patience counter. It lowers figure-of-merit by 23-27% at equal cost and reaches 25% deeper results than prior work at 12% of the tokens.

A distillation study with a clean question: does the political censorship baked into a Chinese teacher model transfer when you distill it into an American base? Using DeepSeek V4 Flash as teacher for GPT-OSS-120B, the answer was no — the distilled model kept its base's behavior while the teacher answered sensitive prompts seven standard deviations differently. The methodology is the strength: 152 matched pairs (Great Leap Forward vs. Holodomor) scored by four LLM judges, validated against human scores at r=0.948. Open weights released.

A local merge queue for people running 4-5 parallel Claude Code agents pushing up to 90 commits a day. The problem is concrete: parallel agents all building, testing, and running dev servers on an 8GB MacBook Air is a fast path to a force-quit, and 90 pushes a day burns CI minutes. The fix serializes commits so they land one at a time, fully tested, locally. Modest in scope but it names a real resource-contention failure mode of high-throughput agent workflows.

Claude had an all-models elevated-error incident on July 29, now resolved; the HN thread drew 265 points and 246 comments. The status page is the only detail here, so there's no root-cause narrative to summarize. The signal for operators is the blast radius: an outage spanning all models is a reminder that a single-provider dependency in your agent stack is a single point of failure, and the comment volume suggests plenty of production workloads felt it.

A writeup on building a decompiler in the age of coding agents. No excerpt beyond the title, so specifics are behind the link, but the domain itself is worth flagging: decompilation is a demanding, correctness-sensitive area where agent-generated code has to survive against ground-truth binaries, which makes it a sharper test of coding-agent utility than typical greenfield app work. If it delivers on engineering substance, it's the kind of concrete tooling report that generalizes.

A tmux-based TUI for orchestrating multiple coding agents — Claude Code, Codex, and OpenCode — from one terminal. The excerpt is just the title, so treat feature claims as unverified, but the shape addresses a real operational gap: once you're running several agents in parallel, watching and steering them across panes beats juggling separate windows. A practical multi-agent ops tool rather than a framework; the 95 points and 76 comments suggest the pain point resonates.

An API gateway that routes agent traffic turn-by-turn between models — sending harder turns to frontier models and easier ones to cheap open weights — claiming to match Claude Fable 5 at half the cost. The framing is a vendor launch, so the headline number wants independent verification, and the routing technique is described in a blog post rather than a paper. Still, turn-level (not request-level) routing is the right granularity for agent loops, where difficulty varies sharply within a single task.

Prompted by last week's OpenAI/Hugging Face episode, Anthropic audited 141,006 evaluation runs and found three incidents (six runs total, four hitting one organization) where Claude reached real systems on the open internet during cybersecurity evals. The root cause is mundane and instructive: the prompt told Claude its environment was a sandbox with no internet access, but a misunderstanding with the eval partner left real egress open. The model took the prompt at its word and treated live targets as in-scope.

A short TIL walking through the steps to connect a custom MCP server to the standard Claude and ChatGPT chat interfaces — not the API, the consumer chat products. The excerpt is light, but the useful signal is that it's "possible but takes quite a few steps," which matches the current state of MCP in first-party UIs: the plumbing exists but isn't frictionless. Worth a read if you want your server reachable from the chat surfaces rather than only from agent frameworks.

OpenAI cut GPT-5.6 prices sharply: 20% off Terra, 80% off Luna. The mechanism is the interesting part. They credit GPT-5.6 Sol with optimizing inference itself — using Codex, Sol autonomously rewrote production GPU kernels in Triton and Gluon, finding work to precompute, avoid, or parallelize to keep GPUs from idling on memory movement and synchronization. It's a concrete instance of a model improving the infrastructure that serves it, and the savings land directly in agent per-turn economics.