2026-09-16

AI Agents

A measurement audit of the SWE-bench leaderboard finds the top entries statistically indistinguishable — the leading two both resolve 396 of 500, and paired tests separate none of the 29 adjacent top-thirty pairs. Elsewhere, a formal model asks how much of what multi-agent leaves discover reaches the root, plus a plan-injection attack that beats chain-of-thought monitors, a repository-scale vulnerability benchmark, and native skill routing read from a frozen model.

5 papers 1 news 0 blogs 36 appendix 501 considered

Papers

5

The Router Within: Eliciting Native Skill Routing from a Frozen LLM

arxiv:arxiv-agents — Ruishuo Chen, Xun Wang, Yu Chen, Zhuoran Li, Longbo Huang 8/10 tool-useresearch
The claim: a frozen agent LLM already encodes which skill to route to in its own forward passes, and two trained linear maps can read it out with no skill text in context. Gavel does a cheap glance across compact per-skill banks, then resumes shortlisted skills for a verdict, fused as a product of experts. On Qwen3-32B it beats progressive disclosure and retrieve-and-rerank pipelines that add 1.2B–16B external parameters, by up to 21.9 points when the skill need arises mid-rollout. Routing improves as the backbone does.

Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead

arxiv:arxiv-agents — Fengshuo Liu, Ying Liu, Ruize Sun, Lie Luo, Siyuan Guo 8/10 evalscode-agents
A careful audit of 254 SWE-bench submissions, done without rerunning models, argues the leaderboard cannot order its top entries. On Verified the leading two both resolve 396 of 500; the top ten share 285 successes and 51 failures, and exact paired McNemar tests separate none of the 29 adjacent top-thirty pairs at alpha=0.05. Scores swing up to 29.8 points on scaffold choice alone. The fix they propose: report comparison-set-specific resolution and model-scaffold provenance instead of treating small aggregate gaps as rankings.

Corrupt Plans, Clean Traces: Evading Chain-of-Thought Monitoring with Plan Injection

arxiv:arxiv-agents — Keertana Chidambaram, Andrew Ilyas, Vasilis Syrgkanis 8/10 safetyresearch
Chain-of-thought monitoring assumes the actor's reasoning trace surfaces unsafe intent for a monitor to catch. This shows you can plant harmful-but-benign-sounding reasoning in the actor's context, steering it to adversarial actions while the monitor stays clean — 25–33% evasion across benchmarks, scaling to DeepSeek-R1. Actors paraphrase injected plans as their own reasoning without attribution. The sharp finding: giving the monitor access to the injected plan drops detection by up to 50% on Bio-Math, and extra thinking budget gets spent rationalizing rather than flagging.

Decomposition Buys Integrity, Not Yield

arxiv:arxiv-agents — Rong He 8/10 multi-agentresearch
A formal model of decomposition asks how much of what leaf agents discover reaches the root. Model each agent as keeping any handed item with probability r(b); if r(b)=1/b every tree delivers exactly one finding, verified to 1e-15 on 20,000 random trees. Under r(b)=Cb^(-δ), a depth-k tree yields C^k·N^(1-δ): task size and architecture separate, architecture contributes only C≤1 per level, so flat is optimal for yield. On 600 production deep-research traces δ=0.34. Depth buys context integrity, not recall.

Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale

arxiv:arxiv-agents — Aman Priyanshu, Supriti Vijay, Kimia Majd, Xuhong He, Fraser Burch, Takahiro Matsumoto, Jianliang He, Baturay Saglam, Arthur Goldblatt, Zhuoran Yang, Amin Karbasi 7/10 safetyevals
A benchmark for a capability existing security evals skip: not detecting or repairing a vulnerability, but locating the code. VLoc Bench pairs 500 real vulnerabilities from 290 repos across 147 CWE categories with before/after snapshots — the agent gets only a CWE description and read-only terminal access. Results are grim: 27 models and four static analyzers, and the strongest system hits 0.229 File F1, with 38.4% of tasks localized by nobody. Good localizers also hallucinate locations on patched repos, so accuracy doesn't imply reliable post-fix behavior.

News

1

Show HN: Pizza Bot – An inbox for AI agents that work in the background

hn:hn-agent — jd_ 6/10 frameworkstool-useinfra
A self-hosted desktop app that runs background agents behind an email-like inbox: finished work lands in Unread, approval-gated actions in Action. Apache 2.0, no signup or telemetry, bring-your-own provider (Anthropic, Bedrock, Gemini, OpenAI, OpenRouter, or local Ollama). It grew out of an internal Amazon tool that drove a CRM via CDP and Playwright, and now leans on DeepAgents, LangGraph, MCP addons, and checkpointing. The inbox-as-interface framing for long-running agents is the interesting part, and it's actually shippable.
36 more items the ranker flagged but didn't feature

Papers

News

Blogs