2026-09-14
AI Agents
A new paper asks whether agents can reliably tell you how far along a task is — and finds most deployed models report accurately at the start and end but lose the thread mid-task, so frameworks shouldn't gate task flow on self-reports alone. Another audits which memory losses are actually irreversible when context gets evicted. Alongside: a coding-agent scaffold that separates test-writing from repair, a skill-graph that evolves from traces, and a cleanup of a leaky SWE benchmark.
Papers
5SE-GoS: Self-Evolving Graph-of-Skills for Skill Library at Scale
SE-GoS evolves an existing Graph-of-Skills from execution traces without training, running three updates: topology (discover and prune skill edges), edge-weight (reinforce retrieval-relevant links by past effectiveness), and description rewriting from execution feedback. On SkillsBench across three LLMs, one evolution round lifts reward from 52.4% to 59.4% while cutting input tokens by roughly a third versus loading all skills, and the graph transfers to a held-out split. Gains vary by model family, but the token savings alone justify a look.
The Unreliable Progress Bar: Can LLM Agents Reliably Report Task Progress Throughout Execution?
This study tests whether models can reliably report task progress at every stage, using τ²-bench and StageIF, a controlled testbed placing reporting checkpoints across a task's lifecycle. The finding: reliability is stage-dependent. Most deployed models are accurate before work starts and after it finishes but lose accuracy mid-task; the newest generation closes that mid-task drop and instead grows conservative near the finish. The concrete takeaway is a warning against building agent control loops on self-reported state.
What Eviction Destroys: A Restore-Counterfactual Audit of Forgetting in Agent Memory
A restore-counterfactual audit distinguishes memory losses that eviction makes irreversible from failures that are merely recoverable retrieval misses. The method reinstates gold evidence at read time and reruns the reader, classifying each error as recoverable, irreversible, or residual. On LongMemEval-S with FIFO, random, redundancy-aware, and LLM-importance eviction: at an 80k budget the irreversible share among corrected errors is 0.60 for LLM-importance versus 0.67-0.73 for the rest; at 8k it hits 1.00 for all four. Crucially, budget-accuracy curves aren't comparable unless the retrieval regime is reported.
ExecCritic: Learn to Test, Test to Improve for Coding Agents
ExecCritic separates test construction from source repair to stop coding agents from writing self-agreeing patches and tests. A Test agent generates repository-native tests, a fail-closed harness freezes them, and a Repair agent revises code against that feedback without touching the tests. The ablation is the point: on SWE-bench Verified, base-agent tests drop resolved rate from 61.2% to 57.3%, while GPT-5.6 tests raise it to 65.3%. Role-specific RL lifts the Qwen Test agent's Base-to-Gold from 22.2% to 62.2%; composing both trained roles reaches 72.6%.
SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents
SWE-Bench Pro Verified reworks a widely used SWE-agent benchmark after finding two failure modes: reward hacking from leaked gold solutions or hidden evaluation info, and task quality issues like misleading problem statements and improperly scoped tests. The fix combines anti-hacking safeguards that close leakage channels without breaking normal agent operation with minimal corrections to flawed instances. Re-evaluation shows some models scoring substantially lower than previously reported, implying prior SWE-Bench Pro numbers overstated real coding ability.