2026-10-03

AI Agents

A 584-run study takes one coding agent on one real ML task and finds that rerunning the same config varies more than switching agents entirely — making few-run comparisons meaningless and a handful of rule-breaking runs the top scorers. Elsewhere: a per-turn context compaction scheme with measured tradeoffs, an audit of whether tool-agent benchmarks actually test what they claim, a privacy-boundary memory architecture, and a local MCP memory recorder.

5 papers 1 news 0 blogs 33 appendix 497 considered

Papers

5

EP-Mem: Elastic Privacy Memory for Social Relationship-Aware LLM Agents

arxiv:arxiv-agents — Fengzhou Sun, Yuan Zhang, Xintong Yu, Jinyao Yan 8/10 safetyobservability
EP-Mem reframes agent memory privacy as user-owned boundary control across social roles, enforcing token-level disclosure rules through a pluggable sidecar during generation, storage, and retrieval. Users set domain-level circulation defaults plus fact-level whitelist/blacklist exceptions. On the authors' own EP-Bench — a long-term multi-party benchmark they built — it hits 94% privacy classification accuracy, lifts disclosure-permission judgment from 22% to 68%, and cuts leakage 75.6% while preserving retrieval. Self-constructed benchmark, but the sidecar enforcement model is the transferable idea.

Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models

arxiv:arxiv-software-eng — Eduardo Ariño de la Rubia, Szilard Pafka 8/10 code-agentsevals
The sharpest reliability result in a while: across 584 runs on one XGBoost-tuning task, run-to-run variance of a single agent-model pairing exceeded the gap between pairings, so ranking agents on a few runs is noise. Resolving real differences took tens to 100+ runs each. Under 1-in-20 runs broke the task's data rules — and those scored highest. Rejecting rule-breakers then keeping the best compliant run reliably improved the result. Cost differed 20x between agents, mostly via prompt cache.

Continuous Context Management

arxiv:arxiv-agents — William Hoy, Jingxuan Fan, Nurcin Celik, Xu Pan 8/10 memorycost-latency
Continuous Context Management compacts the transcript every turn instead of at a threshold: each prompt carries only the task, retained memory, and newest observation. Without fine-tuning on TerminalBench-2, it slashes cumulative input and prompt size but lowers success for most models — Kimi K3 held steady. The fix is GRPO with privileged full-history distillation, scoring student actions against reconstructed full history for dense supervision. On WebShop it beat full-history GRPO for Qwen3-4B but not 8B. Honest about where the tradeoff bites.

Can Agents Design Libraries for Agents?

arxiv:arxiv-software-eng — Gabriel Orlanski, Alex L. Zhang, Avi Trost, Vincent Sunn Chen, Frederic Sala, Aws Albarghouthi, Ludwig Schmidt 8/10 evalscode-agents
LibraryDesignBench asks whether agents can build libraries that other agents actually reuse: 242 expert-validated problems across 15 design tasks in four languages, scored by correctness and simplicity of programs three downstream agents write against them. Agent designers reproduced the production library's abstractions on 11 of 15 tasks, but downstream agents underused every library — reimplementing existing capabilities not because they were missing, but because agent-written libraries were rigid or awkward to call. Prescriptive, agent-first guidance and subagent testing improved reuse.

Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments

arxiv:arxiv-agents — Rohith Reddy Bellibatlu, Zichong Wang, Wenbin Zhang 8/10 safetytool-use
This audit treats a tool's advertised interface as an executable contract and checks the implementation against it, then traces benchmark scores back to the tool state a defective tool should have written. Across 34 mutating tools in four benchmarks, they confirmed seven tool defects and an evaluator property at pinned commits — defects that recur on every rerun because the harness grades what the simulated call reports, not what it did. The static checker found most sites alone; the dynamic half mainly confirmed and traced.

News

1

Show HN: Breadcrumb, record everything on your mac + context manager for AI

hn:hn-mcp — jv22222 6/10 memorytool-useobservability
A local, encrypted Mac recorder that captures screen, meetings, AI transcripts, and subagent/tool activity, then exposes it to coding agents as 30+ MCP tools for search and rule injection. Works with Claude Code, Codex, Cursor, and opencode. The interesting part is the feedback loop: recorded context and user-taught rules get surfaced back into prompts contextually. This is a solo three-month build with 44 HN points — architecture is described but there's no eval of retrieval quality or memory-induced drift.
33 more items the ranker flagged but didn't feature

Papers

News