2026-10-03
AI Agents
A 584-run study takes one coding agent on one real ML task and finds that rerunning the same config varies more than switching agents entirely — making few-run comparisons meaningless and a handful of rule-breaking runs the top scorers. Elsewhere: a per-turn context compaction scheme with measured tradeoffs, an audit of whether tool-agent benchmarks actually test what they claim, a privacy-boundary memory architecture, and a local MCP memory recorder.
Papers
5EP-Mem: Elastic Privacy Memory for Social Relationship-Aware LLM Agents
EP-Mem reframes agent memory privacy as user-owned boundary control across social roles, enforcing token-level disclosure rules through a pluggable sidecar during generation, storage, and retrieval. Users set domain-level circulation defaults plus fact-level whitelist/blacklist exceptions. On the authors' own EP-Bench — a long-term multi-party benchmark they built — it hits 94% privacy classification accuracy, lifts disclosure-permission judgment from 22% to 68%, and cuts leakage 75.6% while preserving retrieval. Self-constructed benchmark, but the sidecar enforcement model is the transferable idea.
Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models
The sharpest reliability result in a while: across 584 runs on one XGBoost-tuning task, run-to-run variance of a single agent-model pairing exceeded the gap between pairings, so ranking agents on a few runs is noise. Resolving real differences took tens to 100+ runs each. Under 1-in-20 runs broke the task's data rules — and those scored highest. Rejecting rule-breakers then keeping the best compliant run reliably improved the result. Cost differed 20x between agents, mostly via prompt cache.
Continuous Context Management
Continuous Context Management compacts the transcript every turn instead of at a threshold: each prompt carries only the task, retained memory, and newest observation. Without fine-tuning on TerminalBench-2, it slashes cumulative input and prompt size but lowers success for most models — Kimi K3 held steady. The fix is GRPO with privileged full-history distillation, scoring student actions against reconstructed full history for dense supervision. On WebShop it beat full-history GRPO for Qwen3-4B but not 8B. Honest about where the tradeoff bites.
Can Agents Design Libraries for Agents?
LibraryDesignBench asks whether agents can build libraries that other agents actually reuse: 242 expert-validated problems across 15 design tasks in four languages, scored by correctness and simplicity of programs three downstream agents write against them. Agent designers reproduced the production library's abstractions on 11 of 15 tasks, but downstream agents underused every library — reimplementing existing capabilities not because they were missing, but because agent-written libraries were rigid or awkward to call. Prescriptive, agent-first guidance and subagent testing improved reuse.
Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments
This audit treats a tool's advertised interface as an executable contract and checks the implementation against it, then traces benchmark scores back to the tool state a defective tool should have written. Across 34 mutating tools in four benchmarks, they confirmed seven tool defects and an evaluator property at pinned commits — defects that recur on every rerun because the harness grades what the simulated call reports, not what it did. The static checker found most sites alone; the dynamic half mainly confirmed and traced.
News
1Show HN: Breadcrumb, record everything on your mac + context manager for AI
A local, encrypted Mac recorder that captures screen, meetings, AI transcripts, and subagent/tool activity, then exposes it to coding agents as 30+ MCP tools for search and rule injection. Works with Claude Code, Codex, Cursor, and opencode. The interesting part is the feedback loop: recorded context and user-taught rules get surfaced back into prompts contextually. This is a solo three-month build with 44 HN points — architecture is described but there's no eval of retrieval quality or memory-induced drift.
33 more items the ranker flagged but didn't feature
Papers
- From Verification Failures to Reusable Guidance for Coding Agentsarxiv:arxiv-software-eng
- EVOKE: Eliciting World Knowledge in Agents for Transferable Decision-Makingarxiv:arxiv-llm-systems
- Autoresearch in Mixed-Integer Linear and Nonlinear Programmingarxiv:arxiv-agents
- CineMR: Tool-Integrated Vision-Language Reasoning for Quantitative Cardiac MRI Assessmentarxiv:arxiv-agents
- Network World Models as Environments for Algorithm Design on Complex Systemsarxiv:arxiv-agents
- Do Better Goal Representations Improve Goal-Conditioned Reinforcement Learning?arxiv:arxiv-agents
- Inference Auctionsarxiv:arxiv-agents
- SEPAL: Separated Expert Pairs with Answer-Level Fusion for Reliable LLM Collaborationarxiv:arxiv-agents
- E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratcharxiv:arxiv-software-eng
- From Dead Code and Static Requirements to Working Engines: Software Revival with Coding Agentsarxiv:arxiv-software-eng
- A Design Theory for AI-Assisted Software Development Derived from Christopher Alexander's Theory of Formarxiv:arxiv-software-eng
- Maat: Independent Deterministic Contract-Based Governance for Multi-Agent LLM Workflowsarxiv:arxiv-software-eng
- Beyond the Prompt: Linking What Developers Ask, Do, and Understand with Coding Agentsarxiv:arxiv-software-eng
- When Does Correction Become Repair? Mechanistic Auditing of Internal Interventions in Tool-Using LLMsarxiv:arxiv-software-eng
- Empty Commitments: When Agents Promise What Their Runtime Cannot Deliverarxiv:arxiv-agents
- Strategies for Deploying AI Agents in Production at Scientific User Facilitiesarxiv:arxiv-software-eng
- No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapsearxiv:arxiv-agents
- RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversationsarxiv:arxiv-agents
- Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamicsarxiv:arxiv-llm-systems
- Personalized State-Transition-Aware Memory for Clinical Agentsarxiv:arxiv-llm-systems
- DuoMind: Enabling Distributed Multi-Robot Coordination with Semantic Communicationarxiv:arxiv-agents
- Architectural Degradation: How to Measure and to Remediatearxiv:arxiv-software-eng
- From Imitation to Reward Discovery: On-Policy Warmup for Agentic RLarxiv:arxiv-agents
- Unknown is not normal: separating language-model extraction from rule-based decision logic for clinical risk scoresarxiv:arxiv-llm-systems
- VISTA: A Visual Harness for Reasoning in an Interactive Worldarxiv:arxiv-agents
- Understanding and Mitigating Library-Related Issues in LLM-Generated Codearxiv:arxiv-software-eng
- ReCast: Contract-Preserving Protection for Fixed-Interface Multimodal Reasoningarxiv:arxiv-agents
- Zero2Repo: Can Coding Agents Build Repositories from Scratch?arxiv:arxiv-software-eng
- How Can Recommendation Feedback Evolve Agent Memory?arxiv:arxiv-agents