2026-06-30
AI Agents
Hamel Husain argues "it's hard to eval" is a product smell—if you can't verify an agent's output, neither can your users, so redesign for verifiability before building evals. Two memory papers collide with reality: "Manufactured Confidence" shows mem0/LangMem-style consolidation upgrades a hedged remark into a confident dated fact agents then obey, with "reportedly" treated like a flat assertion. TraceLab releases 4,300 real Claude Code and Codex sessions (350K LLM steps, 430K tool calls) for serving analysis, and Trellis treats the agent experience graph as queryable database state behind Meta's KernelEvolve.
Papers
5Experience Graphs: The Data Foundation for Self-Improving Agents
Trellis treats the agent's experience graph—artifacts, tool outputs, rewards, sibling comparisons, causal lineage—as first-class, governed, queryable database state rather than disposable JSON checkpoints. The framing is sharp: frontier selection becomes a query, cross-session reuse becomes vector-seeded graph retrieval, training-data extraction becomes a materialized view, and reconstructing past agent state becomes a time-travel query. When the database owns the graph, agents become stateless compute and crash recovery plus horizontal scaling fall out for free. Grounded in KernelEvolve, a production kernel optimizer at Meta.
Manufactured Confidence: How Memory Consolidation Turns Hearsay into Confident Facts
Memory consolidation manufactures confidence: a casual, hedged remark gets rewritten into a flat dated assertion that later steps obey like verified fact—no attacker needed. The agent responds to phrasing confidence, not source: attributed, unattributed, and forged "system of record" claims all grant alike, while the evidential register ("reportedly") is obeyed like a flat assertion on most models. Passive "unverified" tags are ignored; active "do not trust this" escalates even correct memory. The real fix is keeping tentative phrasing in the store, plus one redundant source.
TraceLab: Characterizing Coding Agent Workloads for LLM Serving
TraceLab releases roughly 4,300 real coding-agent sessions—about 350K LLM steps and 430K tool calls—collected from day-to-day Claude Code and Codex use, filling a genuine gap since public traces don't capture serving-relevant workload patterns. The analysis finds long autonomous loops, long contexts with short outputs, heavily-tailed tool-call distributions, and high-but-imperfect prefix cache hit rates. These point to concrete serving wins: lower-overhead tool calling, append-length-aware prefill, semantic tool-latency prediction, and KV-cache management around human-paced gaps. Dataset, pipeline, and analysis code are all released.
Qwen-AgentWorld: Language World Models for General Agents
Qwen-AgentWorld trains language world models that simulate agentic environments across seven domains via long chain-of-thought, in 35B-A3B and 397B-A17B MoE sizes, from 10M+ real interaction trajectories. The three-stage pipeline—CPT for state-transition dynamics, SFT for next-state-prediction reasoning, RL with hybrid rubric-and-rule rewards for fidelity—is paired with AgentWorldBench, built from five frontier models on nine benchmarks. The decoupled-simulator use case is the interesting part: a learned environment model for scaling rollouts without live execution. Self-reported wins over frontier models warrant independent checking.
Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent
Agents-A1 is a 35B MoE that claims trillion-parameter-level results by scaling agent horizon rather than parameters, training on trajectories averaging 45K tokens. The recipe: full-domain SFT, per-domain teacher models, then multi-teacher domain-routed on-policy distillation with salient vocabulary alignment to unify six domains into one deployable student. Reported numbers beat or match Kimi-K2.6 and DeepSeek-V4-pro on SEAL-0 (56.4), IFBench (80.6), and FrontierScience-Olympiad (79.0). The distillation method is the real contribution; treat the unfamiliar competitor model names with caution.
Blogs
2“It’s Hard to Eval” Is a Product Smell
Husain reframes the common "our product is hard to eval" objection as a product defect: if you can't verify an output cheaply, users have to redo the work from scratch to trust it. Drawing on three products he advised—including an internal AI data agent answering business questions—he shows before/after sketches arguing that designing for verifiability comes before building evals. The point lands because it ties eval difficulty to UX surface area, not measurement tooling.
Ornith-1.0: Self-Scaffolding LLMs for Agentic Coding
DeepReinforce's first release is an MIT-licensed coding model built on Gemma 4 and Qwen 3.5 (both Apache 2.0), shipping in 9B/31B dense and 35B/397B MoE variants with self-scaffolding for agentic coding. Willison ran the 35B Q4 GGUF (20GB) in LM Studio against a Datasette checkout, finding it competent over many tool calls—locating cookie-decoding code and dialog handlers on request. Promising open weights, but these are hands-on impressions, not measured agent-harness benchmarks.
25 more items the ranker flagged but didn't feature
Papers
- AI Training Manager: Bounded Closed-Loop Control of Adaptive Training Recipesarxiv:arxiv-agents
- COHORT: Collaborative Orchestration for Hardening via Offensive Replay on Emulated Topologiesarxiv:arxiv-agents
- EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failuresarxiv:arxiv-software-eng
- Automating the Design of Embodied AgentArchitecturesarxiv:arxiv-agents
- Linguistic Firewall: Geometry as Defense in Multi-Agent Systems Routingarxiv:arxiv-agents
- Clarus: Coordinating Autonomous Research Agents toward Web-Scale Scientific Collaborationarxiv:arxiv-agents
- PyMETA: A Benchmark Dataset for Hierarchical Student Code Error Classification with Python-Interpreter-Based Labelsarxiv:arxiv-software-eng
- MirrorCode: AI can rebuild entire programs from behavior alonearxiv:arxiv-agents
- SAGA: Scene-Aware, Goal-Evolving Agents for Long-Horizon CivRealm Strategy Planningarxiv:arxiv-agents
- The Illusion of Agentic Complexity in README.md Generation: Evaluating Single-Agent vs. Multi-Agent RAG Systemsarxiv:arxiv-software-eng
- RoAd-RL: A Unified Library and Benchmark for Robust Adversarial Reinforcement Learningarxiv:arxiv-agents
- GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshotsarxiv:arxiv-agents
- ManimAgent: Self-Evolving Multimodal Agents for Visual Educationarxiv:arxiv-agents
- MicroAgent: Context-Augmented Multi-Agent Framework for Automatic Microservice Decompositionarxiv:arxiv-software-eng
- Optimizing Expert-Designed Crystal Graph Networks for Band-Gap Prediction with an Autonomous LLM Research Looparxiv:arxiv-agents
- Translating Natural Language to Strategic Temporal Specifications via LLMsarxiv:arxiv-agents
- SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematicsarxiv:arxiv-agents
- TopoAgent: An Agentic Framework for Automated Topology Learning in Medical Imagingarxiv:arxiv-agents
- Field Order Should Not Matter: Permutation-Invariant Embedding Model Fine-Tuning for Structured Metadata Retrievalarxiv:arxiv-agents
- BayesEvolve: Explicit Belief States for Autonomous Scientific Discoveryarxiv:arxiv-agents
- Collective cooperation without individual fidelity in LLM agentsarxiv:arxiv-agents
- Neural Procedural Memory: Empowering LLM Agents with Implicit Activation Steeringarxiv:arxiv-agents
- ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learningarxiv:arxiv-agents
- CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agentsarxiv:arxiv-agents
- Towards Continual Motion-Language Agents: LoRA Variants for Incremental Motion Understanding and Generationarxiv:arxiv-agents