2026-06-30

AI Agents

Hamel Husain argues "it's hard to eval" is a product smell—if you can't verify an agent's output, neither can your users, so redesign for verifiability before building evals. Two memory papers collide with reality: "Manufactured Confidence" shows mem0/LangMem-style consolidation upgrades a hedged remark into a confident dated fact agents then obey, with "reportedly" treated like a flat assertion. TraceLab releases 4,300 real Claude Code and Codex sessions (350K LLM steps, 430K tool calls) for serving analysis, and Trellis treats the agent experience graph as queryable database state behind Meta's KernelEvolve.

5 papers 0 news 2 blogs 25 appendix 482 considered

Papers

5

Experience Graphs: The Data Foundation for Self-Improving Agents

arxiv:arxiv-agents — Gang Liao, Yujia He, Abdullah Ozturk, Zhouyang Li, Ying Wang, Zhitong Guo, Hongsen Qin, Yaobin Qin, Tao Yang, Zewei Jiang, Dianshi Li, Jort Gemmeke, Jiangyuan Li, Liyuan Li, Nathan Yan, Masha Basmanova, Uladzimir Pashkevich, Matt Steiner, Pedro Pedreira, Rob Fergus, Anirudh Goyal, Carole-Jean Wu, Gaoxiang Liu, Andrew Witten, Daniel J. Abadi 8/10 memoryinfracost-latency
Trellis treats the agent's experience graph—artifacts, tool outputs, rewards, sibling comparisons, causal lineage—as first-class, governed, queryable database state rather than disposable JSON checkpoints. The framing is sharp: frontier selection becomes a query, cross-session reuse becomes vector-seeded graph retrieval, training-data extraction becomes a materialized view, and reconstructing past agent state becomes a time-travel query. When the database owns the graph, agents become stateless compute and crash recovery plus horizontal scaling fall out for free. Grounded in KernelEvolve, a production kernel optimizer at Meta.

Manufactured Confidence: How Memory Consolidation Turns Hearsay into Confident Facts

arxiv:arxiv-llm-systems — Alex Kwon 8/10 memorysafetyobservability
Memory consolidation manufactures confidence: a casual, hedged remark gets rewritten into a flat dated assertion that later steps obey like verified fact—no attacker needed. The agent responds to phrasing confidence, not source: attributed, unattributed, and forged "system of record" claims all grant alike, while the evidential register ("reportedly") is obeyed like a flat assertion on most models. Passive "unverified" tags are ignored; active "do not trust this" escalates even correct memory. The real fix is keeping tentative phrasing in the store, plus one redundant source.

TraceLab: Characterizing Coding Agent Workloads for LLM Serving

arxiv:arxiv-agents — Kan Zhu, Mathew Jacob, Chenxi Ma, Yi Pan, Stephanie Wang, Arvind Krishnamurthy, Baris Kasikci 8/10 code-agentsinfracost-latency
TraceLab releases roughly 4,300 real coding-agent sessions—about 350K LLM steps and 430K tool calls—collected from day-to-day Claude Code and Codex use, filling a genuine gap since public traces don't capture serving-relevant workload patterns. The analysis finds long autonomous loops, long contexts with short outputs, heavily-tailed tool-call distributions, and high-but-imperfect prefix cache hit rates. These point to concrete serving wins: lower-overhead tool calling, append-length-aware prefill, semantic tool-latency prediction, and KV-cache management around human-paced gaps. Dataset, pipeline, and analysis code are all released.

Qwen-AgentWorld: Language World Models for General Agents

arxiv:arxiv-llm-systems — Yuxin Zuo, Zikai Xiao, Li Sheng, Fei Huang, Jianhong Tu, Yuxuan Liu, Tianyi Tang, Xiaomeng Hu, Yang Su, Qingfeng Lan, Yantao Liu, Qin Zhu, Yinger Zhang, Bowen Yu, Haiquan Zhao, Haiyang Xu, Jianxin Yang, Jiayang Cheng, Junyang Wang, Lianghao Deng, Mingfeng Xue, Tianyi Bai, Yang Fan, Yubo Ma, Yucheng Li, Zeyu Cui, Zhihai Wang, Zhihui Xie, Zhuorui Ye, An Yang, Dayiheng Liu, Jingren Zhou, Ning Ding 7/10 researchinfraplanning
Qwen-AgentWorld trains language world models that simulate agentic environments across seven domains via long chain-of-thought, in 35B-A3B and 397B-A17B MoE sizes, from 10M+ real interaction trajectories. The three-stage pipeline—CPT for state-transition dynamics, SFT for next-state-prediction reasoning, RL with hybrid rubric-and-rule rewards for fidelity—is paired with AgentWorldBench, built from five frontier models on nine benchmarks. The decoupled-simulator use case is the interesting part: a learned environment model for scaling rollouts without live execution. Self-reported wins over frontier models warrant independent checking.

Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agent

arxiv:arxiv-llm-systems — Lei Bai, Zongsheng Cao, Yang Chen, Zhiyao Cui, Shangheng Du, Yue Fan, Shiyang Feng, Zijie Guo, Haonan He, Liang He, Xiaohan He, Shuyue Hu, Yusong Hu, Songtao Huang, Yichen Jiang, Hao Li, Xin Li, Dahua Lin, Weihao Lin, Fenghua Ling, Dongrui Liu, Zhuo Liu, Runmin Ma, Chunjiang Mu, Haoyang Peng, Tianshuo Peng, Jinxin Shi, Luohe Shi, Boyuan Sun, Zelin Tan, Shengji Tang, Qianyi Wang, Yiming Wu, Yi Xie, Xiangchao Yan, Jingqi Ye, Peng Ye, Fangchen Yu, Jiakang Yuan, Bihao Zhan, Bo Zhang, Chen Zhang, Shufei Zhang, Shuaiyu Zhang, Wenlong Zhang, Yiqun Zhang, Junpeng Zhao, Zhijie Zhong, Bowen Zhou, Yuhao Zhou 7/10 researchplanning
Agents-A1 is a 35B MoE that claims trillion-parameter-level results by scaling agent horizon rather than parameters, training on trajectories averaging 45K tokens. The recipe: full-domain SFT, per-domain teacher models, then multi-teacher domain-routed on-policy distillation with salient vocabulary alignment to unify six domains into one deployable student. Reported numbers beat or match Kimi-K2.6 and DeepSeek-V4-pro on SEAL-0 (56.4), IFBench (80.6), and FrontierScience-Olympiad (79.0). The distillation method is the real contribution; treat the unfamiliar competitor model names with caution.

Blogs

2

“It’s Hard to Eval” Is a Product Smell

rss:hamel — Hamel Husain 8/10 evals
Husain reframes the common "our product is hard to eval" objection as a product defect: if you can't verify an output cheaply, users have to redo the work from scratch to trust it. Drawing on three products he advised—including an internal AI data agent answering business questions—he shows before/after sketches arguing that designing for verifiability comes before building evals. The point lands because it ties eval difficulty to UX surface area, not measurement tooling.

Ornith-1.0: Self-Scaffolding LLMs for Agentic Coding

rss:simonw 6/10 code-agentsresearch
DeepReinforce's first release is an MIT-licensed coding model built on Gemma 4 and Qwen 3.5 (both Apache 2.0), shipping in 9B/31B dense and 35B/397B MoE variants with self-scaffolding for agentic coding. Willison ran the 35B Q4 GGUF (20GB) in LM Studio against a Datasette checkout, finding it competent over many tool calls—locating cookie-decoding code and dialog handlers on request. Promising open weights, but these are hands-on impressions, not measured agent-harness benchmarks.
25 more items the ranker flagged but didn't feature

Papers