AI Agents

Systematic empirical comparison of grep vs. vector retrieval across four agent harnesses: a custom one (Chronos) and the three provider CLIs (Claude Code, Codex, Gemini CLI). Across a 116-question LongMemEval sample, grep wins on accuracy in most configurations. But the stronger finding is that harness choice and tool-calling style (inline vs. file-based results) matter more than retrieval strategy. A second experiment progressively pollutes queries with irrelevant history — grep degrades less than vector retrieval under distraction.

DOLORES introduces "Deep Reasoning" — an inference-time method that builds task-specific scaffolds through structured meta-cognition rather than hard-coding a reasoning pattern. The formal language represents decompositions over associative inference, formal computation, and recursive subproblem solving. Results: 24.8% average improvement over the strongest scaffold baseline across four hard benchmarks (multi-hop, long-chain QA, long-context aggregation, deep research). An 8B model beats 32B models on some benchmarks. Three model sizes, two model families tested.

SWE-Chain extends SWE-bench to chained package upgrade scenarios: 12 chains, 9 real Python packages, 155 version transitions, 1,660 grounded upgrade requirements synthesized by aligning release notes with actual code diffs. The chained structure is the key design choice — each transition builds on the prior agent state, so errors compound. Best result is Claude-Opus-4.7 at 60.8% resolving, 80.6% precision, 68.5% F1. The 44.8% average resolving rate across nine frontier configs indicates meaningful room for improvement on a task that is routine maintenance work.

Orchard is an open-source framework for training agentic models, built around Orchard Env — a lightweight service for sandbox lifecycle management that is reusable across task domains and pipeline stages. Three training recipes: Orchard-SWE (107K distilled trajectories, credit-assignment SFT, Balanced Adaptive Rollout RL) reaches 67.5% on SWE-bench Verified from Qwen3-30B-A3B-Thinking, new open-source SOTA for the size class. Orchard-GUI hits 74.1% on WebVoyager with only 0.4K distilled trajectories. The environment abstraction is the core contribution — it's what makes the recipes reusable.

AsyncFC decouples LLM token generation from function execution at the execution layer, enabling overlap between decoding and tool calls, and parallelism across independent tool calls — without fine-tuning the model or changing function implementations. The model reasons over symbolic futures representing unresolved calls. Accuracy is preserved on standard function-calling and adapted SE benchmarks while end-to-end latency drops. The "no model changes required" constraint is the practical differentiator; this layers over existing deployments.

Anthropic's official guidance on running Claude Code against large codebases — 222 HN points, 148 comments suggests it hit a real pain point. The raw text is a one-liner, so the specifics are at the link. Expect concrete guidance on context management, CLAUDE.md conventions, and how to scope what the agent reads. Given that most production repos aren't structured for agent traversal, this fills a real documentation gap.

A solo experiment in self-improving code generation: start with Qwen 2.5 7B, have it generate problem/test/solution triples, fine-tune on (broken attempt, working attempt) pairs with the Python interpreter as the only judge. Initial run made the model worse — a grader bug was silently accepting wrong answers. After fixing it: 25 → 131 correct on HumanEval, beating GPT-3.5, for under $50 in RunPod credits. The grader postmortem is the most instructive part — silent grader failures are the dominant risk in this class of pipeline.

A skill framework for deliberate practice with Claude Code and Codex, structured around intentional skill development rather than task completion. 240 HN points suggests practitioners are actively thinking about how to build their own capability when coding with agents — not just what to delegate, but how to stay sharp. The raw text is sparse; the GitHub artifact is the substance.

Hamel Husain ships evals-skills, a Claude Code / Codex skill set that encodes six concrete evaluators and LLM-as-judge calibration with explicit TPR/TNR targets — patterns distilled from watching 50+ teams get evals wrong. The framing mirrors OpenAI's harness engineering post: the agent that instruments your app, queries traces, and runs annotation experiments is doing data science work, and the skill set gives it the right defaults. All major eval vendors now ship MCP servers, so the integration surface is there.

Mozilla used Claude Mythos to go from ~20-30 Firefox security fixes per month to 423 in April alone — a 14-20x increase. The post details two specific finds: a 20-year-old XSLT bug and a 15-year-old <legend> element bug. The productivity jump is explicitly attributed to harness engineering (steering, stacking, filtering) rather than model capability alone. A meaningful fraction of exploit attempts were blocked by Firefox's existing defense-in-depth, which the write-up frames as validation of layered security rather than a limitation of the approach.

Sebastian Raschka walks through coding agent architecture at the component level: tools, memory types, and repo context — how each layer shapes what the LLM can actually act on. The raw text is a one-liner, so depth is behind the link, but the framing is implementation-level rather than survey-level: how these pieces are wired together in practice, not just named. For engineers who have absorbed the theory and want a structured mental model for their own agent designs, this is the kind of breakdown that holds up as a reference.

Hamel argues that building the evaluation harness for LLM products is fundamentally data science work — and that data scientists and MLEs who've felt sidelined by the foundation-model API era are actually on the critical path again. The argument leans on the OpenAI harness engineering case (three engineers, five months, ~1M LOC) and Karpathy examples: the infrastructure around the model is what makes it work. This is less a career take than a production architecture argument about where the value is built.

Simon Willison reports a personal realization from a Heavybit podcast: vibe coding and agentic engineering are converging in his own practice. He coined the distinction in early 2025 and held it firmly — responsible AI-assisted coding requires understanding the output — but finds himself increasingly trusting generated code he hasn't fully read. The candor here is the value: a practitioner who has written at length about code review responsibility documenting where his own discipline is eroding.

Simon Willison documents shebang-line patterns for running LLM tools as executable scripts: #!/usr/bin/env -S llm -f for fragment-based prompts, -T name_of_tool for inline tool calls, and a YAML template form that defines Python functions as tools directly in the script header. The YAML form in particular is novel — it collapses the gap between a shell script and a tool-using agent into a single file. Concrete code examples, no setup beyond the llm CLI.