AI Agents

AutoPass is a multi-agent framework for compiler performance tuning that rejects the black-box approach. Instead of just trying compiler flags, its agents inspect the compiler's internal state and intermediate representation (IR) to make decisions. The system uses runtime feedback to diagnose performance regressions and refine optimization configurations. On LLVM, it achieves speedups of 1.043x on x86-64 and 1.117x on ARM64 over -O3 without any task-specific training, demonstrating the power of "white-box" agent interaction with complex tools.

This paper details a multi-agent pipeline for automatically generating unit tests in a constrained C firmware environment (AMD's openSIL). The system generates test scaffolds, creates or reuses test doubles, and uses an iterative repair loop driven by compiler logs and line-coverage feedback to fix build errors. The workflow successfully generated compilable unit tests for 73 of 76 functions, achieving 98.8% line coverage in its best configuration, showing a practical path to automating test creation in complex, low-level codebases.

Current benchmarks for LLM-generated GPU kernels often use simple allclose-style checks on a single input shape, creating a "correctness illusion." This paper demonstrates this weakness by building a corpus of buggy kernels that pass standard checks but fail under more rigorous testing. The authors propose an op-schema-aware fuzzing oracle that successfully identifies 10 out of 10 buggy kernels while passing 16 correct ones across five different GPU classes. Every failure is deterministically reproducible from a stored seed.

StaminaBench introduces a new evaluation paradigm focused on "stamina": how many consecutive change requests an agent can handle before failure. Agents must implement and then modify a REST API server over 100 interaction turns, a scenario much closer to real-world development than single-task benchmarks. In their evaluation, all tested models failed within 5-6 turns without a feedback loop. Allowing the agent to see test failures and retry improved performance by up to 12x, highlighting the critical importance of iterative, test-driven workflows.

ORAgentBench evaluates agents on end-to-end operations research tasks, moving beyond simplified or pre-formalized problems. Each task is packaged in an isolated environment with natural language briefs and data files; agents must write and execute code to produce a solution. The results are stark: the best-performing agent (out of 14) passes only 35.5% of tasks. Failure analysis reveals that agents struggle with strategic thinking, missing operational rules, and constructing brittle problem formulations, indicating a significant gap before they are reliable for real-world OR work.

Standard tool-calling agents implicitly manage state by re-parsing the entire prompt context on each turn, leading to errors. LedgerAgent introduces an explicit, structured "ledger" to maintain task state (facts, identifiers, constraints). This ledger is rendered into the prompt for decision-making and is also used to pre-flight tool calls against state-dependent policies, blocking violations before they happen. This architectural pattern improves task success and policy adherence by separating state management from the LLM's reasoning process, making the agent more reliable.

FAPO (Fully Autonomous Prompt Optimization) is a framework that uses an LLM agent to optimize another multi-step LLM pipeline. The meta-agent evaluates the target pipeline, inspects intermediate steps, diagnoses failures, and proposes and validates changes to optimize a score function. Critically, it starts with prompt edits but can escalate to changing the agent's code and chain structure if it identifies a structural bottleneck. The approach shows significant gains over baselines, demonstrating that agents can effectively self-optimize complex workflows.

This paper revisits the classic N-version programming concept for fault tolerance, applying it to code-generating agents. Replicating the seminal Knight-Leveson experiment, the authors generated 48 implementations of a specification using different agents and models. While they found significant common-mode failures, often linked to specification ambiguity, they also found that majority voting across three-version ensembles reduced the mean failure count from 387 to 131. The results provide strong evidence that N-version programming is a practical engineering strategy to improve reliability.

LLM agents often select over-privileged tools when a lower-privilege alternative would suffice, creating a significant security risk. This paper introduces ToolPrivBench, a benchmark specifically designed to measure this failure mode. The authors find that over-privileged selection is common, is amplified by transient tool failures, and is not mitigated by general safety alignment. They propose a privilege-aware post-training defense that successfully teaches agents to prefer lower-privilege tools, demonstrating that explicit training for least-privilege is necessary for tool-using agents.

Agent performance on challenging benchmarks is not a fixed property of a model but is highly dependent on the "inference compute" budget. This large-scale study evaluates 12 models on seven benchmarks, showing that larger token budgets and more submission attempts substantially improve scores. Critically, newer, more capable models benefit more from larger budgets, meaning fixed-budget evaluations increasingly understate their true capabilities. The results are a strong argument that benchmark leaderboards should report performance curves across different compute budgets, not single point estimates.

Simon Willison's new datasette-apps plugin provides a novel pattern for embedding custom HTML/JS applications directly within a Datasette instance. These apps run in a tightly constrained <iframe> sandbox, can execute read-only SQL against the data, and can be granted write access via pre-approved stored queries. The core idea is to enable data-centric micro-apps without compromising the security of the main Datasette instance, with a detailed breakdown of the sandboxing mechanism and its limitations.