AI Agents

This paper introduces PERFOPT-Bench, a benchmark for evaluating coding agents on software performance optimization. It moves beyond simple functional correctness to assess the full engineering loop: profiling, diagnosing bottlenecks, editing code, and verifying speedups. The evaluation of seven agent stacks shows that performance is workload-dependent; no single model or framework dominates. The authors also warn that raw speedup can be a misleading metric, as some agents find benchmark-specific shortcuts rather than generalizable optimizations.

This paper tackles the specific infrastructure challenge of scheduling LLM requests for agentic workloads. The authors find that agent requests have different characteristics than human chat: response completion (TPS) is more important than per-token latency, and KV-cache reuse is much higher. Existing schedulers over-optimize for cache locality, creating load imbalances. SMetric proposes a balanced session-centric approach: route a session's first request for load balancing, and subsequent requests for cache locality, improving overall TPS.

DeepSWE is a new benchmark for coding agents that directly addresses two major flaws in its predecessor, SWE-bench. First, its 113 tasks are written from scratch and not contributed upstream, preventing models from simply recalling solutions seen during pretraining. Second, it uses hand-written verifiers that check for the requested functionality, rather than relying on the specific tests that shipped with one particular fix. This results in a more robust evaluation that better separates the performance of frontier models.

When a multi-agent system fails, which agent is to blame? This paper introduces AgentLocate, a framework for failure localization. It's a hard problem due to long-horizon interactions and coupled behaviors. AgentLocate uses an LLM judge combined with verification from multiple independent evaluators to pinpoint both the responsible agent and the first step where the trajectory went wrong. The framework is designed to be efficient and shows strong performance on two benchmarks, outperforming existing methods for attribution.

This paper presents LogNLQ, a framework for querying raw system logs using natural language. The key insight is that you can't reliably generate structured queries (like SQL) for unstructured text. LogNLQ first parses raw logs to induce a relational schema, then uses an LLM to translate a natural language question into executable SQL against that generated schema. The process involves partitioning logs by template and using semantic grounding to annotate table and column names, creating a structured surface for the agent to query.

This paper proposes Prismata, a defense against prompt injection for web agents that draws on classic security principles. It treats the problem as analogous to Cross-Site Scripting, where mixing trusted and untrusted content is dangerous. Prismata works by dynamically deriving permission labels for page content based on structure and enforcing "contextual least privilege." It redacts untrusted content from the agent's view and restricts its capabilities accordingly, all without requiring developer annotations on websites.

This paper introduces Test-Time Harness Evolution (TTHE), a novel paradigm for agent adaptation. Instead of optimizing an agent's harness (the surrounding code for tool use, verification, etc.) before deployment, TTHE evolves it during evaluation on test inputs. It maintains a population of harnesses and uses an agentic proposer/judge system to refine and select better ones based on unlabeled execution traces. This allows the agent system to adapt to new failure modes without retraining the core model.

WebSwarm is a multi-agent framework for complex web search tasks that goes beyond simple parallel execution. It uses a recursive delegation model where agents can dynamically create child agents to handle sub-tasks. Each "agentic search node" has a local objective and can either solve it or delegate further, returning evidence upwards. This allows the system to build a search plan progressively, adapting to the information structure it finds on the web and enabling both deep and wide searches that a single agent would struggle with.

Bug reports are often incomplete specifications for repair agents. This paper's TrajSpec system addresses this by first running a preliminary agent on the original bug report to generate an execution trajectory. It then analyzes this trajectory to extract evidence about the issue, which it uses to refine the original report into a much more detailed specification. This refined spec, now grounded in repository observations, is then fed to the final repair agent, significantly boosting its success rate.

Databricks provides a detailed look at how they benchmarked various coding agents against their own multi-million-line codebase. This is a far cry from small, synthetic benchmarks. They go into the practical challenges of evaluating agents on large, complex, and proprietary codebases, offering a credible and deeply insightful look at real-world performance. The post is less about crowning a winner and more about the methodology of building a meaningful, internal benchmark for a specific, high-value engineering task.

This project demonstrates a clever approach to agent tool creation: a browser-based agent that observes a web app's internal API calls and automatically generates tools from them. Instead of relying on brittle UI automation or waiting for vendors to ship APIs, this technique reverse-engineers the existing private API surface. The demos on Jira and Spotify are compelling, showing how this can create deeply integrated assistants with minimal upfront engineering, essentially auto-generating an MCP server for any web application.

This "Show HN" presents an interactive atlas of 8.5 million research papers, using SPECTER2 and UMAP to create a 2D map of the research landscape. Each paper has a detailed page with LLM summaries, code links, and peer reviews. Critically for agent developers, the project also exposes this structured data via an MCP server, allowing agents to programmatically query the research graph. It's a powerful tool for human researchers that is also designed from the ground up for agent consumption.

Microsoft has released Flint, an open-source "visualization intermediate language" designed specifically for AI agents. The core idea is that existing visualization languages are too low-level, forcing agents to specify complex details and leading to brittle outputs. Flint provides a higher-level, semantic specification and uses a layout optimization engine to produce good charts from simple specs. An accompanying MCP server makes it directly pluggable into agent applications, treating chart generation as a compiled target.

A fascinating piece of practical engineering, this project—Colibrì—documents getting the massive 744B GLM 5.2 MoE model running on a consumer machine with 32GB of RAM. The process involved int4 conversion and a clever memory management scheme where only a small fraction of expert parameters are swapped from token to token. The result is slow, at 0.1 tok/s, but avoids OOM errors entirely. It's a deep dive into the nuts and bolts of running huge models on constrained hardware.

This YC launch, Context.dev, is an API for extracting structured data from websites. It goes beyond scraping by aiming to provide the "useful thing" for an agent or application: clean Markdown, a screenshot, or structured JSON conforming to a provided schema. Use cases include extracting pricing plans, company logos, or support links. It's a service designed to be a tool in an agent's toolbox, handling the messy reality of web data extraction so the agent doesn't have to.

OpenAI has released its new GPT-5.6 model family in three sizes: Luna, Terra, and Sol. The models feature a 1M token context window and a February 2026 knowledge cutoff. Pricing is competitive, but the headline claim is on agentic performance. On the "Agents’ Last Exam" benchmark, the top-end Sol model reportedly outperforms Claude Fable 5 by 13.1 points, while the smaller Terra and Luna models do so at a fraction of the estimated cost, signaling a focus on efficiency for long-running workflows.

Meta has released an API for Muse Spark 1.1, its latest model focused on agentic tool use and computer interaction. The evaluation report details improvements and includes an interesting section on "Attractor States" discovered when having the model converse with itself. Simon Willison has already released an llm plugin, llm-meta-ai, providing CLI access to the new model, demonstrating how quickly the ecosystem can integrate new APIs. The model appears to be a capable tool-user with a focus on practical application.