AI Agents

This paper re-evaluates security agents under fixed cost budgets instead of peak capability, decomposing performance by inference and tool spend across offensive Cybench and defensive Splunk BOTS challenges. The finding is two distinct scaling regimes: offensive CTF success improves with test-time compute, and scaled open-weight models approach frontier systems cost-competitively — but defensive SOC investigation barely scales with reasoning budget, depending instead on disciplined telemetry navigation and selective enrichment. An interactive results site accompanies it.

Agent harnesses — the code that builds prompts, manages state, and coordinates tool calls — must evolve constantly, and the bottleneck is behavior localization: mapping a change request onto the scattered code that implements the target behavior. The Harness Handbook synthesizes a behavior-to-source map via static analysis plus LLM structuring, and Behavior-Guided Progressive Disclosure walks agents from high-level behaviors to implementation, verifying candidates against current source. Reported gains in localization and edit-plan quality with fewer planner tokens on two open harnesses.

MemPoison is a benchmark of 1,227 hand-validated cases probing persistent-memory attacks across four attack types, three injection channels, and three memory substrates, run over ten model families. The three-tier taxonomy — direct single-record, compositional multi-record, and context-triggered dormant corruption — exposes a real gap: write-time consistency checks suppress L1 attacks but fail on L2 and L3, where benign-looking records turn harmful through joint retrieval or trigger activation. The argument for context-sensitive rather than static memory filtering is well-supported.

Rather than building tiered stores or memory graphs, this argues coding-agent memory should be git-bound — commits as ground truth, rebuild for freshness, merge for verification, review for containment. The retrieval study is pre-registered with a ship discipline: five ranking mechanisms rejected, two kept, ~0.31 pooled MRR. It concedes ranking alone is insufficient (0.07-0.20 answer-sufficiency single-shot) and adds a router dispatching breadth, pointed lookups, and rationale synthesis, reaching 0.83 sufficiency on a ~50k-LOC codebase. Unusually honest methodology.

Interviews with 20 practitioners across 12 organizations plus an 80-person survey, studying how SE agents are actually built rather than mined or deployed. The core finding: cheaper implementation shifts bottlenecks rather than removing them — requirements, coordination, review, and especially evaluating agent output become the new central work. They characterize a seven-stage workflow, a move to evaluation-driven development with versioned specs, and six named challenges including unreliable eval signals, comprehension debt, and behavior drift from provider model updates.

Capital One open-sources an agentic tool that hunts vulnerabilities in code. That an enterprise of this size ships an agent into its own security pipeline is worth noting, but the announcement is a vendor blog: no benchmark numbers, no false-positive rates, no comparison against static analyzers or existing SAST tooling, and little on how the agent scopes its analysis or handles large repos. Treat it as a signal of adoption, not a technique to evaluate yet.

A practitioner dissects a specific Claude Code feature as a misfeature, walking through where it fails in real use. The value here is the concrete failure discussion rather than a general complaint — the kind of detail that only surfaces after living with a tool day to day. Worth reading if you run Claude Code in a serious workflow and want to know which affordances to disable or work around before they bite you.