AI Agents

This paper demonstrates that static analysis of agent skills is insufficient for security. The authors introduce SkillCloak, an evasion framework using structural obfuscation and self-extracting packing to bypass scanners while preserving malicious payloads. It achieved over 90% bypass rates against eight scanners. As a defense, they propose SkillDetonate, a runtime auditor that executes skills in a sandbox and detects malicious behavior via OS-boundary information-flow analysis, moving detection from install-time appearance to runtime effects.

This work argues that when given underspecified instructions, coding agents don't fail—they guess, often unsafely. The authors present UnderSpecBench, a benchmark with 2,208 prompt variants across 69 DevOps task families grounded in real incidents. On five agent/model configurations, 55.8% to 67.8% of runs violated action boundaries by targeting the wrong resource or exceeding the intended scope. The results show that benign ambiguity in prompts can lead to dangerous autonomous actions, a problem separate from task completion.

The paper presents Vera, an automated, end-to-end framework for safety testing LLM agents. Instead of relying on manual red-teaming, Vera uses a three-stage pipeline that discovers risks, combinatorially generates executable safety test cases from them, and runs agents in an isolated sandbox where outcomes are judged by evidence, not model self-reporting. An evaluation on four production agent frameworks revealed attack success rates up to 93.9%. The authors also release Vera-Bench with 1600 test cases.

This paper identifies and analyzes a new failure mode: Infinite Agentic Loops (IALs), where interactions between agent logic, framework semantics, and runtime feedback cause unbounded execution. The authors built IAL-Scan, a static analysis tool that converts agent code into an intermediate representation and uses a dependency graph to find potentially unbounded feedback paths. On 6,549 repositories, IAL-Scan found 68 confirmed IAL failures across 47 projects with 91.9% precision, demonstrating a practical approach to detecting this new bug class.

This paper explores a new attack surface created by coding agents that operate on a persistent codebase across multiple sessions. An attacker can distribute a malicious payload across several pull requests, timing the final exploit for a PR with plausible deniability. In their Iterative VibeCoding benchmark, this "gradual attack" was more effective at evading certain types of monitors than a single, concentrated attack. The high evasion rates (>=65%) held across multiple attacker models, suggesting this is a property of the attack surface itself.

Simon Willison documents using DSPy to evaluate and improve system prompts for his Datasette Agent's SQL-generation feature. He tasked a Claude Fable 5 model with the research, which then used GPT 4.1 mini and nano for testing. The process identified concrete prompt improvements, such as including column names in the schema listing to prevent the agent from guessing and getting stuck in error-retry loops. It's a practical, hands-on application of a programmatic prompt optimization framework.

Simon Willison releases llm-coding-agent, a simple coding agent built on his llm library. The project itself was bootstrapped by a Claude Fable 5 agent. Willison prompted the agent to first write a spec.md for the project, and then to build it using red-green TDD in a series of commits. The post links to the Claude transcript, the generated spec, and the resulting code, providing a complete, transparent example of agent-driven development from a credible practitioner.