AI Agents

An empirical study of how LLM-agent applications are actually tested, built from mined open-source projects plus 2,572 manually labeled test methods across 240 modules and interviews with 10 senior practitioners. The findings are unflattering: testing is dominated by narrow unit tests with simplistic inputs, heavy mocking, and shallow validation, with little coverage of complex interactions, realistic scenarios, or non-functional requirements. They derive a taxonomy of 23 testing patterns. Industry emphasizes non-functional testing more than open source, but the overall picture is thin.

FACTOR separates two credit-assignment decisions in multi-turn agent RL: how much credit each action gets, and how it spreads across that action's tokens. It uses checkpoint-calibrated TD residuals that telescope to the trajectory advantage, plus teacher-student likelihood gaps for token allocation, with per-action normalization to prevent token-length bias and sign flips. Gains hold across ALFWorld, WebShop, and ScienceWorld, largest on the longest-horizon task, and hyperparameters transfer to a larger backbone and different model family without retuning. Ablations point to TD action credit as the dominant driver.

PMCoder couples a hierarchical phase planner with episodic memory for software issue resolution, with bidirectional coupling: the current plan phase conditions memory retrieval, while memory-derived trajectory stats drive stuck detection and replanning. When issue-reproduction verdicts exist, verification is grounded in execution evidence instead of the agent's self-reported completion. On SWE-bench Verified it resolves 25 more cases (+5.0pp) over a harness-matched baseline, with gains persisting even when the reproduction gate never fires, and the direction holds across Claude Haiku 4.5, DeepSeek-V4-Flash, and an OpenHands port.

DiDPO tackles credit assignment specific to coding agents, where a single step packs multiple independent changes into different regions of a diff and outcome or step rewards can't tell which change mattered. The critic-free method builds credit units from diff structure: it splits diffs into sub-diffs, uses a groupability score to select anchors, forms advantage groups, and projects diff-level advantage back to individual tokens. Reported gains exceed 10% over strong agentic RL baselines on long-horizon coding benchmarks, with the codebase available.

Memoir attacks the false-positive problem in SAST tools by turning historical FP alerts into reusable semantic memories. LLM-guided annotation, pattern clustering, and memory synthesis build a knowledge base; retrieval plus semantic verification against taxonomy consistency and security invariants gates predictions, and verified predictions feed back so the memory evolves as new cases accumulate. Evaluated on CWE-Bench-Java with an industrial case study. The evolving-memory design is the interesting part — it addresses the staleness that kills most static FP-reduction methods.

A hands-on guide to GPU passthrough for macOS VMs on Apple Silicon, aimed at getting faster llama.cpp inference inside virtualized environments rather than bare metal. Relevant if you're isolating local agent workloads in VMs but don't want to eat the usual virtualization penalty on token throughput. The writeup drew 292 points on HN; the concrete tradeoff is passthrough setup complexity versus recovered inference speed. Worth reading before you decide local agents must run outside a VM to be usable.

Anthropic, OpenAI, and Google return encrypted chain-of-thought blocks that clients can replay across sessions, users, and models. The attack: take a trace from a frontier model, replay it into a weaker sibling, jailbreak that weaker model, and recover the stronger model's hidden reasoning in plaintext. Simon shows the encrypted blocks are real with a plain curl against the responses API. Concrete, reproducible, and a failure mode of the encrypted-CoT design itself rather than a one-off jailbreak.