AI Agents

A dissection of why LLM vulnerability-patching benchmarks overstate performance, broken into agent-level (prompting and tooling inflate pass rates without improving patch quality), framework-level (permission errors, infra bugs, and timeout handling silently distort scores), and dataset-level factors (single-PoC tests don't check whether a fix addresses root cause). Backed by 112 historical bugs across 84 C/C++/Go/Rust projects with PoC, regression, and developer-intent tests. The sting: developer-test pass rates stay low and barely improve with newer models, meaning agents increasingly suppress symptoms rather than ship upstream-quality fixes.

A controlled study of dynamic concurrency — agents deciding at runtime whether to spawn parallel sub-agents — across matched Codex, Claude Code, and Kimi Code runs with the policy toggled on and off. 354 tasks, 2,124 executions spanning task complexity and horizon length. The finding worth internalizing: on short tasks model capability dominates, but on long-horizon development orchestration becomes the deciding factor. The paper catalogs 13 concurrency-specific failure modes and 28 observable patterns, plus the conditions where parallelism actually helps rather than hurts.

Instead of using uncertainty to decide when to correct an agent, this work learns the value of steering directly. The authors built a stepwise outcome table — ~82,000 counterfactual continuations of 1,864 trajectories across three benchmarks and two agents — by steering at every non-terminal step with four mechanisms and running each to completion. The result: uncertainty flags failing trajectories but doesn't locate the right step to intervene. Their VoS monitor learns that from measured outcomes, uses a harm-budgeted trigger to limit disruption of good runs, and beats five uncertainty-triggered baselines in 11 of 12 settings.

A fault-injection layer wrapped around a function-calling benchmark injects four typed faults — timeouts, missing tools, schema drift, corrupted-but-plausible results — at controlled trajectory points, across six models (half reasoning variants) over 1,920 trials. Agents treat 91.3% of explicit tool errors as problems but only 58.8% of plausible wrong values, against a 26.8% false-alarm baseline. Reasoning models fare worse: they notice fewer failures (-9.3pp) and change plan more (+10.4pp) with no recovery gain. Run-to-run nondeterminism (63.3% same-state) makes most fault effects statistically indistinguishable.

SpecGuard tackles reward-hacking before an agent acts: given a task description and codebase, it autoformalizes the intended behavior into a Lean 4 spec, formalizes the tests independently, and uses the Lean kernel to check whether any implementation could satisfy both. When none can, it emits a machine-checked certificate that the task and its tests conflict — the setup that pushes agents to edit tests or hard-code outputs. On conflicted SWE-bench tasks it detects up to 72.8% of conflicts and certifies 51.1%, with roughly a fifth the miss rate of model-based judgment.

An open-source take on Cloudflare's Durable Objects: a named, single-request-at-a-time server with its own embedded SQLite, pitched as a coordination primitive for multi-agent systems. One actor per agent means websocket fan-out scales horizontally, and the serialized execution model protects shared state like a CRM from data races when multiple agents run concurrently. The main sell over Cloudflare is avoiding lock-in to D1/R2/Workers. Early HN traction (39 points); evaluate the perf claims and operational maturity before betting production coordination on it.

Samsung Labs' LittleBit claims sub-1-bit LLM compression via latent factorization. The HN thread (85 points, 23 comments) is the only context provided here; the actual mechanism, accuracy tradeoffs, and inference-path cost all live in the repo. Sub-1-bit is an aggressive target that typically trades quality for memory footprint, so the question for anyone eyeing it is where the degradation lands and whether the factorized representation plays well with existing serving stacks. Read the methodology before extrapolating cost savings.