AI Agents

CHASE attacks a real problem in agent evaluation: when you optimize a harness against a released benchmark, a "bad genius" proposer can learn benchmark-wide shortcuts that inflate scores without improving the agent. It reframes harness evolution as constraint generation over validity-preserving counterfactuals — a Challenger searches for protocol transformations that destroy the gain, a validity firewall checks task semantics are preserved, and surviving counterfactuals enter an archive. There are statistical guarantees linking finite archives to a shortcut-neutralized benchmark. Evaluated on synthetic data and OfficeQA only.

This names a failure mode step-scoped governance can't catch: a Compositional Policy Violation, where every individual step passes its own check but the composed execution violates a whole-trace policy — referral thresholds, authority limits, cumulative sums. Their point is sharp: a predicate over one step can't evaluate a property that step doesn't determine, so better per-step monitors won't help. The taxonomy (Authority Creep, Threshold Laundering, Cumulative Sum Violation, Context Collapse) and a provenance-aware runtime that recomputes guarded quantities from raw traces make this immediately useful for anyone writing agent guardrails in regulated settings.

PACT benchmarks whether enterprise LLM agents follow standing rules under pressure — a persistent user, a hurried manager, a convenient shortcut — across twelve regulated domains and forty-eight multi-turn scenarios. Each item pairs a rule against a rule-violating shortcut and applies pressure across wordings and system-prompt modes, with LLM-as-judge auditing to keep items unambiguous and ungameable. Six complementary metrics roll into a reliability-weighted PACTScore. The construction care around evaluation-aware behavior is the strongest part; the LLM-judge dependency is the obvious thing to scrutinize.

An empirical study of QA practices across 157 open-source agent projects with 100+ GitHub stars, analyzing docs, code, configs, and tests. The findings are a useful mirror: QA concentrates on basic functionality and high-risk actions, safeguards are applied inconsistently across equivalent execution routes, and tests rarely probe boundary, adversarial, or multi-step tool-use failures. Identified risks seldom become end-to-end checks. The gap they name — feature-level testing versus systematic validation that agents stay within bounds when touching untrusted inputs and persistent state — is exactly where most teams are underinvested.

When you edit a document in a RAG or agentic context, causal attention means even a local edit can stale downstream KV states — and a full re-prefill is expensive. This work formulates in-place repair as budgeted recomputation and compares training-free position-selection policies. The counterintuitive result: a contiguous edit-local window beats attention-based, KV-deviation, and structural selectors, recovering at least 0.94 of the post-edit answer margin at 13-21x faster than full re-prefill. Contiguity matters because scattered positions inherit surrounding staleness; the advantage fades when answer-bearing text moves downstream of the edit.

Foremerge tackles a failure git can't see: two parallel coding agents making architecturally incompatible changes — one replacing a class while another extends it — that only collide if their patches happen to touch the same lines. Before editing, each agent publishes an intent with the symbols and scopes it plans to change (--scope symbol:PaymentService=replace). A second agent's conflicting publish returns a HIGH destructive-vs-additive finding before any code is written. It's a single Rust binary sitting above git via one SQLite file, no hooks or history rewriting.

Anthropic released Claude Opus 5.5, drawing heavy HN traffic (280 points). The announcement page is all the input gives us — no benchmark methodology, no agentic-eval detail, no pricing or latency numbers here. For agent builders the questions that matter are tool-use reliability, long-horizon planning, and cost-per-task deltas over the prior Opus, none of which the launch copy settles. Treat the headline as a prompt to run your own harness against it rather than trust the release framing.

jevals swaps LLM-as-judge for typed Jev decisions in eval pipelines. The premise is sound: a judge that returns a calibrated probability instead of free-text-then-parse removes a parsing step, cuts cost, and gives you a threshold to tune. If it's cheaper and more deterministic than a chat-model judge, that's a real methodology win for anyone running large eval sweeps. HN engagement was thin (43 points, 6 comments) and the input is title-only, so the calibration quality versus a strong LLM judge is unverified here.

A practitioner writeup on using coding agents as an iterative Rust optimizer: rather than hand-tuning, you loop the agent with a directive to make the code faster and measure each pass. It's the kind of concrete, measured technique the abstract benchmark posts skip — the interesting part is the loop structure and how you keep the agent from breaking correctness while chasing speed. Input is title-only here; the mechanics and the actual speedups are behind the link.

A from-scratch continual-learning model built to train on 8GB of consumer VRAM, motivated by the author's frustration that hobbyists can't train even 1B-scale models locally. The interesting idea is a MoE with many experts that get added and pruned during training, plus a claimed promising scaling curve. It's self-admittedly early and not agent-focused, and "AGI" in the name is a red flag the author owns up to. Worth a look for the training mechanics, not the branding.

JetBrains launched Air, a system of products for agentic software development. The framing — a coordinated set rather than a single assistant — hints at IDE-integrated agents spanning the workflow, which is on-beat for anyone evaluating where coding agents live relative to their editor. The announcement page caps what we can say: no detail here on the agent runtime, model choices, tool interfaces, or how it coordinates with existing terminal agents. HN gave it moderate traction (62 points, 98 comments), so the community reaction is worth reading alongside the launch.

TypeSafe AI's Jev takes text in and returns floating-point numbers — category choices, yes/no answers, ratings, and confidence scores — rather than tokens. The pitch is a "frontier-intelligence function call": unstructured state in, typed probabilistic decisions out. It's fast and charges only for input at $0.042/million, undercutting GPT-5 Nano. That's a genuinely different shape for classification, routing, and reranking work you'd otherwise contort a chat model into. The tradeoff is opacity: a black-box scorer with no reasoning trace to audit.

A concrete LLM plugin wiring up Jev for local use. You install it, set a key, and ask yes/no "noul" questions (llm -m jev 'Please refund my last payment.' -s 'Does this explicitly request a refund?' returns {"type":"noul","noul":0.99}) or choice questions with named criteria for routing. This is the fastest way to actually try the decision-model shape against your own routing or classification tasks without committing to an SDK. It's a 0.1a0 alpha, so expect rough edges.

Willison's rebuttal to "MCP was always a bad idea": the critique only holds if you're running a full-blown terminal agent with unfettered internet access, where direct API calls suffice. For anything less YOLO, MCP buys you control over which services the agent reaches, auth that keeps API keys out of the agent's hands, a UI for connecting services, and audit logging. Calling MCP obsolete because coding agents don't need it misses everything else you'd want to build with governed tool access.