AI Agents

A careful reliability study of using Gemini models as audio judges scoring full-duplex voice conversations from raw stereo waveforms, validated against three calibrated humans on 209 sessions across eight dimensions. Gemini 2.5 Flash tracks humans within 0.07 Spearman rho on five of eight dimensions. The sharp finding: rank correlation transfers across the family but calibration does not—3.1 Pro rates several dimensions markedly lower than humans despite comparable ranking. The methodology (human calibration, adversarial defect injection) generalizes well beyond voice.

A framework for generating verifiable terminal environments for RL, releasing SETA-Env: over 4,500 environments, the largest open verifiable terminal RL dataset. Two pipelines share one verification mechanism—one converts diverse sources into standardized environments, the other expands existing ones with adaptive difficulty control. Training Qwen3-8B with GRPO hits 12% on Terminal-Bench 2.0, best reported at 8B scale, and lifts DeepSeek-V4-Flash pass@1 from 40% to 43%. The scarce resource in terminal-agent training is verifiable environments; this attacks exactly that.

A terminal benchmark of 46 long-horizon tasks—experiment reproduction, SE, scientific computing—each decomposed into graded subtasks for dense intermediate rewards and partial credit, rather than the usual pass/fail on final outcome. The cost figures alone are worth reading: agents average 9.9M tokens, ~231 episodes, and 85 minutes per task, and the strongest of 15 frontier models reaches only 15.2%. This exposes both the headroom on genuinely long workflows and the real economics of running them.

A benchmark for visually grounded tool-calling agents: 500+ tools across 16 domains, stateful multi-image multi-turn tasks with goal revisions and error corrections. Twelve models tested; the best clears under 50% success. The revealing part is the failure analysis—53% of failures are wrong information extraction from images, not planning. And a crossover with scale: small models fail at deciding what to do, large models fail at perceiving what they see. That splits the research agenda cleanly by capability level.

A negative result with a fix: LLM agents fail to explore each other, showing myopic and polarized interaction patterns that raise regret and hurt coordination. The authors formalize this as a multi-agent exploration problem over a partially observable stochastic game, then propose MACE, a lightweight framework that promotes exploration through structured peer selection, improving both exploration behavior and downstream performance. They also show theoretically that exploration's value grows with agent diversity. A clean challenge to the assumption that stacking agents yields useful coordination for free.

A tool for handing coding agents a disposable Linux VM instead of your workstation—the obvious isolation boundary that most agent setups skip. The pattern is right: give the agent a throwaway environment where rm -rf and rogue tool calls hit nothing you care about, then discard it. HN traction is solid (199 points). The interesting engineering question is provisioning latency and how state persists across sessions, which the Show HN framing doesn't fully settle.

An AI-designed language whose premise is that agents might prefer syntax not built for humans. The genuinely useful ideas are orthogonal to that framing: effects surfaced in function signatures, a runtime that demands explicit permission to touch filesystem or network, and record-replay of effect interactions. Content-addressed semantic identity means renames skip recompile and retest. These are real capabilities for auditing agent-written code. Whether an AI-authored language earns adoption over Python-with-effects-libraries is entirely unproven—this is a design experiment, not a production tool.

DoorDash describes using panels of LLMs—an "LLM jury"—to generate and validate food metadata, combining multimodal inputs with context optimization. The value here is the production shape: a real engineering team documenting how they aggregate multiple model judgments rather than trusting a single call, and how they manage context to keep it affordable at catalog scale. HN engagement is modest (40 points), and the post is a company blog, so treat cost and accuracy claims as directional rather than benchmarked against a held-out gold set.