The benchmark got tougher just as the agents got more ambitious. If you're just joining us: Novee Security showed that default coding-agent harnesses for Claude Code, Gemini CLI, and Codex could be abused through low-privilege GitHub issue input. Anthropic followed with a run of Claude Code hardening releases: v2.1.233 and v2.1.234 focused on path validation and rejecting Windows NT namespaces, while v2.1.235 tightened permission prompts, session-wide edit grants, approval-dialog disclosure, and background cloud-session resource use. This is AI Coding Daily. Today: tougher tests, always-on cloud agents, and the awkward question of where their sandbox actually ends. Anthropics writes:
Fixed prompt caching for sessions using an LLM gateway or custom base URL - Added a built-in "Concise" output style: Claude leads with results and skips preamble and narration, while doing the work just as thoroughly. Select it under Output style in /config.
Claude Code 2.1.237 fixes prompt caching through LLM gateways and custom base URLs. If your enterprise proxy quietly disabled caching, that may have been a very expensive configuration detail. That matters because gateways are how plenty of teams put policy and observability around model traffic. Cache behavior has to survive that route; otherwise, token budgets become impossible to reproduce. After Novee’s harness findings, Claude Code’s hardening streak gets another patch: macOS wildcard read-deny rules are tighter. A deny on **/.env now wins even inside an allowed read region, and renaming the file won’t get around it. Good. If a sandbox rule loses to a filename change, it isn’t much of a boundary. And /config now has a built-in Concise style: results first, less narration. Fine—but a setting has to do what it says, same as permission grants. From Cursor:
We're continuing to improve cloud agents and the Cursor harness so always-on agents can operate as a system, building and shipping software on their own without the need for intervention at each loop. With this release, cloud agents can automatically pick up work in response to events, hold a goal until it's met, and stay on course through long-running sessions.
Cursor agents can now wake on PR activity, Slack threads, or a schedule—and an agent-created PR can keep fixing CI and bot comments until it lands. So your pull request has acquired a night shift. The big claim is the subagent architecture: separate VMs, clean context, isolated project copies. I want the operational details—what state returns to the parent, what the token budget looks like over a long /goal session, and what audit trail survives the handoff. “Drive it to completion” needs some definition. Fixing a red CI check is useful. But if it’s repeatedly negotiating with bot comments while a rule file changes under that same long-lived goal, a human should review it before merge. Fresh environments for testing parent-agent changes could be genuinely good engineering. But “isolated” has to mean more than no shared working directory—teams need to know the network access, credentials, logs, and exactly who authorized that always-on subscription. Here's Scale AI:
SWE-Bench Pro is a benchmark designed to provide a rigorous and realistic evaluation of AI agents for software engineering. It was developed to address several limitations in existing benchmarks by tackling four key challenges: 1. Data Contamination: Models have likely seen the evaluation code during training, making it hard to know if they are problem-solving or recalling a memorized solution.
Scale’s SWE-Bench Pro finally tests some of the messier parts of software work. It tackles training-data contamination and ambiguous tickets, then makes the tests reproducible. Docker environments plus fail-to-pass and pass-to-pass checks are good methodology. And the top frontier scores land around 23%, where SWE-Bench Verified was clearing 70% plus. A benchmark that removes two-thirds of the victory lap has my attention. I want the harness split measured carefully. That remaining 77% contains genuine reasoning failures, but it may also contain environment friction; Scale needs to make that boundary inspectable before we treat 23% as a clean capability number. Right—23% on containerized open-source tasks still leaves us a long way from trusting an agent with a 50-file refactor and twelve years of monorepo archaeology. And this benchmark at least isn’t grading agents on whether they remember the answer key. SWE-Bench Pro makes the test tougher, but how do we know an agent is becoming a better production engineer—not just a better contestant in a more elaborate coding exam? We need evidence that holds up across different task types, codebases, and evaluation setups. Scale AI says SWE-Bench Pro uses more diverse, harder tasks that code models haven’t seen before. The leading four frontier models still led, but their scores were much lower than on SWE-Bench Verified. That makes memorized or overly familiar tasks a less likely explanation, though it’s still one benchmark family.
A better signal is performance across the engineering loop. Scale’s SWE Atlas covers 284 tasks, from codebase question-answering to test writing and refactoring. It asks whether an agent can understand a system, validate behavior, and preserve structure, instead of simply patching an issue. And human review has to count. METR found roughly half of test-passing SWE-Bench Verified pull requests produced by agents from mid-2024 through mid- or late-2025 would not be merged by repository maintainers, even after accounting for noise in those decisions. A rising solve rate matters most when merge acceptance rises, regressions fall, and results hold on tasks the agent wasn’t optimized for. But couldn’t a company just pick a broader benchmark and still end up measuring its harness—its prompts, tools, and setup—more than the underlying agent? Exactly. Evaluations should report the whole setup alongside the model score. Research-Driven Engineering Leadership found that, in one analysis, a richer agent harness lifted solve rates more than swapping models. So leaderboard comparisons can blur model capability with tooling. In practice, look for repeat performance across varied repositories and engineering tasks, plus review outcomes showing a team can merge and maintain the resulting code. From Simon Willison at Simon Willison's Weblog:
It quickly ran into a problem: the Claude Code for web environment can't run smol machines. Quoting the notes it wrote: This Claude Code container: Linux 6.18.5-fc-v20 (itself a Firecracker guest), 4 vCPU, 15GB RAM. No /dev/kvm, no vmx/svm CPU flags → no nested virt.
Claude Code for web hit a very practical wall: it’s already inside a Firecracker guest, so there’s no /dev/kvm and smolvm can’t launch its own VM. Turns out the sandbox is already inside another sandbox. And the workaround was GitHub Actions, where the Ubuntu runner exposes KVM. That’s a legitimate test path, but the execution boundary changes with the runner you choose—teams need to plan for that before promising isolated execution of untrusted Python or JavaScript. Right. I want those controls tested under the same conditions as production: stopping a while-true loop, capping RAM and CPU, cutting network access, and limiting the filesystem. Don’t validate them in a hosted web container, then quietly move into CI where the privileges differ. Still, Fable recognized the nested-virtualization limit, ran the battery on a KVM-capable runner, collected logs, and removed the temporary workflow. It exposes the runtime constraint and leaves a reproducible route around it. If you’re enjoying AI Coding Daily, please subscribe and leave us a review wherever you’re listening. Reviews help people find the show and help us keep bringing you useful updates every day.
Links to every story we covered are in the show notes, so take a look at the ones that caught your attention and dig in when you have a moment. That’s AI Coding Daily for today. This is a Lantern Podcast.