A new coding benchmark just made all those victory laps sound awfully premature. This is AI Coding Daily. Today: who’s earning your trust, and what happens when a GitHub issue gets a vote in your CI? First up, the benchmark reset. From Scale Labs:
The benchmark is significantly more challenging than its predecessors; top models score around 23% on the SWE-Bench Pro public set, compared to 70%+ on SWE-Bench Verified. This provides a more accurate measure of an agent’s true problem-solving capabilities in environments that mirror professional software development.
Scale’s SWE-Bench Pro has 1,865 tasks across public, private, and held-out code—and top models are landing around 23%. The 70-plus-percent SWE-Bench Verified victory lap just got a pretty big asterisk. That split changes what “solved” even means. Public open source can be inspected and replicated; held-out and startup-partner code test whether an agent can reason through a codebase it hasn’t effectively rehearsed. Twenty-three percent feels like a much more credible starting point for long-horizon work. Twenty-three percent is also not “near professional-grade.” It’s one successful task in four, before we ask how many involved the kind of 50-file legacy refactor that turns a cheerful demo into an import-hallucination festival. Scale is explicitly targeting contamination, ambiguous issues, and reproducible testing—the areas older benchmarks often sand down. Vendors citing Verified scores now need to put their Pro number next to it, or they’re presenting a very selective picture. Here's Molly Vorwerck at Claude by Anthropic:
Auto mode closes most of that gap. In internal evaluations, the classifier caught more dangerous actions than developers did when clicking through permission prompts by hand, and its performance held up under third-party red-teaming. And because sessions pause less often, Claude works 9x longer between interruptions than under the previous default—across all Claude Code usage.
Claude Code has made auto mode the default, so permission prompts aren’t the main event anymore. Anthropic says its classifier caught more dangerous actions than developers clicking approvals by hand, and sessions now run nine times longer between interruptions. Nine times longer is excellent right up until the agent is confidently touring a regulated repo with an unreviewed shell command. Anthropic’s internal result is a serious claim, but Anthropic is also grading that exam. The detail I care about is that it’s the default. Nuro, Gusto, and Garner Health may have the operating discipline for this; plenty of teams will inherit that setting before they’ve defined which commands, credentials, and production paths their agent may touch. And after that 23 percent SWE-Bench Pro result, I’m not handing an agent a longer leash because it can keep working without asking permission. Show me the classifier’s behavior on our CI policy, our secrets boundaries, and a 50-file PR. This one's from GitHub Changelog:
This week’s updates across GitHub Copilot in the desktop app, CLI, and VS Code help you resume and organize work, review changes, and ask questions without losing context. - Auto now shows which model handled each completed request, plus AI credit and cache details when they’re available.
GitHub’s /btw side-chat is the useful little detail in this August 3 Copilot release. If that shared prompt cache carries context into the side question, it could save a lot of repeated tokens—as long as the credit meter actually tells you what happened. “AI credit and cache details when available” is a very carefully chosen phrase. I’d like usage visibility to be available precisely when the bill is being created. Also, Auto now identifies the model behind each completed request. Good—on a serious codebase, you can’t evaluate a routing layer that changes behavior if it won’t identify the model. And /side had better stay genuinely separate. Parallel questions are great until the supposedly harmless detour picks up enough repository context to become an expensive little mystery. Here's InfoSec Today:
A GitHub issue opened by an account with no repository privileges was enough to execute code on the CI runners behind Anthropic’s and Google’s own coding-agent repositories. On OpenAI’s, it was enough to hijack the next agent run. Novee Security ran the attack against each vendor’s agent in the configuration that the vendor ships by default, and presented the work at Black Hat USA on August 5. Two CVEs came out of it. Both are patched.
A GitHub issue from an account with zero repo privileges got code onto CI runners in the default setups for Claude Code and Gemini CLI. That threat model belongs in every agent adoption doc before somebody wires one into a release workflow. And Gemini’s CVE-2026-12537 is a 10.0: a crafted .gemini/.env reaches host execution before the sandbox even starts. Novee traced the problem to the surrounding harness, which granted authority it hadn’t earned. Here are the patch numbers, because vulnerable builds linger: Gemini CLI 0.39.1, run-gemini-cli 0.1.22, Claude Code 2.1.163. Claude’s affected range starts all the way back at 0.2.54, so ‘we update sometimes’ is not a security control. We just heard Anthropic’s case for auto mode reducing manual permission fatigue. Novee ran this in the shipping default configuration, and Claude Code’s CVE-2026-54316 could leak an API key one character at a time through Hugging Face’s public download counter. Auto mode’s fewer clicks are useful, but untrusted issue content needs a hard boundary before it reaches a privileged runner. If you’re enjoying AI Coding Daily, please subscribe and leave us a review wherever you’re listening. Reviews help other people find the show, and they help us keep bringing you the latest in AI coding.
Links to every story we covered are in the show notes, so check out the ones that caught your attention and dig a little deeper. Thanks for listening to AI Coding Daily. This is a Lantern Podcast.