Agent evaluations are finally checking whether the work actually got done. And now the protocol underneath them is changing. If you're new to this story, here's where things stand. Novee Security showed that the default coding-agent harnesses for Claude Code, Gemini CLI, and Codex could be driven by low-privilege GitHub issue input. Since then, Claude Code has shipped hardening releases covering path validation, Windows NT-namespace rejection, permission-prompt changes, macOS wildcard read-deny behavior, and prompt-caching fixes for LLM gateway or custom base URL sessions. This is AI Coding Daily: plugin auth, broken MCP assumptions, and a benchmark that asks whether the hotel preference actually got saved. Alice, start there. For updates on this story — Coding-agent harness security — tap follow so the next episode lands in your feed. Here's Anthropic:
Plugin marketplaces: headersHelper on a url marketplace or a catalog entry runs a command that mints HTTP headers (e.g. a short-lived token) for catalog and same-origin archive fetches A catalog entry's headersHelper runs only when you install or update that plugin, after its command is shown; claude plugin install/update ask (or pass -y )
On the Claude Code hardening front, v2.1.238 shifts the focus from prompt-cache fixes to plugin-header auth and runner durability. A marketplace entry can now run a displayed command to mint short-lived headers before an install or update, then asks you for y/N. Better than silent egress, but it's still a remarkably exciting thing to approve when you're just trying to install a plugin. For self-hosted deployments, the runner changes are the material part: shutdown now defers on SIGTERM, and every connection gets a freshly issued Proxy-Authorization header. They also fixed healthy runners getting reassigned after one slow or lost poll—the kind of intermittent failure that only shows up under real load. They also fixed unbounded memory growth in long interactive sessions by releasing subagent results once they leave the recent display window. Good. Your agent shouldn't need the archaeological record of every failed import it hallucinated. Releasing old tool results saves memory, but teams running long sessions should check what the model can still access versus what merely disappears from the terminal. A tidy display alone doesn't prove the context held up. From Liang-Chun Tsai at Microsoft:
We built ThinkingBox to catch this kind of failure. It runs agents in isolated, stateful tool environments, lets a simulated user answer follow-up questions, and inspects the side effects left behind. We’re also releasing ThinkingBox-Bench, a dataset of 507 executable tasks across five business domains.
Microsoft's ThinkingBox-Bench runs 507 executable tasks, gives each of 12 models 20 trials, then checks the records the agent actually changed. Finally, a benchmark where “I fixed it” doesn't pass while the database field is still blank. The quiet-room example nails it: an agent can have a perfectly plausible conversation and still leave special_requests empty. ThinkingBox checks the final state of the booking and the support ticket—the unit of work developers actually need to trust. It'd also catch the PR-agent special: touch three unrelated modules, claim success, and hope nobody reads the diff. A final-state assertion gives CI something concrete to reject. The key engineering detail is that Microsoft split the execution framework from the benchmark package. Teams can update the harness without quietly changing the task definition, so replication might actually mean something. From DEV Community:
That text string is the entire feedback channel back to the model. So write it for the model, not for a human log reader: say what failed, why, and what a valid retry looks like."Error 500" teaches nothing."Date must be YYYY-MM-DD, got '19/08/2026'" fixes the next call.
A successful empty MCP result is the nastiest failure here. Claude Code treats that as evidence, then spends six turns building a plan on data your server never returned. The DEV Community breakdown gets the contract right: JSON-RPC protocol errors are for broken calls; isError: true is the message the model actually sees. Put the invalid date format and the retry condition in that text, not "Error 500." "Empty success" is apparently a status code now. If a weather lookup returns nothing, the tool should say it found nothing—or say it failed—not invite the agent to confidently invent a forecast. And that brings us back to ThinkingBox: inspect the final state. A transcript can sound perfectly rational through six recovery turns, even when the underlying tool result was blank from the first call. Scrolltest, with Promode:
Second, the protocol itself moved, from the 2025-11-25 revision to the 2026-07-28 revision. The new revision removes the connection handshake, removes the session concept, and removes every server-initiated request. That is a bigger deal than the rename, because it changes what your tests can even assert.
The MCP Python SDK 2.0 became stable on July 28, and a plain pip install mcp now pulls it by default. If your QA checks asserted session state, a dependency update may have deleted the thing you were testing without ever throwing an import error. Scrolltest rightly separates the two: the SDK has new dispatch machinery and renames, while the protocol revision removes the handshake, sessions, and server-initiated requests. So the test contract changed underneath your client. The MCP failure modes we just covered get nastier here. A hang or empty-success result is already hard to classify; now a v1-era session assertion can quietly pass because there's no session left to inspect. Congratulations, the smoke test is green and the alarm panel has been removed. For teams that aren't ready to migrate, the official upper bound is mcp>=1.28,<2. Version 1 is maintenance-only and gets security fixes. Pin it now, then rebuild assertions around observable tool outcomes rather than a handshake the 2026-07-28 protocol no longer has. Marcin Uchacz, writing in Unabyss:
We ran the same 20 tasks 60 times in Claude Code, changing one thing: whether the assistant had our context structured and waiting, or had to go assemble it. With context ready, the same work came out 36% cheaper, 38% faster, needed 60% fewer follow-ups, and scored higher on quality - 8.9 vs 7.7 out of 10.
Unabyss ran 20 tasks 60 times in Claude Code. With structured context ready at session start, it saw 36% lower cost, 38% less time, and 60% fewer follow-ups. That's a useful replication target, with numbers behind it rather than a vibes-based productivity claim. And their hand-written CLAUDE.md barely moved the needle. So congratulations to everyone maintaining a lovingly curated text file that the agent skims before asking Slack the same question six turns later. I’d want the task mix, model selection, and token ceilings pinned before treating 36% as portable. But the 8.9 versus 7.7 quality score matters: structured context may reduce both the bill and the cleanup work. Ten MCP servers were connected—GitHub, Slack, Gmail, Linear, Vercel, the lot. If your context layer can't distinguish useful state from a giant permissioned junk drawer, it'll only make expensive wrong answers arrive faster. Got feedback, a story idea, or a correction? Email us at aicodingdaily at lantern podcasts dot com. Your notes help shape future episodes of AI Coding Daily.
You'll find links to every story in the show notes, so dig into the ones you want to explore. Thanks for spending part of your Friday with us. That's AI Coding Daily for today. This is a Lantern Podcast.