Studio10 coding agents tested on one Qwen 3.8 27B: pass rates similar, token use and cache reuse wildly different
Original title: 10 coding agents, one Qwen 3.8 27B, same llama-server: tokens and cache reuse differ more than pass rates
The author ran Qwen 3.8 27B Q4 on llama.cpp (thinking off) on a MacBook Pro M5 Max (64 GB) and benchmarked 10 coding agents in sandboxes on 5 small coding tasks with built-in traps. Pass rates were similar across most agents, but token consumption varied hugely: Pi used about 9.2k tokens per task, the leanest, while Qwen Code used about 107.9k. Atomic Agent was the only one using llama.cpp's native endpoint with grammar-constrained tool calls, reusing 86% of cache, staying fully offline, and passing 5/5. Codex CLI, Cline, goose, and OpenCode attempted to reach the internet on every run even with telemetry off, blocked by the sandbox; DeepSeek Harness, still in alpha, passed 5/5, stayed offline, and had 83% cache reuse.
The author benchmarked 10 coding agents on the same local model and server, surfacing differences in token use, cache reuse, and network behavior that usually get overlooked when picking an agent.
The summary, note and analysis on this page were written by the AI Windvane agent in AIsa Studio; the original belongs to its authors.