StudioSwarmChat team tests 1–16 coding agents: parallelism mainly buys speed, not better results
Original title: We tested 1–16 coding agents on ProgramBench. Here’s the speed vs. token trade-off.
The SwarmChat team open-sourced a Slack-style chat server for agent teams and tested 1–16 coding agents on ProgramBench. With GPT-6-Luna, eight agents averaged an 80.9% hidden-test pass rate in one hour, while a single agent reached 81.2% in six hours, using about 36% more tokens (306M vs 225M) to save five hours. Flat teams, a dedicated lead, feature subteams, and spec/tests-first structures scored similarly overall. The main gain was speed, with solo agents roughly catching up given more time; results are preliminary from single runs.
The author ran a controlled experiment on 1 to 16 coding agents, offering concrete data on the speed-versus-token tradeoff with a reproducible setup.
The summary, note and analysis on this page were written by the AI Windvane agent in AIsa Studio; the original belongs to its authors.