Last week I attempted to solve a thorny political and economic issue by running a small experiment. It involved LLMs, about $20, and the results made me think about how we make decisions when building AI systems.
The setup was simple. An AI agent plays a datacenter developer negotiating with a county over four things: the tax rate, how much power and water it gets, and how fast its permits clear. The developer wants cheap inputs and quick approvals. The county wants tax revenue without its constituents’ utility bills spiking.
Four agents ran the same harness: prompts, tools, and configuration were identical across each agent. Only the model changed. Claude Opus 5 (Anthropic), GPT-5.6 Sol (OpenAI), Kimi K3 (Moonshot), and GLM 5.2 (Z.ai). The last two are open weight models – I used Fireworks.ai to host them. I used OpenRouter for all models.
There are 56,316 possible deals in this negotiation. Each run started with a different set of starting conditions. But, the experiment was designed to make the output judged via simple arithmetic. I have another experiment around adversarial negotiation with another LLM (coming soon).
Three things came out of this, somewhat contrived, “experiment”.
First, everything just worked. One set of prompts ran unmodified across four vendors. Tool calling, reasoning, structured output, and caching all held. A year ago I would not have assumed that.
Second, the models performed about the same and the costs did not. All four captured 85 to 95 percent of the best available deal. Opus 5 finished two points ahead of GLM 5.2, the cheapest model in the lineup. A full run with Opus cost 8x what the same run cost with GLM.
The chart below shows one element of these costs: input tokens. Here caching does most of its work. Published input prices across the four models differ by 3.6x. What I actually paid differed by 13.2x, and the order changes. Sol starts level with Opus on the price sheet and lands cheaper than Kimi K3 through some aggressive prompt caching.
Third, the frontier models were the ones that needed attention. Anthropic requires explicit cache_control breakpoints, which I left untuned, so Opus cost more than it might otherwise. Sol scored lowest and would likely have done better with some prompt tuning. Kimi K3 and GLM 5.2 worked out of the box. I expect both frontier models improve with care, which is somewhat surprising.
Defaulting to the frontier tier can be an expensive decision. It took me a few hours to set up this experiment. It might be worth doing something similar when making architectural decisions.
** Before you flame me:
This is not a particularly scientific experiment. It has a small sample size, uses a contrived scenario, and is probably buggy. I attempted an “apples to apples” comparison. I wanted to show that there could be significant variability in costs for a small variability in performance. This is something any engineer should be very careful about. I have more experiments coming …
