Particle.news

Benchmark Finds Coding-Agent Harness Can Multiply Model Inference Costs

Runta and Baseten’s large-scale evaluation attributes large cost swings to harness design.

Overview

  • The benchmark ran 22 harness–model combinations 27,280 times and processed roughly 83 billion tokens to measure pass rates, inference cost, and run time.
  • The choice of underlying model explained most cost variance at 92.8 percent, but harness design still shifted median rankings by 11 places and caused up to fivefold cost differences for the same model.
  • Failed runs accounted for 52.7 percent of estimated inference spending and timeouts added 11.8 percent, together consuming about 65 percent of the benchmark’s estimated spend.
  • Runta found agent step count was the only variable with a statistically significant link to cost per successful task (correlation 0.51); prompt size, tool count, and cache hit rates did not show significant correlation.
  • The lowest-cost harnesses tended to rely heavily on shell commands and avoid subagents, but the study did not isolate those design choices so developers must test harnesses directly to know what will cut real costs.