Overview
- A code‑review vendor compared a low‑cost model (GPT‑5.6 Luna) to a frontier model (Astra) on 50 public pull requests and reported Luna found 69 verified bugs to Astra’s 92 while running at a small fraction of the inference cost.
- The cheap model matched the frontier on many everyday data, logic, and concurrency defects but performed substantially worse on security and authorization bugs, especially in an identity server (Keycloak) where permission reasoning matters.
- Luna produced far more false positives, with roughly one in four findings judged incorrect versus four total incorrect findings for Astra, which raises reviewer noise and raises the human time needed to vet AI comments.
- Authors and commentators recommend a practical routing rule: use low‑cost models as a high‑volume triage layer and send any changes touching authn, authz, or permission logic to higher‑capability models or human review.
- The evaluation has important caveats because Astra helped judge results and the benchmark PRs predate the models’ training cutoffs, so absolute counts should be treated cautiously even though the tradeoffs appear robust.