AI Coding Evaluation, Testing & Reliability

When companies evaluate AI coding tools, they often start with a public leaderboard and a price-per-million-tokens table. Both are useful, but neither tells you what it actually costs to finish work in your own codebase. A benchmark may not represent your languages, build system, or engineering conventions. A cheap token can also become expensive when an agent reads too much context, retries repeatedly, or fails often enough to require human rescue.

In July 2026, Databricks published an internal benchmark built from real pull requests against its multi-million-line codebase. The tasks span Python, Go, TypeScript, Scala, Rust, Java, Protobuf, gRPC, Bazel, and other technologies. Instead of answering isolated coding questions, agents had to inspect a repository, use tools, modify files, and pass held-out tests.

The official findings are valuable on their own. The more important question for engineering leaders is what happens after pass rate, retry cost, harness behavior, and human rework are included. This article extends the Databricks benchmark with success-adjusted cost calculations, a closer look at context overhead, and a practical enterprise evaluation framework.

Bottom line: The correct procurement unit is not “one model” or “one million tokens.” It is a complete configuration: model × harness × reasoning effort × tool permissions × version. The metric that matters is the cost of work that clears your quality bar.

Alternate GLM 5.2 banner from the original WeChat article.

Chinese-localized screenshot of a social post by Databricks CEO Ali Ghodsi about AI coding costs and model routing — A Chinese-localized screenshot of a post by Ali Ghodsi on X, reproduced from the source article. The benchmark itself was documented by Databricks engineering leaders.

Databricks Measured Real Engineering Work, Not Coding Puzzles

Public benchmarks are useful for tracking the frontier, but they cannot fully represent a private codebase. A dataset built mostly from public GitHub issues will not capture internal APIs, build tools, framework conventions, and years of architectural decisions. Once tasks and solutions are public, benchmark answers may also enter future training data.

Databricks therefore selected candidate tasks from recent internal pull requests and applied several filters:

Bot commits, service accounts, fully AI-generated changes, and generated files were excluded.
Tasks needed high-quality tests that could validate behavior.
Changes were kept relatively self-contained across a small number of modules.
The sample covered frontend, backend, systems code, interface definitions, and build configuration.
Reviewers extracted the real goal and constraints from each pull request while removing solution clues.

Databricks benchmark task pipeline showing filtering, task construction, and human vetting — Databricks filtered recent human-authored pull requests, reconstructed blind coding tasks, and manually vetted the resulting benchmark samples. Source: Databricks.

During evaluation, the agent had to reconstruct the non-test changes from a clean benchmark state. Relevant tests were held back and applied only after the agent declared the task complete. The result was scored with executable tests rather than an LLM judge.

That approach is more objective than asking another model whether the answer “looks right,” but the tests themselves still required engineering. Databricks found cases where an exact-string assertion rejected a valid alternative implementation, so reviewers rewrote tests to validate behavior instead of one developer's precise wording.

Before and after example of replacing a brittle exact-string assertion with a behavioral benchmark test — Some tests were rewritten to check behavior rather than an exact string or implementation detail. Source: Databricks.

Held-out test workflow for evaluating an AI coding agent against a rewound repository — The repository was rewound, the agent worked without the original patch or tests, and held-out tests produced an objective pass-or-fail result. Source: Databricks.

The team also discovered a classic benchmark leak: an agent could walk through Git history and recover the original implementation. Databricks fixed this by isolating the working copy from repository history during each run.

Example of a coding agent recovering an original solution from Git history during a benchmark — A coding agent can recover the original patch from Git history, so the benchmark isolated history to prevent answer leakage. Source: Databricks.

The lesson is broader than this benchmark: building a credible private evaluation set requires task design, held-out verification, environment isolation, and leak prevention. It is not enough to sample a few historical tickets and run an agent against them.

Four Findings That Change How Coding Agents Should Be Compared

1. The Efficient Frontier Does Not Belong to One Vendor

The benchmark's quality-cost Pareto frontier includes OpenAI, Anthropic, and open-weight models. On this workload, no single supplier dominated every useful price-quality point.

Databricks coding-agent benchmark chart plotting pass rate against mean task cost — The Pareto frontier spans proprietary and open-weight models, showing that no vendor dominates every cost and quality point. Source: Databricks.

The charts compare complete model-harness-effort configurations, not isolated models. Databricks found substantial cost differences even inside the same capability band:

In the top band, with pass rates around 82%–90%, GPT-5.5 with Pi at medium effort was about 2.2× cheaper than Opus 4.8 with Claude Code at high effort.
In the middle band, around 71%–82%, Sonnet 4.6 with Claude Code was about 1.8× cheaper than Sonnet 5 with Claude Code.
In the lower band, around 51%–60%, GPT-5.4 Mini with Codex was about 1.3× cheaper than Haiku 4.5 with Claude Code.

Three capability bands for model and harness combinations in the Databricks benchmark — Model-harness combinations cluster into capability bands, with large cost differences among configurations delivering similar pass rates. Source: Databricks.

The practical question is therefore not “Which model is best?” It is: Which complete configuration meets the quality threshold for this task class, under these permissions and this failure budget?

2. Open-Weight Models Have Entered the High-Capability Tier

GLM 5.2 landed in the benchmark's top capability tier and was statistically tied with Opus 4.8 on quality. Databricks reported a mean task cost of $1.28 for GLM 5.2 versus $1.94 for Opus 4.8.

This result aligns with the direction described in Z.ai's GLM 5.2 release. The model targets long-horizon work, provides a one-million-token context option and adjustable reasoning effort, and releases its weights under the MIT License. Z.ai also describes architectural changes such as IndexShare and speculative decoding optimizations intended to reduce long-context serving cost.

The two sources must still be separated. Z.ai reports vendor benchmarks, while Databricks reports a private benchmark from one company's codebase. Neither proves that GLM 5.2 can replace a closed frontier model across every language and repository. The defensible conclusion is that it deserves a place in an internal candidate pool, not that every team should switch by default.

3. A Lower Token Price Can Produce a Higher Task Cost

Sonnet 5 was about 1.7× cheaper per token than Opus 4.8, yet Databricks measured the following task-level results:

Configuration	Mean cost per attempt	Task completion rate	Relative token usage
Opus 4.8	$1.94	87%	1.0×
Sonnet 5	$2.09	81%	1.9×

Sonnet 5 worked longer and read more context, erasing its unit-price advantage. We can extend the published data with a simple success-adjusted calculation:

Opus 4.8: $1.94 ÷ 87% = about $2.23 per first-pass success.
Sonnet 5: $2.09 ÷ 81% = about $2.58 per first-pass success.

On this benchmark, Sonnet 5 cost roughly 16% more for each first-pass success despite its cheaper tokens. The estimate excludes retries, developer waiting time, and rework, so it is not a full total-cost model. It is still enough to show why a pricing table is not a procurement model.

4. The Harness Can Multiply Context Cost

A harness is the execution layer that turns a model into a coding agent. It decides how files are discovered, how the working set is maintained, which tools are called, when the agent retries, how context is compacted, how results are verified, and when the run stops.

Databricks compared Pi with Claude Code or Codex while holding the model and reasoning effort constant. Some paired configurations differed by more than 2× in cost.

Cost comparison between Pi and native coding-agent harnesses using the same underlying models — For the same model, Pi was 1.20×–2.08× cheaper across the tested configurations, while pass-rate differences ranged from −7 to +3 percentage points. Source: Databricks.

The context chart explains much of the gap:

With Opus 4.8, Claude Code re-fed a median of roughly 742k context tokens per task, versus 236k for Pi—a 3.2× difference.
With GPT-5.5, Codex re-fed roughly 1,235k tokens, versus 665k for Pi—a 1.9× difference.

Median context tokens re-fed per task by Claude Code, Codex, and Pi — Repeated context, not only model pricing, drives harness-level cost. Pi re-fed substantially less context in these comparisons. Source: Databricks.

Less context does not automatically mean equal quality. In the Opus 4.8 max-effort pairing, Pi was cheaper but passed 82% of tasks versus 89% for Claude Code. In other pairings, Pi was similar or slightly better. Harness optimization therefore needs a quality floor: the goal is to reduce repeated context and unnecessary turns without crossing the organization's failure threshold.

From Model Price to Expected Business Cost

A more useful comparison starts with a simple equation:

Expected business cost = agent attempt cost + (1 − success rate) × failure impact

Failure impact includes far more than a second API call:

Developer time spent reconstructing context and taking over the task.
Additional CI runs, test-environment usage, and reviewer time.
Delivery delay and requirement rework.
Rollbacks, security incidents, and technical debt when a defect reaches the main branch.
Interactive latency from long-running agent sessions.

For a low-risk configuration change, failure impact may be small enough to justify a cheaper configuration. For authentication logic, database migrations, security fixes, or cross-service refactors, the failure impact may dwarf the model-price difference. A more expensive but more reliable configuration can therefore have the lower business cost.

Routing should not be based only on task complexity. It must also account for risk and verifiability.

The Task Distribution Supports Tiered Routing

Databricks used Unity AI Gateway logs to classify its internal coding-agent sessions:

Trivial: 7.6%
Low: 19.3%
Medium: 61.1%
High: 12.1%

In other words, 26.9% of work was below medium complexity, while only about one in eight tasks was high complexity. Making the most expensive model the default for every task creates a structurally inefficient cost profile.

Databricks internal coding-agent task complexity distribution — Most sessions were medium-complexity tasks; only 12.1% were classified as high complexity. Source: Databricks.

A safer three-tier routing design looks like this:

Low-Risk Tasks With Strong Verification

Examples include configuration changes, dependency updates, and local styling fixes. Start with a lower-cost model and lightweight harness, restrict writable directories, and require deterministic tests. Escalate automatically when verification fails or confidence is low.

Medium-Complexity Tasks

Examples include routine feature work and single-service debugging. Use the primary coding configuration, provide module-level context and repository rules, and enforce token, turn, and time budgets.

High-Risk or Cross-System Tasks

Examples include authorization, data migrations, performance-critical paths, and cross-service refactors. Use a stronger model, require plan review, execute in stages, and add independent verification, security checks, and human approval.

Routing keys should include complexity, risk, test coverage, language, repository area, and escalation conditions—not merely the title or prompt length.

Enterprises Also Need a Control Plane Above the Harness

Databricks extends model and harness portability through Omnigent. According to the Omnigent documentation, it provides a common layer across Claude Code, Codex, Cursor, Pi, and custom agents. Teams can preserve tools, prompts, skills, and policies while changing the underlying harness or model.

The related Unity AI Gateway addresses platform concerns such as identity, permissions, rate limits, budgets, usage tracking, model access, MCP access, and policies that can allow, deny, or require approval for a request.

This direction shows how AI coding is moving from “buy an assistant for each developer” toward “operate a probabilistic software-delivery system.” Omnigent and the current Unity AI Gateway experience are still early-stage or beta products, and some policy and budget features have coverage limits. Intelligent routing should not be presented as a fully solved enterprise capability.

A Practical 30-Day Internal Benchmark Pilot

The following plan is a Graphify recommendation derived from the Databricks method, not an official Databricks prescription.

Week 1: Build a Minimum Task Set

Select 30–50 merged pull requests from the last three months and stratify them by language, module, complexity, and risk. Exclude generated changes and tasks without meaningful verification. Hide the original implementation and Git history.

Thirty to fifty tasks can reveal obvious cost and failure patterns. Do not treat a one-to-three-point pass-rate difference as significant without repeated runs and a power analysis. The pilot should eliminate clearly unsuitable configurations, not produce a universal leaderboard.

Week 2: Standardize the Run Protocol

Pin the model version, harness version, reasoning effort, tool permissions, network policy, system instructions, time budget, and stopping rule. Preserve the full trajectory, patch, test result, token usage, latency, and failure category for every run.

Week 3: Measure Quality and Cost

Track more than pass rate:

Metric	Why it matters
Model cost per successful task	Prevents token-price bias
First-pass and post-escalation success	Measures routing value
Total and repeated context	Identifies harness waste
Median and P95 latency	Tests suitability for interactive work
Human review and rework minutes	Approximates real TCO
Rollbacks, security defects, regressions	Prevents over-optimization for tests
Results by language and task type	Stops averages from hiding weak segments

Week 4: Shadow and Canary

Run candidates in shadow mode before they affect a developer's primary workflow. Then roll out the best configurations as canaries on low-risk tasks. Expand default routing only after quality floors, cost ceilings, and security policies remain stable.

What This Benchmark Still Cannot Tell Us

The method is more production-relevant than a generic leaderboard, but it has important limits:

Databricks did not publish the full sample size, tasks per language, repeated-run count, random seeds, confidence intervals, or the statistical test behind “statistically tied.”
Requiring strong tests and relatively self-contained changes creates selection bias against ambiguous requirements, multi-service migrations, greenfield architecture, and long-running refactors.
Passing held-out tests does not guarantee security, performance, maintainability, architectural consistency, or low technical debt.
Reported cost is primarily a token-cost proxy. It does not fully include open-model infrastructure, latency, developer waiting time, review, and failure recovery.
The private code, tasks, prompts, tool configuration, and trajectories are not public, which limits reproducibility.
Models, default prompts, prices, and harness behavior change quickly. The results are a point-in-time snapshot, not a permanent ranking.

Benchmark contamination is not hypothetical. The SWE-rebench paper continuously collects fresh tasks for decontaminated evaluation and reports evidence that some model performance on SWE-bench Verified may be inflated by contamination. Recent private tasks reduce that risk, but teams must still isolate Git history, caches, logs, and internal knowledge systems that might expose answers.

Procurement and Rollout Checklist

Does the comparison unit include model, harness, reasoning effort, permissions, and version?
Are results segmented by language, task type, complexity, and risk?
Are tests hidden, non-vacuous, behavior-focused, and capable of distinguishing configurations?
Are Git history, networks, caches, and other answer-leak paths isolated?
Is cost measured per successful task rather than per token?
Are latency, human rework, security defects, and rollbacks included?
Do cheaper routes have a quality floor and automatic escalation?
Are new models and harnesses introduced through shadow and canary stages?
Can the organization switch vendors and harnesses without rewriting the entire workflow?
Is the benchmark rerun regularly as models and tools change?

Conclusion: Optimize the Delivery System, Not the Model Score

The most valuable part of the Databricks benchmark is not that one model “won.” It is the shift from token economics to delivery-system economics.

The model sets the capability ceiling. The harness determines how context and tools are used. Routing decides which capability is assigned to which task. Tests and governance decide whether the output can safely enter production. A failure in any layer can make a cheap model expensive—or turn an expensive model into waste.

Engineering teams should stop searching for one default model that solves everything. The better approach is to use their own code, tests, failure costs, and security boundaries to build a continuously updated evaluation and routing system.

Category: AI Coding Evaluation, Testing & Reliability

Databricks AI Coding Agent Benchmark: Why Token Prices Mislead