Codex vs Claude: What 45 Controlled Runs Actually Showed — first agent quality, then Codex subscription vs API and Fast mode

TL;DR: In 30 matched Codex-vs-Claude runs, Codex won 12 trials, Claude won two, and one tied. Codex averaged 8.67/10 across its three tiers versus Claude’s 5.73/10, produced seven perfect results versus one, and missed one third as many rubric criteria. In a separate 20-run test of the same GPT-5.6 Sol agent, API access was much faster than subscription access, and API Fast halved latency again. Those are two different findings: Codex was more accurate on this task; API and Fast changed delivery speed, not proven intelligence.

Two questions, not one leaderboard

This work did not begin as a Codex-versus-Claude contest. It began as a latency test: how fast is the same Codex Sol agent through subscription access, Fast mode, and the API?

Once that harness was reliable, we expanded it to a second question: how carefully do different coding agents follow the same repository instructions? We compared three intended tiers over 30 runs: GPT-5.6 Sol against Claude Opus 5, Terra against Sonnet 5, and Luna against Haiku 4.5. Each condition ran five times.

The access test used Subscription Standard, Subscription Fast requested, API Standard, and API Fast requested: 20 runs, five of which are the Sol subscription trials also used in the agent comparison. The two experiments therefore contain 45 unique runs.

The intelligence test

Each agent received a small repository with an inconsistent notification-sound specification. The runtime was already correct; the MCP schema and documentation were not. A successful agent had to discover that distinction rather than “fix” working code.

The exact repair was a triangle-wave G4 → G4 → E5 motif with 75ms, 75ms, and 140ms notes, configured 50ms gaps, and gain 0.8. The agent also had to correct two timing descriptions, explain the browser/native envelopes, and leave runtime byte-identical.

The ten-point rubric covered MCP terminology, timing corrections, exact motif values, untouched runtime, accurate reporting, and minimal scope. It tested evidence gathering, precision, restraint, and truthful reporting: focused agentic coding intelligence, not general intelligence.

Every run started from byte-identical input at high reasoning effort and concurrency one. Exact model IDs were required, fallbacks and multi-agent execution were disabled, and all 45 jobs completed with valid scores and parsed usage.

Part 1: Codex vs Claude

The bars show mean quality and each dot shows one run, exposing the variance hidden by averages.

Sol averaged 9.0 against Opus at 7.0 and won all five trials. Terra averaged 9.8 against Sonnet at 6.2, winning four while Sonnet won one. Luna averaged 7.2 against Haiku at 4.0, with three wins, one tie, and one loss. Across the whole matched set, Codex produced seven perfect results; Claude produced one.

Codex was not always slower or faster. Opus beat Sol by nine seconds and Haiku beat Luna by 32, but both sacrificed accuracy. Terra beat Sonnet on both quality and latency: 9.8 versus 6.2 and 77.6 seconds versus 89.1.

Claude reported more accounted tokens in every pair: 37% more for Opus, 99% for Sonnet, and 136% for Haiku. Provider accounting is not a universal unit of effort, but Claude’s lower scores did not come from smaller recorded sessions.

Why Codex won this task

The error pattern is more revealing than the aggregate score. Across 15 runs per agent family, Codex accumulated 12 failed rubric criteria; Claude accumulated 36.

Both agents struggled most with exact motif values, but Claude missed them 13 times against Codex’s six. Claude also missed MCP terminology in 12 runs; Codex did so twice. Only Claude failed the paired timing requirement, modified runtime, or exceeded scope.

Codex was better at preserving a chain of small constraints while editing only permitted files. Claude often produced plausible prose but dropped contractual details.

The tier-by-tier reading

Frontier: Sol was safer for correctness. Opus was slightly faster, but never reached 10/10 and lost every trial.

Balanced: Terra was the standout: nearly perfect, faster than Sonnet, and using roughly half the recorded tokens. Here, the frontier tier did not beat the well-matched balanced model.

Economy: Luna beat Haiku on quality, but ranged from 5 to 10. Haiku was fast yet too inaccurate for unattended repair. Both require stronger verification.

Part 2: Codex subscription vs API

We then held the agent, model, task, rubric, and reasoning effort constant. Only authentication channel and requested service tier changed.

Subscription Standard averaged 87.6 seconds; Subscription Fast took 26.9% less at 64.0. API Standard averaged 49.7 seconds. API Fast averaged 25.5: 48.6% less than API Standard and 71% less than Subscription Standard, with runs tightly grouped between 24.1 and 28.1 seconds.

Quality did not follow the same ordering. Subscription Standard averaged 9.0, Subscription Fast 9.4, API Standard 9.6, and API Fast 9.0. With five stochastic runs and overlapping scores, this does not prove that API access improves intelligence or that Fast harms it. It shows that the channel changed observed latency while quality remained noisy.

Fast has an explicit trade-off: official short-context Sol per-token rates are twice Standard. The useful question is whether the latency reduction justifies that rate; the experiment’s total bill says nothing about agent quality.

One caveat remains. We requested service_tier="fast", but Codex CLI 0.147.0 did not expose the response field needed to verify the effective tier of every call. We can prove the request and the measured latency, not the provider’s hidden routing decision.

The benchmark failed before the agents did

Our first channel comparison was invalid. A parent environment exported an API-specific CODEX_HOME, causing jobs labelled “subscription” to inherit API authentication. The outputs looked credible; the labels were wrong. We discarded them, isolated both profiles, added authentication preflights, and reran the matrix.

An earlier three-run Subscription Fast sample also scored 10/10 every time while containing a 179-second latency outlier. The five-run replacement ranged from 8 to 10. One-shot and tiny-sample benchmarks can manufacture certainty from ordinary variance.

What we take away

The test started with Codex access speed. API was faster than subscription in these runs, and Fast reduced waiting time again. That is a delivery result.

The model comparison came later and was deliberately not a test of raw coding ability. Frontier agents are increasingly close at producing code. This fixture tested following dense instructions, noticing contradictions, finding the authoritative source, preserving exact values, and not changing correct code.

On that axis, Codex was decisively better: 12 wins to two, seven perfect results to one, and one third as many rubric misses. This confirms what our manual comparisons suggested: Codex reads repository instructions more carefully and deviates less.

Codex also scored higher with fewer accounted tokens in every tier, making it more economical per correct outcome. This is not a dollar-for-dollar claim because subscription calls have no separable invoice.

We cannot conclude that Codex is universally better at coding. We can conclude that, on August 11, 2026, it followed instructions more attentively, preserved constraints more reliably, and reached correct outcomes more efficiently on this task.

The date matters. We have seen these roles reverse after training runs and model updates. This is a snapshot, not a permanent hierarchy; the test should be rerun whenever either agent changes.

For now, the conclusion is simple: Codex follows instructions better than Claude. It notices inconsistencies, preserves constraints, and reaches the intended result with less waste.

That may change with the next model update. When it does, we will run the test again.

A final reality check: Codex performed exceptionally well in this controlled test. It was also a poor collaborator during the review of this article: it repeatedly failed to follow editorial instructions and rewrote material it had not been asked to touch. That contradiction is the point. AI performance is context-dependent, and even a strong benchmark result should be treated with caution rather than as a universal verdict.