🔍 Read the full analysis: FrontierCode: Would A Maintainer Merge AI Code Into Their Project? on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis evaluated 23 models on FrontierCode, a coding benchmark that tests whether a change meets maintainer-style review criteria, not just whether it passes tests. Claude Opus 5.5 led with a 52.2% score and cleared all blocking checks in 57.7% of attempts; the results do not establish how models perform across all real-world pull requests.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that asks whether a proposed repository change meets maintainer-style review criteria. Claude Opus 5.5 scored highest at 52.2%, but it cleared every blocking check in only 57.7% of attempts, indicating that even the leading result does not make review unnecessary.
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, running the models through the same open-source agent harness, Stirrup. Each attempt was capped at 250 turns, with three attempts per task. The benchmark’s Main set contains 100 tasks drawn from repositories including Celery, Budibase, uppy and Mattermost.
Claude Opus 5.5 (max with fallback) ranked first with a 52.2% FrontierCode score and a 57.7% pass rate on blocking checks. GPT-6.1 Sol (max) scored 49.1%, followed closely by Claude Fable 5.1 (max with fallback) at 49.0%. Xiaomi’s MiMo-V2.6-Pro led the open-weight models listed, scoring 45.4%.
FrontierCode does not award credit just because ordinary tests pass. It evaluates hidden tests, command checks, tests that demonstrate they catch the original bug, implementation-specific tests, limits on file changes and diff size, and an LLM code review against maintainers’ criteria. Failing any blocking check results in a zero for that attempt, according to the benchmark description. Artificial Analysis used GPT-6 Luna (medium) for the review step.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
A Passing Test Is Not a Merge
The results address a gap between writing code that appears to work and producing a change that a project maintainer would accept. A patch can pass visible tests yet still introduce regressions, change unrelated files, or include tests that do not actually detect the problem. By scoring those issues, FrontierCode puts reviewability and scope alongside functional correctness.
For teams considering AI coding agents, the benchmark suggests that model output still needs human review and repository-specific checks. The top model cleared every blocking check in 57.7% of attempts in this evaluation, not all of them. That figure describes performance on FrontierCode’s tasks and setup; it is not a measured rate for production pull requests or every kind of software project.
Cost also changes the practical picture. Artificial Analysis reported per-task costs ranging from $0.09 for GPT-6 Luna to $10.15 for the leading Opus configuration, with GPT-6.1 Sol at $0.81. Opus scored 3.1 percentage points above Sol, while costing about 12.5 times as much per task in the reported comparison. Those figures make the accuracy-cost trade-off visible, but do not account for every deployment expense or the value of a particular task.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Scores Changes
FrontierCode was built by Cognition and open-source maintainers and announced in June 2026. Version 1.1 followed in July with rules against agents looking up solutions online. The full benchmark contains 150 tasks; Artificial Analysis evaluated the 100-task Main set. The source report says more than 20 maintainers developed tasks from 36 repositories and spent more than 40 hours on each task.
The tasks are private, according to the benchmark account, to reduce the chance that models encountered their solutions during training. The grading also penalizes behaviors that can make a proposed change harder to review, including oversized diffs and edits beyond the relevant files. Attempts flagged by Cognition’s trajectory scanner for looking up a solution receive zero, the source report says.
The rankings do not simply mirror general intelligence scores. In the comparison provided by the source, Claude Sonnet 5.5 moved from second among the listed models on the Intelligence Index to fifth on FrontierCode; Grok 4.7 ranked 18th on FrontierCode. The benchmark’s results therefore offer a distinct measure of coding-agent performance, rather than a general ranking of model capability.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The results cover 23 models on a 100-task subset, using one agent harness, a stated turn limit and three attempts per task. They do not establish how the models would perform on every repository, programming language, team’s review standards or production workload. The report also does not provide enough information here to independently assess how sensitive the ranking is to the selected judge model or to changes in task selection.
Several models had no FrontierCode result as of October 13, according to the source: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published ranking is not a comparison of every model that might be relevant to coding. The benchmark’s score and pass rate are separate measures, and neither should be read as a guarantee that a given patch will be accepted by a real project maintainer.
As an affiliate, we earn on qualifying purchases.
More Models and Evaluations Ahead
The next useful update would be results for the models absent from the October 13 comparison, alongside further testing on FrontierCode or other repository-level evaluations. The source identifies several missing systems but gives no release dates or confirmed plans for their results.
Readers assessing coding agents can compare both the benchmark score and blocking-check pass rate, then weigh those results against task costs and their own review requirements. Until broader results are available, the published scores are best treated as evidence from a specific benchmark setup, not as proof that AI-generated changes can be merged without human oversight.
machine learning code analysis tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark built by Cognition with open-source maintainers. It assesses AI-generated repository changes using checks for correctness, regressions, test quality, change scope and code review criteria.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s October 2026 results, with a score of 52.2%. It cleared all blocking checks in 57.7% of attempts.
Does the leading score mean AI code is ready to merge automatically?
No. The result reports performance on FrontierCode’s 100-task Main set, not a general production acceptance rate. The source report says even the leading model failed to clear all blocking checks in a substantial share of attempts; human review remains relevant.
How much did the evaluated models cost per task?
The source reports costs from $0.09 to $12 per task across the evaluated models. Examples on its cost frontier include GPT-6 Luna (max) at $0.09, MiMo-V2.6-Pro at $0.18, GPT-6.1 Sol (max) at $0.81 and Claude Opus 5.5 (max with fallback) at $10.15. These are benchmark-reported task costs, not a full accounting of deployment expenses.
Were all major coding models included?
No. As of October 13, the source report listed Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no FrontierCode result. It did not give a timetable for their evaluation.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
