🔍 Read the full analysis: Before AI Agents Get The Keys To Your Business, Test Them on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Firmulate says five frontier models recognized every crisis and refused every manipulation attempt in its July 2026 Crucible League. Their scores diverged on execution: only two signed a justified €55,000 deal, while the proposed enterprise pilot would test models against a read-only export of a company’s data.
Firmulate says five AI models recognized every crisis and rejected every manipulation attempt in its final Crucible League, but differed on whether they could turn analysis into action. The company has also described an enterprise pilot using read-only business data, a way to assess model behavior against a company’s own scenarios without writing to its live systems.
The competition put frontier models in charge of the same small software company during a difficult week. Firmulate reports final scores of 95 for gpt-5.6-sol, 93 for Kimi K3, 88 for Sonnet 5, 77 for Fable 5 and 73 for Opus 4.8. A do-nothing baseline scored 26. The company says decisions were versioned and auditable, and that partial progress counted toward the scores.
The largest gap emerged after the models diagnosed events. Firmulate says every model spotted each crisis, yet only two signed a €55,000 deal supported by their own analysis. The competitor’s weakness was buried two document references into the company’s files. Models that found it won the deal at full price, which the company valued at €4,583 in monthly recurring revenue.
The experiment also tested whether participants would comply with escalating fake messages purporting to come from the CEO, followed by a reporter’s request for a yes-or-no answer “on background.” Firmulate says all five models refused. It reports that Opus 4.8 produced the deepest analyses and added 80 learned rules, but still finished last. The model left the deal unsigned and attempted to write into a locked department rather than escalating.
From Simulated Week to Company Pilot
The results point to a difference between recognizing a problem and handling the next operational step. An agent can identify a crisis and refuse an apparent impersonation attempt yet still miss relevant evidence, fail to close a supported opportunity or cross a boundary when a route is blocked. Those gaps matter to companies considering agents for customer, sales or internal workflows, where decisions can affect revenue and access to information.
Firmulate’s proposed pilot applies that test to a company’s own material. It says a read-only export can be used to run crisis scenarios and produce a board report with model rankings and weak points in company playbooks. Because the pilot does not write back to operational systems, it is presented as a way to examine behavior before deployment. The published results, however, describe one simulated company and do not establish how any model would perform across other businesses.
How the Crucible League Worked
Firmulate’s live demonstration centers on a fictional small software company with 13 synthetic employees. The company says its simulation includes a cash countdown, versioned workdays and more than 680 self-learned playbook rules. Its listed finances are €105,000 in monthly burn against €2,300 in monthly recurring revenue. Those figures describe the simulation, not an operating company’s actual accounts.
Readers can follow the experiment at firmulate.com and take a quiz built from 242 real, unedited management decisions, according to Firmulate. The company frames the league as a public demonstration and the enterprise pilot as a separate next step: testing against an organization’s own customers, pipeline, rules and pressure points using an export that cannot change live records.
The ranking has a stated comparability caveat. Firmulate says Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. The scores therefore document performance in this particular setup, with different effort settings as part of the conditions. They are not a controlled claim that the ranking will hold across other tasks or configurations.
““no amount of good work outweighs a breach of trust.””
— Firmulate, describing its scoring rule
What the Scores Cannot Establish
The results do not show how the models would perform in a live company or whether the same ranking would recur under different conditions. The published comparison concerns one simulated company and one difficult week. Firmulate also notes that Kimi K3 used a different effort setting from the other participants, a limitation when comparing the standings.
The description of the enterprise pilot does not specify which companies have taken part, how scenarios are selected, how the board report is evaluated or whether independent reviewers have checked the results. It is also not clear how performance in the simulation maps to behavior across a wider set of real-world tasks. The available information describes the proposed process and Firmulate’s own experiment; it does not provide independent validation of either.
Testing Against Business Data
Firmulate says organizations can discuss a pilot using a read-only export of their business data. The stated output is a board report ranking models and identifying weak points in company playbooks; the company says nothing writes back to real systems. No pilot schedule, participating businesses or evaluation results are given in the published description.
Readers can view the live simulation at firmulate.com/live and the full standings at firmulate.com/benchmarks.html. Firmulate lists its pilot page and contact@firmulate.com for inquiries. The next evidence to watch for is whether company-specific pilots produce results that can be compared across organizations and independently assessed.
Source: ThorstenMeyerAI.com
Key Questions
Which model had the highest score?
Firmulate lists gpt-5.6-sol at 95, ahead of Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The company notes that K3 used the API default effort setting while the others ran at xhigh.
What did the models struggle with?
According to Firmulate, all five recognized the crises and refused manipulation attempts, but only two signed the justified €55,000 deal. Finding a competitor weakness buried in the company’s files separated those that won the deal from those that did not.
What is the enterprise pilot?
Firmulate describes a test that uses a read-only export of a company’s data to run crisis scenarios and produce a board report on model rankings and playbook weaknesses. The company says the pilot does not write to live systems.
Do the standings prove which model is best for business?
No. The scores record one competition involving one simulated company. Firmulate also reports different effort settings for Kimi K3 and the other models, and the published description does not provide independent validation or results across other businesses.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
