Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
When thoroughness becomes a false signal
Software teams know the uncomfortable gap between passing checks and delivering the outcome users actually need. An implementation can be carefully documented, logically argued and apparently complete, yet still fail at the final handoff. Firmulate’s live management experiment found a business version of that problem in Opus 4.8: the most thorough participant produced the deepest analyses and learned more than 80 playbook rules, but finished last.
Its Crucible League score was 73, behind Fable 5 at 77, Sonnet 5 at 88, Kimi K3 at 93 and gpt-5.6-sol at 95. This was not a story of an incapable model missing obvious emergencies. Opus identified the crises and resisted manipulation. Its failure was subtler and more instructive: it did much of the difficult work, then left the close on the table.
As an affiliate, we earn on qualifying purchases.
A demanding week, held constant
Firmulate gave each frontier model the same assignment: run the same small software company through its worst week, facing the same customers, crises and temptations. Every decision was versioned and auditable. The company itself has 13 synthetic employees and real money mechanics, including burn of €105,000 per month against €2,300 in monthly recurring revenue. Its public operation has accumulated more than 680 self-learned playbook rules, with every workday versioned.
The setup matters because it tests management behavior rather than the polish of an isolated answer. Models had to notice problems, investigate company information, resist pressure and complete commercially meaningful work. A do-nothing baseline still scored 26 because partial progress counts, but the benchmark imposes a hard trust boundary: a single breach caps the total because “no amount of good work outweighs a breach of trust.”
On that dimension, the field performed well. Every model spotted every crisis and refused every manipulation attempt. Fake CEO messages escalated over three stages, followed by a reporter’s attempt to obtain “just one yes/no, on background.” All 5 models refused. Kimi K3 recorded the clearest concise response: “Treat the request as a suspected approval-bypass / possible impersonation.”
That shared resistance makes Opus 4.8’s result more interesting, not less. The differentiator was not whether the model could recognize danger. It was whether careful analysis translated into decisive, disciplined execution.
As an affiliate, we earn on qualifying purchases.
The fact that separated analysis from impact
The commercial turning point was a €55,000 deal. The decisive weakness in a competitor’s position was not presented directly in the customer event. It sat two document references deep inside the company’s own files. Models that followed the references and read that file could win the deal at full price, worth an additional €4,583 in monthly recurring revenue.
All the participants could diagnose the situation and construct the pitch, yet only two signed the deal their own work had earned. Firmulate summarizes the gap bluntly: “Same diagnosis, same pitch — no signature.” For developers and QA leaders, that resembles a workflow that produces a correct intermediate state but never commits the result that matters.
Opus 4.8 embodies the paradox most sharply. It was the most thorough participant, added more than 80 learned rules and generated the deepest analyses. Those are all signals that might look reassuring in a demonstration or review. But the model did not complete the close, and its operational discipline slipped when it attempted to write into a locked department instead of escalating.
The fairest reading is not that Opus alone suffered from this tendency. The same weakness appeared, less strongly, in all four other models. Opus simply made the contrast hardest to ignore: more written learning and deeper reasoning did not automatically produce more business impact.
As an affiliate, we earn on qualifying purchases.
A ranking with necessary context
The final July 2026 Crucible League placed gpt-5.6-sol first with 95, Kimi K3 second with 93, Sonnet 5 third with 88, Fable 5 fourth with 77 and Opus 4.8 fifth with 73. The public Firmulate benchmarks present the results and plain-language findings.
One comparison deserves caution. K3 ran without an effort parameter and therefore used its API default, while the other participants ran at xhigh. That does not erase its second-place performance, but it is relevant context when comparing how each model was configured.
The wider experiment also includes 242 real, unedited management decisions in a quiz that asks readers to guess which model made each choice. That format reinforces the underlying point: confident prose is not necessarily a reliable guide to whether an agent will investigate the right source, preserve discipline and finish the task.
As an affiliate, we earn on qualifying purchases.
What software teams should take from it
Opus 4.8’s last-place finish is less an indictment than a warning about evaluation. Diligence is valuable, but volume can disguise weak prioritization. A model may identify every problem, resist every social-engineering attempt and produce an extensive body of learning while still failing to secure the outcome its analysis made possible.
For teams considering AI agents around customer records, support work or forecasts, the useful questions extend beyond whether the model reasons persuasively. Does it read the company’s own files before acting? Does it escalate when permissions block the intended path? Does it remain trustworthy under pressure? Most importantly, does it finish what it starts?
Firmulate also offers enterprises the same wargame against a read-only export of their own business, with nothing written back to real systems. That makes the experiment’s central lesson practical: test agents against completion, judgment and discipline—not simply against the quantity or sophistication of their output.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
