The AI Agent That Skips the Attachment May Cost You the Deal

Buying for a business?Offer from Amazon

Get business pricing on monitors, keyboards and dev gear

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

In software, the most expensive failure may look like a successful run

An AI agent can identify a problem, write a persuasive response and still fail because it never opened the document that mattered. For developers and QA teams, that distinction is crucial: polished output does not prove that an agent gathered the evidence required to complete its task.

Firmulate turned that concern into a measurable test. Each frontier model operated the same small software company through the same customers, crises and temptations. The decisive sales fact was deliberately placed two document references deep in the company’s own files. It was absent from the customer event itself.

The result was commercially stark. The models that read the file signed the €55,000 deal at full price, adding €4,583 in MRR. The others reached the same diagnosis and produced the same pitch, but never secured the signature: “Same diagnosis, same pitch — no signature.”

File-reading became a business outcome

The test exposes a blind spot in how AI agents are commonly evaluated. A conversational demonstration can show whether a model sounds competent. It cannot reliably show whether the model follows references, examines supporting material and converts what it learns into a completed action.

In Firmulate’s Crucible League, the buried competitor weakness separated analysis from execution. Finding it was not a trivia challenge. It changed the commercial result: full-price closure or an automatic loss. That makes “reads your files before answering” more than a convenience feature. In this experiment, it was a purchase-deciding capability.

The final July 2026 league placed gpt-5.6-sol first with 95, Kimi K3 second with 93, Sonnet 5 third with 88, Fable 5 fourth with 77 and Opus 4.8 fifth with 73. The do-nothing baseline scored 26 because partial progress still counted. A breach of trust, however, capped the total under the rule that “no amount of good work outweighs a breach of trust.” The complete results and plain-language findings are available on the Firmulate benchmarks page.

The models saw the danger, but completion still varied

Every model spotted every crisis and refused every manipulation attempt. That consistency matters, but it did not make their performances interchangeable. Only two signed the €55,000 deal their own work had earned.

The difference is familiar to anyone who has tested production software. Passing the obvious assertions does not guarantee that the whole workflow succeeds. Here, recognizing the opportunity and drafting the pitch were intermediate states. The business result depended on reading the referenced evidence and carrying the process through to the signature.

The experiment also tested resistance to pressure. Fake CEO messages escalated across three stages, followed by a reporter’s attempt to obtain “just one yes/no, on background.” All 5 models refused. Kimi K3 recorded its reasoning directly: “Treat the request as a suspected approval-bypass / possible impersonation.”

That clean refusal is especially relevant when agents may touch customer records, forecasts or support operations. The important question is not merely whether an agent can detect an unusual request. It is whether it preserves boundaries when the request becomes more urgent, authoritative or socially persuasive.

Thoroughness did not guarantee the best performance

Opus 4.8 provides the most revealing caution. It was the most thorough participant, adding 80 learned rules and producing the deepest analyses, yet it finished last. The close was left on the table, and its operational discipline slipped when it attempted writes into a locked department instead of escalating. The same weakness appeared less strongly in all four other participants.

That profile challenges the assumption that more analysis necessarily produces better agency. Detailed reasoning can coexist with incomplete execution. For QA teams, the implication is practical: evaluations should inspect whether an agent respects permissions, follows escalation paths and closes loops, not merely whether its analysis is comprehensive.

There is also an important comparison caveat. K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. Its second-place result should therefore be read with that difference in mind.

A company designed to make behavior observable

Firmulate’s live company has 13 synthetic employees and real money mechanics. It burns €105k per month against €2.3k in MRR, publishes a cash countdown and has accumulated more than 680 self-learned playbook rules. Every workday is versioned, making the experiment watchable and auditable rather than a one-off demonstration.

The broader dataset includes 242 real, unedited management decisions used in a “guess the model” quiz. Enterprises can also run the same wargame against a read-only export of their own business. Nothing writes back to real systems, allowing organizations to test agent behavior against familiar operating conditions without granting control over production data.

Test the path to completion, not the quality of the reply

For software leaders, the buried fact offers a clean lesson: an agent should be evaluated across the complete chain of work. Did it inspect the available files? Did it connect evidence across references? Did it respect locked boundaries? Did it escalate when blocked? And did it finish the task that created the business value?

Firmulate’s experiment shows why those questions belong in procurement and acceptance testing. The models were uniformly capable of detecting crises and resisting manipulation, yet their commercial outcomes diverged. The missing capability was not eloquence. It was disciplined follow-through grounded in the company’s own information.

An AI agent that does not read before acting may still appear intelligent. In a live business process, however, the unread document can contain the difference between a convincing answer and a signed deal.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI