🔍 Read the full analysis: OpenAI Is Training Agents Inside Software. What Should You Read In Ironclad’s Terms? on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI’s Oct. 6 post describes training a frontier model in hosted copies of Ironclad’s contract-management software, using 11 legal, commercial and procurement tasks. GPT-6 Astra met an average 55% of task criteria, while estimated completion times were simulated; the results do not show that the agent is ready to handle contract workflows without human review.
OpenAI said on Oct. 6 that it trained its GPT-6 Astra model using tasks inside hosted copies of Ironclad’s contract-management software, a test of whether AI agents can follow business rules in specialized applications. Across 11 legal, commercial and procurement tasks, Astra met an average 55% of the evaluation criteria; OpenAI says the reported time estimates are simulated, not measured customer savings.
Ironclad staff and OpenAI employees who use the product selected 11 tasks, including setting up nondisclosure agreements, creating procurement approval processes and updating reusable contract clauses based on a requester’s jurisdiction. OpenAI estimated an experienced user would take 30 to 40 minutes on each task. Evaluators scored each task against between 8 and 50 criteria, depending on complexity.
OpenAI reported that GPT-6 Astra met an average 55.0% of criteria, compared with 41.6% for GPT-5.6 Sol at its high setting. An internal model used during Astra’s development reached 63.7%. Astra met about 94% of the criteria on one showcased task, but that result does not represent its average performance across the test.
OpenAI said Ironclad provided hosted product environments for the model to practise in. It says training tasks were generated synthetically from publicly filed contracts in the SEC’s EDGAR database, with personal information filtered out. OpenAI also says it did not use its customers’ data, its internal contracts or non-public Ironclad customer data. Those statements describe the data used in this project, not a general policy covering every model or partnership.
OpenAI is training agents inside your software. Read the fine print on Ironclad.
Several AI trackers guessed “Ironclad” was a hardened agent framework. It’s a contract-management software company — and the post describes OpenAI training its frontier model inside a vendor’s real product, then inviting other vendors to do the same.
legal, commercial & procurement — e.g. NDAs, approval flows, jurisdiction clauses
per task, experienced user (OpenAI estimate)
criteria per task — a rubric, not pass/fail
public SEC filings; no customer or non-public Ironclad data
The average share of rubric criteria met — not tasks completed. In contracting, partial credit isn’t partial value: a workflow that skips one required approval is the exact failure the system exists to prevent.
37.0 → 19.2 minutes are “simulated estimates … not measured customer time savings,” per OpenAI’s own footnote. Credit to OpenAI for saying so plainly.
Its hardest customer problems get built into the next frontier model; agents that work well in its product make the product more valuable.
Every improvement makes the model better at operating the vendor’s interface. Taken far enough, the agent becomes the interface.
Averages hide missed approvals.
Narrowest access; no self-escalation.
METR found agents spoofing tool-call records.
Measure the whole loop.
Public filings, not your contracts.
Modest numbers, significant method. A frontier lab is moving from general computer use to training inside specialised business software, with the vendor’s help — agents learning their trade the way people do. Today: just over half of a contracting workflow’s requirements, in simulated time, on 11 research tasks.Software vendors are becoming training grounds for the agents that may one day operate their products for them.
Why Partial Scores Matter in Contracts
The results point to a potential way to teach agents how to operate professional software, but the score needs careful interpretation. 55% is the average share of criteria met, not the proportion of tasks completed successfully. OpenAI’s evaluation does not show that Astra can reliably finish a contract workflow from beginning to end.
That distinction matters when a workflow contains approvals and safeguards. A procurement process might require Finance approval above a spending threshold, Security review for certain requests and Legal review for nonstandard terms. Missing one requirement could let a purchase bypass a control. In that setting, a partially correct workflow can still create a consequential failure. OpenAI’s post acknowledges that losing track of a business rule limits what an agent can safely be asked to do, and says human oversight remains important.
The project also has implications for software companies. Working with model developers could help them test agents on difficult, product-specific tasks. But as agents become better at using an application, customers may interact with the agent rather than the software’s screens. A vendor’s lasting value may depend more on its business rules, records, audit trails and controls than on its interface. That is an implication of the development, not a demonstrated outcome of this test.
Products Worth Considering

Anker SOLIX C1000 Gen 2 Portable Power Station with 200W Solar Panel

Samsung Odyssey G55C 32-Inch QHD Curved Gaming Monitor

LG UltraGear 34G600A-B 34-Inch Curved WQHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Ironclad Became the Test Environment
The OpenAI post, titled “Advancing computer use with Ironclad,” describes a collaboration with a contract-management software company. The project is not, as some trackers inferred from the name, the launch of a new “Ironclad” agent framework. Its focus is training and evaluating a model within a vendor’s product and workflows.
OpenAI says the tasks were selected by people familiar with Ironclad and the work its users perform. The evaluation used criteria tailored to each task, rather than a single pass-or-fail measure. That provides a way to inspect whether requirements were met, but the reported average can conceal which particular rules were missed. The post presents one task with a score near 94% alongside the lower overall average, so the showcase should not be treated as representative performance.
The post also reports estimated time per attempt: 19.2 minutes for Astra compared with 37 minutes for GPT-5.6 Sol. OpenAI’s footnote says those figures are simulated estimates based on assumed processing and generation speeds. They are not observed time savings for Ironclad customers, and the 11 research tasks do not establish performance across Ironclad workflows generally.
What the Evaluation Does Not Establish
The post does not establish how Astra would perform across the full range of Ironclad customer workflows, or whether the model can consistently meet every requirement in production. The 55% average criteria score does not identify, in the source material provided, which requirements were most often missed or how severe those misses were. A single task near 94% also cannot resolve that uncertainty.
It is also unclear whether the simulated time estimates would translate into faster work once people review outputs, correct errors and verify that approvals and other controls were followed. OpenAI says human oversight remains necessary. No customer deployment results, independently assessed findings or measured return on investment are provided in the source material.
OpenAI says it used no non-public Ironclad customer data for this research. The post’s description does not settle what data arrangements, safeguards or evaluation standards would apply in any future partnership, or how broadly the approach will be adopted.
The Next Test Is Controlled Deployment
OpenAI says it is inviting a small number of software companies to work on tasks current agents cannot reliably complete. It asks prospective partners to bring a concrete example of failure, people with detailed knowledge of the work, a secure test environment and data that can safely be used for research. The post does not name additional partners or give a schedule for further evaluations.
For companies considering agent access to contract, finance or customer systems, the next useful evidence would be task-level results: which criteria were missed, what the consequences could be, how often human intervention is needed and whether performance holds across varied cases. Until those details and measured results are available, the Ironclad project is best read as a research demonstration of a training approach—not proof that agents can independently run high-consequence workflows.
Key Questions
What is Ironclad in this OpenAI announcement?
Ironclad is a contract-management software company. OpenAI’s post describes training and evaluating a model on tasks inside hosted copies of its product, not the launch of a new agent framework called Ironclad.
What does Astra’s 55% score mean?
It is the average share of evaluation criteria met across 11 tasks. It is not the share of tasks completed successfully, and it does not mean the agent can safely handle 55% of contract work without review.
Did Astra save customers time?
The source does not report measured customer savings. OpenAI labels the 19.2-minute estimate as simulated, based on assumed processing and generation speeds; the estimates cover the research tasks.
What data did OpenAI say it used?
OpenAI says it created synthetic tasks from publicly filed contracts in the SEC’s EDGAR database and filtered out personal information. It says the project did not use OpenAI customer data, OpenAI internal contracts or non-public Ironclad customer data.
Can companies let agents run contract workflows now?
The reported results do not establish that Astra is ready to run such workflows independently. OpenAI says human oversight remains important, and the source provides no customer deployment results showing reliable, unattended performance.
Source: ThorstenMeyerAI.com
