How Enterprise AI Agents Can Be Trained With AutoSynthData

🔍 Read the full analysis: How Enterprise AI Agents Can Be Trained With AutoSynthData on ThorstenMeyerAI.com

Prime Big Deal Days · Oct 6–7Offer from Amazon

Get monitors, keyboards and dev gear delivered free — and shop member deals

• Fast, free delivery on millions of items
• Access to Prime Big Deal Days deals on October 6–7
• Prime Video, Amazon Music and more included

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

ServiceNow CoreAI says it has built AutoSynthData, a system that uses an enterprise agent’s failures and a stronger model’s successful runs to create new training tasks for that environment, as detailed in the original analysis. The company illustrates the approach with the released EnterpriseOps Gym dataset, but the supplied account reports no measured improvement, comparison with other training methods or details on how many generated tasks were used.

AutoSynthData begins by testing a target model on diagnostic tasks in an agentic environment. A more capable teacher model attempts the same tasks. ServiceNow CoreAI says those runs help identify the capability being tested, the tools and workflow involved, where the target model fails, how the teacher succeeds, and what conditions a valid result must meet.

The system condenses that information into sanitized capability specification cards. According to the description, generators receive these cards rather than the original evaluation prompts, entities, action trajectories or verifier details. They then create tasks with varied wording, starting states, entities, tool combinations and difficulty. Each task includes an environment specification, a user prompt and a verifier intended to check whether the agent completed the request within the environment’s constraints.

Generated tasks are checked in the environment before accepted examples are used for post-training. The updated model can then be evaluated again, with remaining weaknesses informing another round of task generation. ServiceNow CoreAI says the tasks are meant to be feasible, realistic and challenging for the current model. The account does not state how many tasks were generated or accepted, or whether the process improved performance.

Training for Enterprise Workflows

Enterprise agents must do more than produce plausible text. They may have to update records, follow access rules and use tools in an appropriate sequence, while leaving a system in a required state. A model that performs well on broad evaluations can still mishandle local policies or workflows. AutoSynthData is designed to focus training on those environment-specific gaps.

The approach could give organizations a way to generate varied practice tasks from observed weaknesses rather than relying solely on manually written examples. That matters when agents can change operational data: mistakes may affect records or workflows, not just the quality of an answer. But this potential is not evidence of an outcome. The supplied material does not show that AutoSynthData improves reliability, lowers costs or transfers successfully to other enterprise systems.

The verifier is a key dependency. If it approves an incorrect result, training could reward unwanted behavior; if it rejects a valid solution, it could penalize sound behavior. ServiceNow’s account says verifiers should reject failures and policy violations while accepting valid solutions without requiring one exact sequence of actions. Whether that balance works in practice needs to be shown through evaluation results.

How AutoSynthData Builds Tasks

In the framework described by ServiceNow CoreAI, an agentic environment defines what an agent can observe and change, which tools and APIs are available, and how its actions affect the system. A task can also include instructions, policies and setup information, such as a seeded database or knowledge articles.

A task’s plausibility is distinct from whether it is technically executable. A request may be possible but unrealistic; another may sound realistic but be impossible because a tool is unavailable, information cannot be accessed, a state change cannot be made or policy forbids it. The described generator is intended to avoid such cases while producing tasks that test weaknesses in the target model.

For its example, ServiceNow CoreAI points to EnterpriseOps Gym and cites Malay et al. (2026), saying it uses the released dataset. The supplied material does not give the dataset’s size, name the specific workflows tested or report model scores. It also does not provide a publication date for the AutoSynthData account.

““A model may be broadly capable and still struggle with a particular environment.””

— ServiceNow CoreAI

Performance Evidence Still Missing

The account describes a training method, but does not provide before-and-after performance results, explain how success was measured or compare AutoSynthData with a suitable baseline. It also omits the target and teacher models, training volume, task-generation and verifier-acceptance rates, and the time or cost of running the pipeline.

It remains unclear whether generated tasks generalize beyond EnterpriseOps Gym or whether the approach works across different enterprise systems. The description says generators receive capability cards rather than original evaluation-task details, but provides no analysis of overlap between generated tasks and evaluation material. Without those details, readers cannot independently assess the reported example or judge how strong the evidence is.

Results Needed to Judge the Method

The next useful evidence would be a reported evaluation of the post-trained model, including task counts, verifier acceptance rates and before-and-after scores. A clear comparison with an appropriate baseline would help establish whether the method adds value beyond other ways of producing training data.

Results across multiple workflows and environments would help show whether gains persist outside the example dataset. Until such information is reported, AutoSynthData is best understood as a described pipeline, not a demonstrated performance improvement. The supplied material does not identify a date for further results or another evaluation milestone.

Key Questions

What is AutoSynthData?

AutoSynthData is a system described by ServiceNow CoreAI for generating and validating enterprise-agent training tasks based on a target model’s observed failures and a stronger teacher model’s successful runs.

How does it create new training tasks?

The system turns findings from model evaluations into sanitized capability cards. A generator uses those cards to create varied prompts and environment setups, and a verifier checks whether each task is feasible and completed within the environment’s rules.

Has ServiceNow shown that AutoSynthData improves agent performance?

No measured improvement is reported in the supplied account. It provides no before-and-after scores, comparison baseline, task totals or verifier acceptance rates.

What is EnterpriseOps Gym’s role?

ServiceNow CoreAI cites the released EnterpriseOps Gym dataset as an example of the approach. The supplied material does not specify its size, the workflows tested or the models’ results on it.

What evidence would help evaluate the approach?

Useful evidence would include before-and-after results, comparison with a baseline, task and acceptance counts, and evaluations across multiple workflows and enterprise environments.

Primary source: Hugging Face · via ThorstenMeyerAI.com