AutoSynthData: Generating Training Data for Enterprise Agents

ServiceNow CoreAI has introduced AutoSynthData, a framework designed to generate training data for enterprise AI agents that struggle with specific environments, tools, or rules. While broadly capable models often falter on particular enterprise workflows, converting individual failures into robust training data has historically been difficult. AutoSynthData addresses this by using a target model's failures and a stronger teacher model's successes to determine what the model needs to learn next. The pipeline generates and validates new tasks that exercise identified capability gaps. As the target model improves, the curriculum dynamically shifts toward remaining weaknesses. The system defines useful training tasks based on three core properties: feasibility within the environment, realism resembling actual user requests, and appropriate difficulty that targets unresolved capabilities. Furthermore, verifiers used in the pipeline must satisfy consistency, soundness, and completeness to ensure accurate evaluation of trajectories. AutoSynthData operates in two main phases. The target phase creates core training samples from sanitized capability specification cards, subjecting each candidate to validation, execution, and solver evaluation. The multiply phase then expands the dataset by generating novel variants of accepted target samples while preventing unbounded drift. By separating generation control from environment-specific execution, AutoSynthData provides a structured, scalable approach to producing training-scale datasets grounded in specific operational environments.