Chapter 4. Generate synthetic data

The knowledge tuning pipeline example shows how to generate synthetic training data from enterprise documents, so that a fine-tuned model can accurately recall domain-specific content and facts in response to user queries.

Pre-trained language models typically encounter most facts only once or twice during training, leaving proprietary or domain-specific knowledge incomplete or absent. The knowledge tuning pipeline example addresses this problem by generating n augmentations per document, where n is a configurable parameter (NUMBER_OF_SUMMARIES), and converting each augmentation into synthetic question and answer (Q&A) pairs. Increasing summary token count leads to superior memorization.

This pipeline example implements the following augmentation instances:

For each summary, the system generates three Q&A pairs; it discards excess pairs during post-processing. This architecture is modular. You can integrate additional augmentation types by editing existing flows or by adding new ones.

The output is a high-quality JSONL data set that is ready for consumption through training. This example provides a complete walkthrough of data generation and preparation for training, from document parsing through data set combination.

For benchmark results, token scaling statistics, and custom domain evaluations, see the SDG Hub Knowledge Tuning example repo.

Multilingual knowledge tuning example

The knowledge generation pipeline also includes a multilingual knowledge tuning example that supports creating training data in any language. The example includes pre-built flows for Spanish that are auto-discovered with zero configuration. For other languages, the library automatically translates flows on demand by using an LLM, with a verification pass to ensure accuracy.

To get started, set SDG_LANG (for example, Spanish) and SDG_LANG_CODE (for example, es) in your environment, then run the knowledge generation notebook as normal. The notebook checks for a pre-translated flow variant and uses it if available. If no pre-translated flow exists, the notebook translates all prompt templates, verifies them with a second LLM pass, and registers the new flow for immediate use with no extra code.

Performance benchmarks on the QuALITY benchmark (translated to Spanish) show that multilingual fine-tuning improves domain-specific accuracy while preserving general model capabilities. SFT on Spanish synthetic data improved benchmark accuracy from a 44.47% baseline to 68.88% in RAG settings, while retaining 84-90% of the baseline model’s behavior. For configuration details, benchmark tables, and coverage analysis, see the multilingual documentation.

The text analysis example shows how to generate data for teaching models, to extract meaningful insights from text in structured format. Create custom blocks and extend existing flows for new applications.

Reliable, repeatable evaluation is essential before you ship RAG or agentic updates to production. The RAG Evaluation example provides a flow for generating data for evaluating RAG systems at scale. The input of the flow is user documents. The output data set is post-processed to work with the RAGAS framework. The flow creates Q&A pairs with ground truth context for evaluating RAG systems. This flow simplifies data generation for RAG workflows.

The Teaming & AI Safety example shows how to generate adversarial prompts for testing model safety, guardrails, and robustness against harmful content. It uses a challenger LLM as teacher. This example demonstrates how to create custom safety testing datasets based on your organization’s policies and concerns. The workflow generates harmful prompts with explanations. You can integrate it with security testing tools, such as Garak for automated safety validation on Red Hat OpenShift AI.

The agentic examples demonstrate training data generation for custom MCP servers and agent tool-calling.

SDG Hub enables the creation of training data for AI agents and tool-calling applications through the following two complementary capabilities: generating tool-use training data from real tool interactions (MCP Server Distillation), and evaluating fine-tuned models through the same agent stack used for data generation (MCP Server Evaluation).

Agent Connector Framework

The Agent Connector Framework provides a standardized way to incorporate external tools, agents, APIs, and services into SDG pipelines. Instead of writing custom integration code for each tool, the connector framework offers a pluggable architecture that handles tool-specific communication while SDG Hub orchestrates execution at scale.

The Agent Connector Framework has the following capabilities: * Pluggable connector architecture for any external tool or service * Async orchestration with automated execution at scale * Production-ready error handling and retry logic * Extraction of results from tool interactions to generate rich, behavior-driven training data * Positioning of SDG Hub as a universal orchestration layer for multi-tool workflows, enabling AI engineers to build extensible data generation pipelines that incorporate proprietary tools and third-party services * Existing integrations, such as Langflow and Langgraph

Additional resources

The Domain Customization Data Generation using Kubeflow Pipelines (KFP) example demonstrates how to generate synthetic data for domain-specific model customization by using Kubeflow Pipelines.

The Kubeflow Pipelines guided example uses a minimal test pipeline to run a simple three-block flow that generates a question from each input document.

You can replace this flow with your own to perform more complex data generation tasks, such as multi-step augmentation, knowledge Q&A generation, or domain-specific transformations.

The example pipeline uses llm_test_flow.yaml, a minimal three-block flow designed to demonstrate the SDG component end-to-end. It takes a JSONL data set with document and domain columns and generates a question for each document:

The prompt template uses a system message ("You are a helpful assistant that generates a short question about the given text") and a user message that inserts the {{document}} content.

When complete, the pipeline produces a KFP data set artifact containing the generated synthetic data in JSONL format (original columns plus the generated question columns), along with execution metrics (input rows, output rows, and execution time). If the pipeline server is linked to Cloud Object Storage, the artifacts are persisted there.