Synth-Forge: A Local-First Synthetic Data Generation and Privacy Protection Testing Framework

Synth-Forge Overview

Synth-Forge is a local-first synthetic data generation and privacy protection testing framework designed for intelligent agent workflow testing. Key highlights:

• Local-first: Runs entirely offline, no cloud dependency, zero API cost.
• Privacy protection: Automatically identifies and desensitizes PII (Personal Identifiable Information).
• Agent workflow adaptation: Supports multi-round dialogue simulation, tool call scenarios, and boundary condition testing.

Basic Info:

• Author/Maintainer: outsidegem
• Source: GitHub (https://github.com/outsidegem/synth-forge)
• Release/Update Time: 2026-06-14T17:16:18Z

Project Background & Motivation

When building and testing agentic workflows, developers face a core dilemma: obtaining diverse, realistic test data without exposing sensitive information. Traditional solutions have limitations:

• Cloud-based models: Risk of data leakage and ongoing API costs.
• Static test sets: Fail to cover complex real-world scenarios.

Synth-Forge addresses this by adopting a local-first architecture, enabling offline synthetic data generation with automatic PII desensitization.

Core Features

1. Local Data Synthesis Engine:

   • Runs locally (zero API cost, offline available, low latency).
   • Supports structured (tables, JSON), semi-structured text, and multi-modal metadata simulation.

2. PII Recognition & Desensitization:

   • Identifies direct identifiers (name, ID, phone, email), quasi-identifiers (location, DOB), and sensitive attributes (medical/financial records).
   • Desensitization strategies: anonymization (pseudonyms), partial masking (e.g., 138****8888), generalization (address to region).

3. Agent Workflow Adaptation:

   • Multi-round dialogue simulation (context coherence, memory testing).
   • Tool call scenario simulation (external API response formats).
   • Boundary condition generation (edge cases, abnormal inputs).

Technical Architecture

• TypeScript Stack: Ensures type safety and cross-platform compatibility. Core modules:

  • Generator: Synthesizes various data patterns.
  • Detector: Rule/pattern-based PII identification.
  • Sanitizer: Executes desensitization strategies.
  • Test Suite: Validates data quality and desensitization effectiveness.

• Extensible Plugin Mechanism:

  • Custom data generation rules for specific business scenarios.
  • Extend PII recognition patterns for industry-specific needs.
  • Integrate external data sources for hybrid generation.

Application Scenarios

1. Agent Development Testing:

   • Generate realistic user queries (common intents + edge cases).
   • Simulate multi-round dialogue to test memory and reasoning.
   • End-to-end testing without real customer data.

2. RAG System Evaluation:

   • Generate QA pairs with ground truth for retrieval accuracy assessment.
   • Simulate document libraries to test chunking strategies.
   • Adversarial queries to evaluate robustness.

3. Privacy Compliance Pre-check:

   • Validate desensitization process effectiveness.
   • Assess re-identification risk of de-identified data.
   • Test data analyzability after anonymization.

Comparison with Existing Solutions

Feature	Synth-Forge	Cloud API Solutions	Static Test Sets
Data Privacy	Fully local, zero leakage risk	Data transmitted to cloud	Local storage
Generation Diversity	Highly configurable	Dependent on model capability	Fixed/limited
Cost	One-time development cost	Ongoing API fees	No running cost
Offline Availability	Supported	Not supported	Supported
Scenario Customization	Flexible extension	Limited by API parameters	Manual maintenance

Future Prospects & Community Contribution

Potential Directions:

• Multi-modal data support (images, audio, video synthesis/desensitization).
• Federated learning integration for distributed collaborative data generation.
• Differential privacy enhancement (mathematically provable privacy protection).

How to Contribute:

• Expand PII recognition patterns for more languages/regions.
• Optimize data generation authenticity and diversity.
• Develop visual configuration interfaces.

Conclusion

Synth-Forge represents an important trend in agent development tools: balancing large model capabilities with local data processing to ensure privacy and efficiency. For enterprise teams building agent applications, it provides a secure, controllable test data solution, making it a valuable addition to the tech stack.