TensorGate: AI Security Middleware for Production Environments, Enabling Real-Time LLM Traffic Detection and Semantic Cleaning

TensorGate is an open-source AI security middleware based on ASP.NET Core and YARP, designed specifically to address unique security risks in LLM applications such as prompt injection and malicious payloads. It enables real-time traffic detection and semantic cleaning via a local ONNX inference engine, balancing high performance (zero memory allocation reverse proxy), data privacy (local inference), and customizability to provide enterprise-level security protection for production environments.

Traditional API gateways or WAFs mainly target conventional web attacks and are ineffective against semantic-level risks unique to LLMs, such as prompt injection and jailbreak attacks. Therefore, the TensorGate team designed a security middleware that can understand semantics and identify intentions. They chose to build it with ASP.NET Core and YARP to leverage the high-performance features of the .NET ecosystem, ensuring compatibility with existing tech stacks and enabling frictionless integration.

YARP Reverse Proxy with Zero Memory Allocation

Based on Microsoft's YARP reverse proxy library, it implements a zero-memory-allocation request processing path, avoiding performance jitter caused by memory allocation and garbage collection under high concurrency, ensuring the security detection layer does not become a bottleneck.

Local ONNX Inference Engine

It uses local ONNX runtime inference, with advantages including: data privacy (sensitive data does not leave the local environment), low latency (millisecond-level detection), controllable costs (no cloud API call fees), and offline availability. It supports model export from multiple frameworks, facilitating custom updates of security models.

Real-Time Payload Inspection Mechanism

1. Syntax-level analysis: Detect known prompt injection patterns (e.g., role-playing instructions, system prompt overrides);
2. Semantic-level understanding: Identify real intentions via embedding models;
3. Content classification: Perform security rating on inputs to distinguish between normal, gray-area, and risky content.

1. Enterprise API Gateway Enhancement: Deployed between the API gateway and LLM services to block all malicious requests;
2. Multi-Tenant SaaS Protection: Supports configuration-based policy routing to set differentiated detection rules for different tenants;
3. Development and Testing Environment Security: Acts as a sandbox gatekeeper to prevent data leakage or inappropriate content generation during testing.

Feature	TensorGate	Traditional WAF	Cloud AI Security API
Deployment Location	Local/Private Cloud	Network Edge	Cloud
Semantic Understanding	Supported	Limited	Supported
Data Privacy	Fully Local	Partially Local	Requires Transmission to Cloud
Latency	Low	Low	Medium-High
Cost Model	Fixed Infrastructure	Fixed Infrastructure	Pay-per-Call
Customization	High	Medium	Low-Medium

Unique Value of TensorGate: Combines the semantic understanding capabilities of cloud solutions with the privacy and performance advantages of local deployment, and is open-source and customizable.

1. More Model Support: Expand dedicated detection models for architectures like Llama and Mistral;
2. Response Content Detection: Implement bidirectional protection for input and output;
3. Observability Enhancement: Integrate OpenTelemetry to provide fine-grained security event tracking;
4. Policy as Code: Support declarative configuration or code-defined security policies for easier version management and collaboration.

TensorGate deeply integrates AI security capabilities into the application infrastructure layer, rather than being an external add-on. For production-grade AI application teams, this architectural approach is worth considering. As LLMs become more popular, dedicated security layers like TensorGate may become standard architectural components, as indispensable as current API gateways and identity authentication services.