SentGuard: Sentence-Level Streaming Guard for Real-Time Unsafe Content Detection During Inference, 90.5% Detection Rate with Only 7.41% False Positive Rate

SentGuard proposes a sentence-level streaming content moderation solution. It detects security risks at sentence boundaries using a lightweight waiting buffer, achieving a 90.5% detection rate and a 7.41% false positive rate across 5 security benchmarks, balancing the timeliness and accuracy of moderation in streaming generation scenarios.

Characteristics of Streaming Generation

• Incremental output: tokens generated and sent one by one
• Long responses: modern LLMs often generate lengthy content
• Inference-intensive: involves complex reasoning processes

Polarization of Existing Guards

• Response-level moderation: Moderates after full response, accurate but delayed intervention
• Token-level moderation: Moderates each token in real time, timely but semantically incomplete and prone to over-triggering

Neither method balances timeliness and accuracy.

Core Insight: Sentences as Moderation Units

• Semantically complete: sentences are the smallest complete semantic units
• Clear boundaries: punctuation marks indicate the end
• Feasible for streaming: natural sentence boundaries exist

Architecture Design

• Lightweight waiting buffer: Aggregates tokens into sentence chunks, releases complete sentences to users, introducing minimal delay
• Parallel moderation mechanism: Runs in parallel with LLM without blocking generation
• Coarse-to-fine training objectives: First identify risks, then locate types, training early detection capabilities

StreamSafe Benchmark

• Sentence-by-sentence annotation: each sentence has an independent safety label, covering 8 types of harmful content
• 8 harmful categories: violence, hate speech, self-harm, sexual content, harassment, dangerous activities, illegal behavior, privacy leakage
• Distinguishes between reasoning and response paragraphs

Experimental Results

• Detection rate: detects 90.5% of unsafe cases within two sentences
• False positive rate: only 7.41%
• Baseline comparison: outperforms token-level (low detection, high false positives) and response-level (high latency) methods
• Cross-benchmark consistency: stable performance across 5 benchmarks

Method	Detection Rate	False Positive Rate	Latency
Token-Level	Lower	Higher	Lowest
Response-Level	High	Low	Highest
SentGuard	90.5%	7.41%	Medium

Applicable Scenarios

• Real-time chat systems
• Content generation platforms
• Enterprise-level deployments
• Multilingual applications

Deployment Architecture

• Independent service: microservices running in parallel
• Integration module: embedded into existing inference frameworks
• Edge deployment: local moderation on client/edge nodes

Integration and Configurability

• Supports frameworks like vLLM and TensorRT-LLM
• Configurable sensitivity thresholds, risk category weights, and latency tolerance

Limitations

• Language dependency: sentence boundary definitions vary by language
• Long sentence processing: extremely long sentences may affect performance
• Adversarial attacks: vulnerable to adversarial examples

Future Directions

• Multilingual expansion: optimize for non-Latin scripts
• Adaptive thresholds: dynamically adjust sensitivity
• Interpretability: provide decision explanations
• Human-machine collaboration: introduce human moderation for high-risk scenarios

SentGuard finds a balance between response-level and token-level methods through sentence-level moderation. The 90.5% detection rate and 7.41% false positive rate prove its effectiveness while maintaining a streaming experience. The StreamSafe benchmark provides a standardized evaluation platform for future research, offering a robust solution for user protection in real-time LLM interactions.