Hermia: A Cross-Model, Cross-Backend LLM Inference Security Assessment Framework

Hermia is an open-source, vendor-agnostic security assessment framework designed to detect behavioral differences in LLM inference stacks across models, backends, and hardware configurations. Its core goal is to help developers identify potential security risks and consistency issues, addressing gaps in traditional model evaluation that overlook cross-platform consistency.

Key keywords: LLM, security assessment, inference consistency, cross-platform testing, AI safety, open-source framework.

Source: GitHub (maintained by scottblydotcom, updated on 2026-05-28T18:13:57Z, link: https://github.com/scottblydotcom/hermia)

With LLMs widely deployed in production, a critical issue emerges: the same model may produce inconsistent outputs across different backends, hardware, or software versions. This 'behavioral divergence' affects user experience and poses security risks (e.g., security filters failing on some backends).

Traditional methods focus on accuracy/performance but ignore cross-platform consistency, a pain point for enterprises using multi-cloud or heterogeneous hardware (GPU, TPU, AI chips).

Hermia's key features include:

1. Behavioral Divergence Detection: Sends identical inputs to different endpoints, comparing outputs for semantic differences, security policy gaps, format inconsistencies, or probability distribution deviations.

2. Cross-Stack Compatibility: Supports mainstream backends like PyTorch, Hugging Face Transformers, vLLM, cloud APIs, and local open-source models—acting as a universal test layer.

3. Hardware-Aware Testing: Identifies hardware-related issues (GPU models, drivers, CUDA versions) by running parallel tests across multiple environments.

Hermia applies to:

• Multi-Vendor Strategy: Verifies consistent security filtering across OpenAI, Anthropic, Azure OpenAI, helping adjust routing if any vendor under-filters harmful content.

• Model Upgrade: Compares old/new model outputs to detect 'behavior drift' before deployment.

• Self-Hosted vs Cloud: Quantifies differences to support smooth migration decisions.

Hermia extends red teaming to 'infrastructure red teaming'—testing deployment environments alongside models. Attackers may exploit backend-specific weaknesses (e.g., prompt injection effective on vLLM but not OpenAI API). Hermia helps uncover such blind spots, ensuring security across all deployment paths.

Hermia's structure includes:

• src/hermia: Core framework code.
• test-datasets: Boundary cases and adversarial samples.
• analysis: Result visualization tools.
• scripts: Automation scripts.
• docs: Documentation.

Security configs like .gitleaks.toml reflect maintenance focus on code security.

Hermia shifts LLM operations from 'how well does the model perform' to 'is it consistent across all environments'. It’s valuable for enterprise teams building LLM apps.

Future-wise, as multi-modal models and Agent systems emerge, cross-platform consistency testing will grow more critical, and Hermia-like frameworks will drive industry standardization.