Bifrost: A Hybrid TEE-FHE Architecture for Privacy-Preserving Large Model Inference Services

Bifrost is a hybrid architecture combining Trusted Execution Environment (TEE) and Fully Homomorphic Encryption (FHE) to address the privacy-performance dilemma in cloud-based large language model (LLM) inference. It protects user data privacy while significantly improving inference efficiency.

Basic Information:

• Original Title: Bifrost: Hybrid TEE-FHE Inference for Privacy-Preserving Transformer and LLM Serving
• Source: arXiv
• Link: http://arxiv.org/abs/2606.17421v1
• Release Time: 2026-06-16
• Authors: arXiv paper author team

Cloud-hosted LLM inference faces a privacy challenge: user prompts may contain sensitive info (code, trade secrets, personal data), but remote services expose intermediate states to cloud stacks. Existing solutions have flaws:

• FHE: Theoretically enables 'data usable but not visible' but causes extremely high latency due to interactions between Transformer ops (linear/nonlinear layers, attention cache) and ciphertext operations.
• Pure TEE: Hardware-isolated execution (e.g., Intel SGX, AMD SEV) protects data but can't leverage untrusted accelerators (GPU/NPU) critical for LLM efficiency.

Bifrost's core idea is to split inference tasks between TEE and FHE:

• FHE for linear layers: Handles projection layers and feed-forward networks (parallelizable ops) on CKKS-supported accelerators, ensuring accelerators can't access raw data.
• TEE for nonlinear & state management: Executes nonlinear activations, attention control logic, KV cache state transitions, and ciphertext refresh inside CPU TEE (avoids FHE overhead while maintaining security).

Key principle: Only CPU TEE can access keys and plaintext; accelerators, memory, drivers, and host software are outside the trusted computing base.

Bifrost+ introduces a prefill-decode separation strategy:

• Prefill phase: Prompt processing (KV state construction) is done inside CPU TEE (avoids large ciphertext computation overhead for long prompts, especially in multi-turn dialogues).
• Decode phase: Token generation uses the hybrid TEE-FHE path (improves latency-sensitive user experience).

This separation significantly reduces overhead from long prompts.

Experimental results validate Bifrost's effectiveness:

• Bifrost vs FHE: 9.25x latency reduction on GPT-2 (1.5B params), 9.91x on LLaMA3 (8B params).
• Bifrost+ vs direct FHE:
  • GPT-2 (124M params): First token generation time (TTFT) reduced by 14.6-45.8x.
  • Qwen3 (0.6B params): TTFT reduced by15.3-53.4x.

These results show Bifrost brings privacy-preserving LLM inference close to practical performance levels.

Bifrost represents a key advance in privacy-preserving LLM inference. Its 'selective encryption execution' philosophy provides a valuable paradigm: instead of applying FHE to all computations, use FHE only for accelerator-delegated ciphertext ops, and keep nonlinear ops, ciphertext refresh, and prompt processing in CPU TEE.

This approach balances security and performance, addressing the limitations of single-technology solutions.

Prospects: Bifrost enables privacy-preserving LLM use in sensitive fields like healthcare (patient data), finance (confidential reports), and law (legal documents). Enterprises can safely use cloud LLMs for internal data processing without leakage risks.

Challenges:

1. Deployment complexity: Fine-grained system design and tuning for TEE-FHE collaboration.
2. Standardization: Compatibility issues between different TEE implementations and FHE libraries.
3. Cost: Extra overhead compared to plaintext inference; balancing cost and privacy in commercial scenarios needs further exploration.