# ConjFormer: Privacy-Preserving Large Language Model Inference via Orthogonal Equivariant Transformers

> Addressing the privacy leakage risks in large language model (LLM) inference, the research team proposes the ConjFormer architecture. Through orthogonal obfuscation and O(d) equivariance design, it reduces the token recovery rate from 35% to 1.3% without introducing noise or re-encryption, enabling efficient and practical privacy-preserving inference.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-06-15T09:31:24.000Z
- 最近活动: 2026-06-16T02:52:20.467Z
- 热度: 133.7
- 关键词: 隐私保护, 大语言模型推理, 正交等变, ConjFormer, 分割推理, Transformer架构, RMSNorm, 云端推理安全
- 页面链接: https://www.zingnex.cn/en/forum/thread/conjformer-transformer
- Canonical: https://www.zingnex.cn/forum/thread/conjformer-transformer
- Markdown 来源: floors_fallback

---

## Introduction: ConjFormer—A New Solution for Privacy-Preserving LLM Inference

Addressing the privacy leakage risks in large language model (LLM) inference, the research team proposes the ConjFormer architecture. Through orthogonal obfuscation and O(d) equivariance design, it reduces the token recovery rate from 35% to 1.3% without introducing noise or re-encryption, enabling efficient and practical privacy-preserving inference. This solution balances privacy and performance, providing a new path for cloud-based LLM inference.

## Background: Privacy Dilemma of Cloud-Based LLM Inference

Local deployment of LLMs is limited by resources, while outsourcing inference to the cloud poses privacy risks. Although traditional split inference schemes distribute computation between clients and servers, attackers can recover original tokens from hidden layer representations via nearest-neighbor search, leading to serious privacy vulnerabilities.

## Methodology: Core Innovative Design of ConjFormer

ConjFormer combines orthogonal obfuscation and equivariant architecture:
1. The client transforms hidden representations using a secret orthogonal matrix, disrupting the attacker's cosine similarity search;
2. An orthogonal equivariant Transformer is designed, including scalar RMSNorm (parameter-free normalization to maintain equivariance) and block orthogonal conjugate weights (Q W Q^T transformation), achieving O(d) equivariance and ensuring the server can reason correctly in the rotated space.

## Evidence: Experimental Validation and Performance Evaluation

Experiments were conducted on GPT-2 and Llama3.2 1B models, fine-tuned with PubMed medical texts:
- Privacy: The token recovery rate dropped from over 35% to at most 1.3%, eliminating direct cosine inversion attacks;
- Performance: Perplexity increased by only 0.4%, downstream task performance was comparable to the original model, and additional computational overhead was minimal.

## Technical Advantages: Comparison with Existing Privacy Solutions

Compared to traditional solutions, ConjFormer has significant advantages: no noise injection (avoids performance degradation), no re-encryption (reduces computational overhead), and lightweight implementation (only modifies normalization layers and weight initialization, easy to integrate).

## Application Prospects and Deployment Considerations

Potential application scenarios include medical text processing, financial data analysis, and enterprise document processing. Deployment considerations: the client should securely store the orthogonal matrix (e.g., using a hardware security module) and negotiate keys via a key exchange protocol.

## Limitations and Future Directions

Limitations: Existing models need fine-tuning to adapt to the equivariant architecture; key management requires additional security mechanisms; resistance to active attacks remains to be verified. Future directions: Explore more efficient equivariant architectures, study resistance to other attacks, and extend to multimodal and distributed inference.