ConjFormer: Privacy-Preserving Large Language Model Inference via Orthogonal Equivariant Transformers

Addressing the privacy leakage risks in large language model (LLM) inference, the research team proposes the ConjFormer architecture. Through orthogonal obfuscation and O(d) equivariance design, it reduces the token recovery rate from 35% to 1.3% without introducing noise or re-encryption, enabling efficient and practical privacy-preserving inference. This solution balances privacy and performance, providing a new path for cloud-based LLM inference.

Local deployment of LLMs is limited by resources, while outsourcing inference to the cloud poses privacy risks. Although traditional split inference schemes distribute computation between clients and servers, attackers can recover original tokens from hidden layer representations via nearest-neighbor search, leading to serious privacy vulnerabilities.

ConjFormer combines orthogonal obfuscation and equivariant architecture:

1. The client transforms hidden representations using a secret orthogonal matrix, disrupting the attacker's cosine similarity search;
2. An orthogonal equivariant Transformer is designed, including scalar RMSNorm (parameter-free normalization to maintain equivariance) and block orthogonal conjugate weights (Q W Q^T transformation), achieving O(d) equivariance and ensuring the server can reason correctly in the rotated space.

Experiments were conducted on GPT-2 and Llama3.2 1B models, fine-tuned with PubMed medical texts:

• Privacy: The token recovery rate dropped from over 35% to at most 1.3%, eliminating direct cosine inversion attacks;
• Performance: Perplexity increased by only 0.4%, downstream task performance was comparable to the original model, and additional computational overhead was minimal.

Compared to traditional solutions, ConjFormer has significant advantages: no noise injection (avoids performance degradation), no re-encryption (reduces computational overhead), and lightweight implementation (only modifies normalization layers and weight initialization, easy to integrate).

Potential application scenarios include medical text processing, financial data analysis, and enterprise document processing. Deployment considerations: the client should securely store the orthogonal matrix (e.g., using a hardware security module) and negotiate keys via a key exchange protocol.

Limitations: Existing models need fine-tuning to adapt to the equivariant architecture; key management requires additional security mechanisms; resistance to active attacks remains to be verified. Future directions: Explore more efficient equivariant architectures, study resistance to other attacks, and extend to multimodal and distributed inference.