Deliverance: A High-Performance LLM Inference Engine Based on Java

Deliverance is a high-performance LLM inference engine developed in Java, designed to fill the gap in the Java ecosystem for LLM inference. It provides native LLM inference capabilities for Java enterprise applications, enabling core tasks such as model loading, inference computation, and text generation without relying on external Python services, helping Java systems integrate AI capabilities with low barriers.

In the field of LLM inference engines, Python dominates with frameworks like PyTorch and TensorFlow. However, many enterprise applications based on the Java tech stack face additional system complexity and operation costs when integrating Python inference services. Deliverance was created to fill this gap by providing native Java LLM inference capabilities.

Advantages of Pure Java Implementation

• Ecosystem Integration: Seamless integration with Java microservice architecture
• Performance Optimization: Leveraging JVM's JIT compilation and GC optimization
• Type Safety: Static typing reduces runtime errors
• Simplified Deployment: Single tech stack lowers operation complexity
• Concurrent Processing: Mature concurrency model supports high throughput

Core Capabilities of the Inference Engine

Model Loading and Management: supports loading of quantized formats like GGUF, memory-mapped caching, multi-model concurrency, and dynamic switching Text Generation: autoregressive generation, configurable sampling strategies (Temperature/Top-p/Top-k), streaming output Inference Optimization: KV Cache reuse, batch processing, memory optimization

Architecture Design

Modular architecture includes core layer (Transformer/attention), model layer (Llama/Mistral adaptation), quantization layer (INT8/INT4), and API layer (Java-friendly interface)

Enterprise Java Application Integration

Applicable to banking, insurance, telecom industries: intelligent customer service, document processing, code assistance, local inference for sensitive data (compliance requirements)

Edge Computing and IoT

Lightweight design adapts to edge devices: edge gateway local inference, industrial control system real-time decision-making, smart terminal offline AI

Cloud-Native Deployment

Supports containerization, Spring Boot integration, Kubernetes elastic scaling, Prometheus observability metrics export

Pure Java Tensor Operations

No dependency on external C++/CUDA libraries; implements core tensor operations, bringing better portability and deployment convenience with considerable performance in CPU scenarios

Memory Management Optimization

• Memory-mapped loading of model weights
• Efficient KV Cache reuse
• Inference memory pool management
• Paged loading and swapping for large models

Modular Expansion

Reserved extension points support new model architectures (MoE/Mamba), quantization schemes, custom sampling strategies, and pluggable preprocessing/postprocessing

Dimension	Deliverance	llama.cpp	vLLM	Python Transformers
Language	Java	C/C++	Python	Python
Java Ecosystem	Native support	JNI wrapper	Remote call	Remote call
Deployment Complexity	Low	Medium	High	High
Performance Optimization	JVM tuning	Extreme optimization	GPU optimization	Framework-dependent
Applicable Scenarios	Java enterprise applications	High-performance inference	High-throughput services	Research experiments

Getting Started Path

1. Environment Preparation: JDK17+, G1GC/ZGC recommended
2. Model Acquisition: Download GGUF-compatible models
3. Dependency Introduction: Add project dependencies via Maven
4. API Call: Implement text generation using high-level APIs
5. Performance Tuning: Adjust JVM parameters and inference configurations

Production Deployment Recommendations

• Reserve sufficient heap memory (model size + inference overhead)
• Configure GC strategy (use ZGC/Shenandoah for low latency)
• Manage concurrent requests with thread pools
• Monitor memory usage and inference latency

Deliverance proves that Java can handle LLM inference tasks in specific scenarios, providing Java developers with an AI integration solution without cross-language dependencies. Future expectations include:

• Support for more model architectures
• Deep integration with frameworks like Spring AI
• Improvement of enterprise-level features (security, monitoring, multi-tenancy)
• Maturation of cloud-native deployment solutions

For Java teams, Deliverance is a noteworthy option for scenarios requiring local deployment, sensitive data privacy, or tight integration with existing Java systems.