Gemma4.java: A High-Performance Gemma 4 Inference Engine Implemented in Pure Java

Gemma4.java is an open-source project developed by mukel, providing a pure Java implementation of the Google Gemma4 series large language model inference engine. Its core features include zero dependencies (single Java file), support for multiple quantization formats, MoE architecture, and GraalVM native image. It aims to enable Java developers to deploy high-performance local LLM inference in enterprise and edge scenarios without relying on Python ecosystems.

Java is a dominant language in enterprise applications, but Python leads in AI development. Deploying LLMs in Java stacks faces challenges due to dependency and compatibility issues. Gemma4.java addresses this gap by offering a zero-dependency, lightweight solution for Java-based LLM inference.

Gemma4 is Google's latest open LLM series based on Gemini's underlying tech. It includes four models:

• E2B: ~5B dense, instruction-tuned, suitable for edge devices.
• E4B: ~8B dense, balanced performance and cost.
• 31B: ~310B dense, strong at complex tasks like code generation.
• 26B-A4B: ~260B MoE, only 4B activated per inference, balancing capability and efficiency.

Key features of Gemma4.java:

1. Single file zero dependency: Easy deployment, no version conflicts.
2. Full GGUF format support: Compatible with open LLM ecosystem's standard.
3. Multiple quantization types: F32/F16/BF16, Q4/Q5/Q6/Q8.
4. MoE architecture support: Efficient routing for sparse activation.
5. Hybrid attention: Sliding window + full attention layers.
6. KV cache optimization: Reduces redundant computation.
7. Java Vector API: SIMD acceleration for matrix operations.
8. GraalVM native image: Faster startup, lower memory.
9. AOT preload: Eliminates model parsing overhead.

Quick start steps:

• Env: Java21+ (required for MemorySegment), GraalVM25+ (optional).
• Get models: Download GGUF files from Hugging Face (e.g., unsloth/gemma-4-E2B-it-GGUF).
• Run: Use JBang (recommended: jbang Gemma4.java --chat), direct execution, JAR, or GraalVM native image (with AOT preload option).

Optimization tips:

• Choose quantization (Q4 for memory, Q8/BF16 for precision).
• Enable Vector API (add JVM params: --enable-preview --add-modules jdk.incubator.vector).
• Use GraalVM for better performance.
• AOT preload for low latency.

Application scenarios: Enterprise integration, edge devices, microservices, education/research.

Current limitations: Only supports Gemma4 models, CPU-only inference, requires Java21+.

Future plans: Extend to other models, add GPU acceleration, support distributed inference, improve quantization algorithms.

Gemma4.java breaks the stereotype that AI development must use Python, opening local LLM deployment for Java developers. Its zero-dependency design simplifies deployment and customization, promoting AI democratization in enterprise applications.