gptoss.java: A High-Performance GPT-OSS Inference Engine Implemented in Pure Java

A zero-dependency, single-file pure Java inference engine that supports OpenAI's GPT-OSS models (including MoE variants), uses Java Vector API for efficient matrix operations, and provides GraalVM Native Image support.

gptoss.java is an impressive technical project that implements a high-performance inference engine for OpenAI's GPT-OSS models using pure Java. The entire project consists of just one Java file with zero external dependencies, yet it fully supports GPT-OSS models ranging from 20B to 120B parameters, including Mixture of Experts (MoE) architecture variants. This project demonstrates Java's potential in modern AI inference scenarios, breaking the stereotype that "Python monopolizes large model inference". By fully leveraging new features of Java 21+, especially the Vector API and MemorySegment, the project achieves performance comparable to C++-based inference frameworks.

The project's most distinctive feature is its minimalist architectural design. The entire inference engine is contained in a single Java file without any external library dependencies. This design offers several significant advantages:

Easy Deployment: No need to handle complex dependency management—just run the single file. This is a huge advantage for scenarios requiring lightweight deployment packages (e.g., edge computing, embedded systems).

Auditability: The code is fully visible with no hidden dependencies, making it easy for security audits and compliance checks.

Portability: It can run on any platform with a Java 21+ runtime environment, without being restricted by specific machine learning frameworks.

The project implements an efficient GGUF format parser that supports various quantization and data types:

• Floating-point types: F16 (half-precision float), BF16 (Brain float), F32 (single-precision float)
• Quantization types: Q4_0, Q4_1, Q4_K, Q5_K, Q6_K, Q8_0 (4 to 8-bit quantization)
• New format: MXFP4 (Microscaling FP4, 4-bit float)

This extensive format support means users can directly use community pre-quantized models without needing to convert them themselves, greatly lowering the barrier to use.

The project fully leverages Java's Vector API (JEP 469) to implement an efficient matrix-vector operation core. The Vector API allows Java code to directly utilize the CPU's SIMD (Single Instruction Multiple Data) instruction set, achieving computational performance close to hardware limits on modern processors. Compared to traditional scalar operations, SIMD acceleration can bring several-fold or even order-of-magnitude performance improvements, especially in compute-intensive tasks like large model inference. This is the key reason why gptoss.java can achieve high performance in a pure Java implementation.

The project fully supports GraalVM Native Image compilation, which can compile Java code into a native executable. This brings two important benefits:

Startup Speed: Native images eliminate JVM cold start overhead, achieving millisecond-level startup—critical for interactive applications requiring fast responses.

Memory Footprint: The memory footprint of native images is significantly lower than traditional JVM applications, making it possible to run large models in resource-constrained environments.

The project supports preloading model data into the executable file at compile time. By setting the PRELOAD_GGUF environment variable and recompiling, you can generate a dedicated binary file containing a specific model. This AOT (Ahead-of-Time) preloading eliminates runtime model parsing overhead, significantly reducing the Time-to-First-Token (TTFT), which notably improves the user experience for interactive chat applications.

The project provides multiple usage methods to adapt to different scenario requirements: