Core58: An Inference Framework for Running 1.58-bit and Ternary LLMs on Windows

Core58 is an inference framework optimized for the Windows platform, supporting the operation of BitNet 1.58-bit and ternary quantized large language models (LLMs) on CPU/GPU. It provides out-of-the-box precompiled versions and built-in chat tools, aiming to lower the threshold for LLM deployment and allow ordinary PC users to experience the inference capabilities of locally run extreme quantization LLMs.

Quantization technology converts model weights from high precision (e.g., FP32/FP16) to low precision (e.g., INT8, 1.58-bit). Its core motivations include: reducing storage requirements (70B FP16 model: 140GB → 1.58-bit: only 13GB), alleviating memory bandwidth pressure, improving inference speed, and lowering deployment costs. BitNet 1.58-bit, proposed by Microsoft, restricts weights to {-1,0,1}, requiring only about 1.58 bits per weight; ternary quantization is a similar variant. These technologies enable resource-constrained devices to run large models.

• Platform Focus: Optimized specifically for Windows, making full use of Windows ecosystem resources;
• Multi-Precision Support: Supports both BitNet 1.58-bit and ternary quantized models;
• Heterogeneous Computing: Compatible with CPU and GPU inference, flexibly adapting to hardware;
• Out-of-the-Box: Provides precompiled versions, no source code compilation required;
• User-Friendly Interaction: Built-in chat tool to simplify user operations.

1. Solving 1.58-bit Inference Challenges: Custom implementation for non-standard data types, optimizing computational efficiency via lookup tables/bit operations, and designing quantization-dequantization strategies to maintain precision;
2. CPU Inference Optimization: Utilizes SIMD instruction sets like AVX/AVX2/AVX-512, optimizes memory layout (cache-friendly), and supports multi-threaded parallelism;
3. GPU Inference Support: Adapts to NVIDIA CUDA and AMD ROCm platforms, with efficient video memory management and asynchronous execution to maximize GPU utilization.

• Local AI Assistant: Windows PC users run local models to protect privacy without needing an internet connection;
• Edge Deployment: Windows edge devices (industrial control, retail terminals, etc.);
• Development and Testing: AI developers quickly test models without a complex Linux environment;
• Educational Use: Students/researchers learn LLM technology (with limited hardware resources);
• Offline Environments: Scenarios where internet access is unavailable or cloud services are prohibited.

• vs llama.cpp: llama.cpp is cross-platform, but Core58 is optimized for Windows, offering better performance and experience;
• vs Ollama: Ollama is easy to use, but Core58 focuses on extreme quantization (1.58-bit), which is more advantageous in resource-constrained scenarios;
• vs Native PyTorch/Transformers: Native frameworks are flexible, but Core58 has higher optimization efficiency for specific quantization formats.

Core58 lowers the usage threshold through:

• Precompiled Versions: Provides release-ready builds for direct download and use;
• Simple Configuration: Specifies model paths and inference parameters via configuration files or command-line arguments;
• Chat Interface: Built-in interactive tool similar to ChatGPT;
• API Support: May provide interfaces compatible with OpenAI API for easy integration into existing applications.

• Limitations: Only supports specific 1.58-bit/ternary quantized models, exclusive to Windows, extreme quantization has precision loss, and still requires certain hardware performance;
• Future Trends: Popularization of edge AI, green AI (low energy consumption), democratized access (lowering hardware thresholds), dynamic adjustment of mixed precision;
• Conclusion: Core58 provides Windows users with an option for locally run extreme quantization LLMs. Although there is a compromise in precision, it significantly reduces deployment costs and will play an important role in the popularization of AI.