TensorSharp: A New Inference Engine for Running Large Language Models Locally with C#

TensorSharp is a C#-based local inference engine for large language models, supporting GGUF format model files, providing command-line and web interfaces, and enabling multimodal conversations.

With the rapid development of Large Language Model (LLM) technology, more and more developers and enterprises want to run these models in local environments to protect data privacy and reduce reliance on cloud services. However, most existing inference engines are written in Python or C++, which often creates barriers for developers in the .NET ecosystem to integrate and use these tools.

TensorSharp fills this gap. It is a fully C#-developed inference engine that allows .NET developers to run large language models locally within their familiar ecosystem, without relying on external Python environments or complex C++ bindings.

TensorSharp was created by developer Zhongkai Fu and is an open-source C# inference engine specifically designed for running large language models in GGUF format. GGUF is a model file format promoted by the llama.cpp project, known for its efficient storage and loading performance.

The project is not just a simple model loader but a complete inference solution, including:

• Core Tensor Library (TensorSharp)：Provides tensor types, storage abstractions, and an extensible operation registry. The CPU implementation uses System.Numerics.Vectors for SIMD acceleration.
• GGML Backend Binding (TensorSharp.GGML)：Provides GPU acceleration implementation via a native C++ bridge.
• Inference Engine：Implements model-specific logic, including GGUF parsing, tokenization, chat template rendering, and forward propagation for various architectures.
• Application Layer：Offers a command-line console application and a web chatbot interface based on ASP.NET Core.

TensorSharp currently supports a variety of mainstream open-source large language models:

Model Series	Supported Versions	Multimodal Capabilities
Gemma 4	gemma-4-E4B, gemma-4-31B	Image, Video, Audio
Gemma 3	gemma-3-4b, etc.	Image
Qwen 3	Qwen3-4B, etc.	Text-only
Qwen 3.5	Qwen3.5-9B, etc.	Image

Notably, support for the Gemma 4 model enables TensorSharp to handle image, video, and audio inputs. For video input, the system uses OpenCV to extract up to 8 frames (1 frame per second) for processing; for audio input, it supports WAV (16kHz mono), MP3, and OGG Vorbis formats.

TensorSharp has designed three computing backends to adapt to different hardware environments:

Enabled via the --backend ggml_metal parameter, it uses the Apple Metal framework to achieve GPU acceleration on macOS. This is the optimal performance choice for Apple Silicon devices.

Enabled via the --backend ggml_cpu parameter, it uses the native GGML library for CPU inference, including optimized kernel implementations.

Enabled via the --backend cpu parameter, this is a fully portable CPU inference implementation that does not rely on any native libraries, making it suitable for scenarios with special deployment environment requirements.