SharpCortex.NET: A Local LLM Inference Orchestration Platform Based on .NET 10

SharpCortex.NET is a production-ready local LLM inference orchestration platform built with .NET10, React, and Ollama. Its core positioning is to provide enterprises and developers with locally deployable AI infrastructure, emphasizing maintainability, observability, streaming inference, and clean architecture. It ensures data privacy and security while offering production-grade features such as OpenAI-compatible APIs.

SharpCortex.NET is not just a simple chatbot application; it is a production-oriented AI orchestration platform. Its core positioning is to provide enterprises and developers with locally deployable LLM inference infrastructure, emphasizing operational simplicity, observability, and clean architecture. Developed by douglasbarbin, the project uses .NET10 as the backend tech stack, React as the frontend framework, and Ollama as the local model runtime environment, balancing backend performance stability and modern interactive experience.

SharpCortex.NET provides a range of production environment features:

• Local-first LLM Orchestration: All inference tasks are completed locally, ensuring data privacy by keeping data within the local environment
• ChatGPT-style Conversation Interface: Intuitive and familiar user experience
• OpenAI-compatible API Interface: Facilitates integration with existing tools and frameworks
• Streaming Token Response: Real-time output with typewriter effect
• Conversation Persistence: Supports saving and retrieving historical records
• Production-grade Observability: Integrates monitoring tools like OpenTelemetry and structured logging
• Clean Architecture Boundaries: Layered design ensures code maintainability

Backend Tech Stack

• ASP.NET Core (.NET10): High-performance web service
• Minimal APIs: Lightweight API definition reduces boilerplate code
• PostgreSQL: Reliable relational database for persistence
• EF Core: Object-relational mapping simplifies data access
• OpenTelemetry: Distributed tracing and metrics collection
• Polly: Resilience and transient fault handling
• Server-Sent Events (SSE): Foundation for streaming response technology

Frontend Tech Stack

• React + TypeScript: Type-safe component development
• Vite: Fast development and build tool
• TailwindCSS: Atomic CSS framework
• TanStack Query: Data fetching and state management
• Zustand: Lightweight state management

Architecture Design

Adopts a layered architecture with clear module responsibilities:

Module	Responsibility
SharpCortex.Api	HTTP API interface layer, middleware, dependency injection composition
SharpCortex.Core	Core orchestration logic and abstract definitions
SharpCortex.Infrastructure	Persistence, providers, external integrations
SharpCortex.Contracts	DTO and shared contract definitions

Development Philosophy

Prioritize: Simplicity, operational practicality, async-first approach, observability, clear dependency boundaries, scalability (avoiding premature complexity) Deliberately Avoid: Microservices architecture, Kubernetes orchestration, heavy CQRS/MediatR, distributed orchestration systems, plugin ecosystems, proxy frameworks A pragmatic approach keeps the project lightweight and reliable.

Completed Infrastructure

• Solution structure setup
• OpenAPI/Swagger documentation
• Structured logging system
• Basic OpenTelemetry integration
• PostgreSQL connection
• Health check endpoints

Next Steps

• React frontend initialization
• Conversation persistence storage
• Streaming inference pipeline
• Ollama provider integration
• Chat interface development

Hardware Support

Special support for AMD GPU's ROCm, with future potential to support NVIDIA CUDA, enabling efficient local inference.

Summary

SharpCortex.NET is a pragmatic solution for local LLM deployment. It does not aim to cover all AI scenarios but focuses on providing a reliable, observable, and easy-to-maintain local LLM inference platform. It is suitable for teams that want to deploy large language models in private environments without introducing excessive complexity.